Line folding can can break in the middle of a UNICODE character

[fawkesley]

Hello again!

I was struggling with a mysterious issue where a character in my calendar was getting mangled.

After quite a lot of research I pinned it down to the way serialize counts line length. The implementation allows it to break a line in the middle of a character.

For example, ⌘ requires 3 bytes to encode. If it occurs on a line after 74 bytes, the folding routine breaks in the middle of it.

I've made a fix on my fork with some tests to pin down the behaviour:

fawkesley@67e08db

Happy to raise a pull request (in a rush atm), but just wanted to show you what I found for now.

[arran4]

Hi Paul,

Thanks again! I would have gotten away with it too if it wasn't for the meddling rest-of-the-world. Okay. I have read the RFC and glanced at your code. I'm happy with your approach. Just a couple things I added comments on.. Please submit the PR when ready. :)

[fawkesley]

Haha yes, these pesky users! Thanks again for making it, it's been a really helpful start. You might like to know we're using it to power the iCal feed for the Internet freedom festival

I've found another issue with the new folding approach (it should escape newlines I think). Once I'm more confident it's working correctly for different cases I'll open a PR 👍

[arran4]

Thanks..

What was the issue you found?

[arran4]

@fawkesley bump.

[arran4]

@fawkesley
I will close this on or after the 15th of February if I haven't heard from you.

[fawkesley]

No worries! If I recall correctly, I implemented the necessary changes on my own fork to get things working. It wasn’t a clean patch as I had to fix the tests which were broken at the time.

[arran4]

Ah.. Did you want to fix it up, then create a draft PR and link me so I can have a look at the diff?

[arran4]

Do you have any tests which prove if this is still an issue? I am happy to incorporate them and try fix them? I remember someone doing some work along these lines.