Conversation

Notices

1. Embed this notice
   Brian Campbell (unlambda@hachyderm.io)'s status on Sunday, 01-Jun-2025 13:38:47 JST Brian Campbell

   • Rich Felker

   @dalias The process of normalizing to NFC is to decompose to NFD, which has a normalized ordering of combining characters, and then re-composing to NFC.

   It's possible for this to end up with more characters than you started with if you start with, say, two characters, one of which is precomposed combination of a character and two combining characters, and the third is a separate combining character

   When you decompose these into four characters, the base character and the three following combining character, the combining characters may be re-ordered for the canonical ordering.

   Then when you re-compose them, it may be the case that only the base and first combining character are available pre-composed, leaving you with one pre-composed character with the first combining character, followed by two combining characters.

   So a two-character sequence has now become a three-character sequence when normalizing under NFC.

   Contrived example: U+1EB7 U+0328 (Latin Small Letter A with breve and dot below / Combining Ogonek) normalizes in NFC to U+0105 U+0323 U+0306 (Latin Small Letter A with ogonek / Combining Dot Below / Combining Breve)

   • Haelwenn /элвэн/ :triskell: likes this.
   • Embed this notice
     Rich Felker (dalias@hachyderm.io)'s status on Monday, 02-Jun-2025 01:30:15 JST Rich Felker

     in reply to

     @unlambda Yep, there are even more cursed and simple examples tho. I'll point some out once the poll is over.

   • Embed this notice
     Brian Campbell (unlambda@hachyderm.io)'s status on Monday, 02-Jun-2025 01:58:40 JST Brian Campbell

     in reply to

     • Rich Felker

     @dalias oh, I don't doubt it. pretty much the only constant of dealing with Unicode is that there's always some way in which it can be more cursed.

     And even once you go beyond the spec into dealing with the real world and all of the ways people do things that don't actually match the spec, or match the spec from 20 years ago and haven't been updated since, or the like.

     I at one point had the fun of trying to implement some kind of sane cross-platform network filesystem filename normalization, in a way that did anything sensible given all the different ways in which different platforms handle filenames, including macOS which normalizes using some ancient frozen version of NFKC.

     And then we had some applications that we needed to support on top of this which only supported legacy codepages so you needed to test them out with your system set to the appropriate locale in order to run them using various different legacy codepages.

     Had a lot of fun getting my QA team to try and navigate the applications for testing when the interface was all in Hangul.