On 2016-03-27 13:29, John Cowan wrote: Haha. Yes... Except that you now have multiple representations of each character within one character set. So what has improved??? Johnny -- Johnny Billquist || "I'm on a bus || on a psychedelic trip email: bqt(a)softjar.se || Reading murder books pdp is alive! || tryin' to stay hip" - B. Idol

On 2016-03-27 23:49, Greg 'groggy' Lehey wrote: Another one I noted a while ago was that functions and command in Unix, such as lpq, which try to print things in nice columns now fail, because the code don't actually know how many characters have been output. And let's not even talk about such wonderful concepts as colors in the character set definition... Unicode seems to have it all... I wonder how many code points exist for 'A'. It's definitely more than one... Johnny -- Johnny Billquist || "I'm on a bus || on a psychedelic trip email: bqt(a)softjar.se || Reading murder books pdp is alive! || tryin' to stay hip" - B. Idol

On 2016-03-27 23:59, Greg 'groggy' Lehey wrote: Oh, definitely. I'm trying to limit myself to Latin-A at the moment. Otherwise the list will just be ridiculously long. You have, of course, U+41, but you also have U+FF21. But if you want to go slightly silly, you also have U+1F110, U+1F130, U+1F150, U+1F170, U+1F1E6, U+E0041... And god know if I've missed some other ones. Of course whitespace and other typographic details matters. That's why we have different code points for the letter, depending on things like whitespace. Johnny -- Johnny Billquist || "I'm on a bus || on a psychedelic trip email: bqt(a)softjar.se || Reading murder books pdp is alive! || tryin' to stay hip" - B. Idol

On Sun, 27 Mar 2016, Charles Anthony wrote: That, sir, is sheer genius! I dips me lid to you... -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer."

On Sun, 27 Mar 2016, John Cowan wrote: Wow - thanks! -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer."

Johnny Billquist scripsit: Mojibake, though not unknown, is now much less common, and the number of documents on the web that are in UTF-8 (including its ASCII subset) is at 85% and rising. That particular piece of code still works if the encoding is UTF-8. Fundamentally, Unicode is complicated because human writing systems are complicated. Well, if the font isn't fixed-width, you're screwed anyway. But if it is, there is information in the Unicode tables that tells you which characters have widths of 0, 1, or 2. Print programs can be modified to use that information. Colors are optional. Other than Greek and Cyrillic A letters, there are the math letters, which are used *in plain text* to designate semantic differences: plain A, italic A, and bold A mean different things mathematically. Using the math italics for emphasis or book titles is a Bad Thing. -- John Cowan http://www.ccil.org/~cowan cowan(a)ccil.org There was an old man Said with a laugh, "I From Peru, whose lim'ricks all Cut them in half, the pay is Look'd like haiku. He Much better for two." --Emmet O'Brien

Johnny Billquist scripsit: Yes, there is also incidental complexity required by the need for various pre-existing factors. I doubt if the program is really just 4K any more, and there are such things as shared libraries. The Asian width table is not very big by itself, especially if you use runs of characters rather than individual characters and do a binary search. See <http://unicode.org/emoji/charts/full-emoji-list.html> (slow to load) and examine the fourth column ("Chart") for rows 1063-65. Basically, GREEN BOOK has vertical stripes on the cover, BLUE BOOK has horizontal stripes, and ORANGE BOOK is black with white dots. 1-1 convertibility with various Japanese character sets. Unicode is not Cleanicode: it was designed not to do the best possible job, but the best job possible under the circumstances. -- John Cowan http://www.ccil.org/~cowan cowan(a)ccil.org "Any legal document draws most of its meaning from context. A telegram that says 'SELL HUNDRED THOUSAND SHARES IBM SHORT' (only 190 bits in 5-bit Baudot code plus appropriate headers) is as good a legal document as any, even sans digital signature." --me

On Sun, Mar 27, 2016, at 19:30, John Cowan wrote: Sure it does, but replace that != " " with !isblank(*c), and it doesn't work anymore since it ignores multibyte characters. Often you don't care, but you've got to remember to set LC_ALL=C when running grep etc on large data sets or it will be much slower, since \w and \s care about multibyte characters (as does case-insensitive matching, etc).

On Sun, Mar 27, 2016, at 21:58, John Cowan wrote: See, no, that's a trick question. None of the other blank class characters are single-byte, so of course isblank doesn't. The following characters return true on is*w*blank for me: U+00a0 U+1680 U+2000 U+2001 U+2002 U+2003 U+2004 U+2005 U+2006 U+2007 U+2008 U+2009 U+200a U+200b U+202f U+205f U+3000 (Oddly enough, isblank(0xA0) is true even in the UTF-8 locale, though of course U+00a0 is actually a multibyte character "\xc2\xa0".) So, if what you _want_ is to find the next blank character, doing this loop with isblank won't work. If what you want is to find space or tab, sure. But that's why grep for patterns containing \s are so slow.