Slides for a talk about the history and representation of strings and characters in code and communications and where things can go wrong.

1. A test of character
ASCII silly question get a silly ANSI

15. ASCII What’s wrong with it?
• Actually called US-ASCII
• [a-zA-Z0-9] + …
• It uses 7 bits not 8 bits
• Defining 128 ‘characters’
• Not defining the other 128
• Just about covers English script
• Minimum level of interoperability

16. CJK and font encoding for other languages
• Upper 128
• Completely different often 2-byte encoding
• Standards? Often not

17. Wingdings, Tengwar, etc.,
Encoding for symbols fictional languages and
silly things.

19. J and L

20. Email with
‘J’ and ‘L’

22. Quotation Marks and Apostrophes
• Dumb quotation marks "…" x22 (34th)
• Dumb single quotation marks '…' x27 (39th)
• Smart quotation marks “…” u201c…u201d
• Guillemets français «…»
• Grave Accent…apostrophe `…'
• Grave Accent…Acute Accent `...´ x60…u00B4
• Heavy double turned quotation mark ornaments ❝…❞
u275D…u275E

24. Unicode
1,114,112 possible characters
Evolving standard

25. “Face with a look of triumph”
😤
• Wait what?

26. Encoding
• Multibyte
• 1 byte space for 256 things
• 2 bytes space for 65536 things
• 3 bytes 16,777,216
• 4 bytes 4,294,967,296
• UTF encodings
• UTF-16 – two bytes
• UTF-32 – four bytes
• Variable size
• UTF-8 can encode lots of things

27. UTF-8
• 00000000 – 0111111 (just like ASCII)
• 11000000,10000000 – 11011111,11111111
• U+0080 – U+07FF
• 11100000,10000000,10000000 – 11010000,11111111,11111111
• U+0800 – U+FFFF
• 11110000,10000000,10000000,10000000 – …
• Bigger code points
• Now we can encode things and not break (old stuff) {much}

29. The good stuff
• Good stuff
• Room for everything
• CJK fits
• Hieroglyphs fit
• Emojis fit

31. http://www.itchyfeetcomic.com/2013/12/creative-guesswork.html
(With
permission,
unblurred
in
original)

32. The Bad stuff 1
• Lots of ways to represent the same thing
• Strange control characters

33. Hiding code

34. Names names names
• Homoglyphs
• You’ve seen www.micros0ft.com
• Punycode
• www.géant.org -> www.xn--gant-bpa.org
• Inserting into existing software …

36. The Bad stuff 2
• Emojis fit
•💩”pile of poo”
•😤”face with look of triumph”
• face-with-open-mouth-vomiting,Really?

39. http://www.asciimation.co.nz/index.php