go-utf8/doc.go

94 lines
10 KiB
Go
Raw Normal View History

2018-07-02 06:06:22 +00:00
/*
Package utf8s provides tools for working with Unicode encoded as UTF-8.
Example
var reader io.Reader
// ...
r, n, err := utf8s.ReadRune(reader)
Example
var writer io.Writer
// ...
var r rune
// ...
n, err := utf8s.WriteRune(w, r)
Example
var r rune
// ...
s := utfs.FormatBinary(r)
fmt.Printf("%q encoded in UTF-8 as binary is %s \n", r, s)
UTF-8
UTF-8 is a variable length encoding of Unicode. An encoding of a single Unicode code point can be from 1 to 4 bytes longs.
Some examples of UTF-8 encoding of Unicode code points are:
UTF-8 encoding
byte 1 byte 2 byte 3 byte 4 value code point decimal binary name
0b0,1000001 A U+0041 65 0b0000,0000,0100,0001 LATIN CAPITAL LETTER A
0b0,1110010 r U+0072 114 0b0000,0000,0111,0010 LATIN SMALL LETTER R
0b110,00010 0b10,100001 ¡ U+00A1 161 0b0000,0000,1010,0001 INVERTED EXCLAMATION MARK
0b110,11011 0b10,110101 ۵ U+06F5 1781 0b0000,0110,1111,0101 EXTENDED ARABIC-INDIC DIGIT FIVE
0b1110,0010 0b10,000000 0b10,110001 U+2031 8241 0b0010,0000,0011,0001 PER TEN THOUSAND SIGN
0b1110,0010 0b10,001001 0b10,100001 U+2261 8801 0b0010,0010,0110,0001 IDENTICAL TO
0b11110,000 0b10,010000 0b10,001111 0b10,010101 𐏕 U+000103D5 66517 0b0001,0000,0011,1101,0101 OLD PERSIAN NUMBER HUNDRED
0b11110,000 0b10,011111 0b10,011001 0b10,000010 🙂 U+0001F642 128578 0b0001,1111,0110,0100,0010 SLIGHTLY SMILING FACE
UTF-8 Versus ASCII
UTF-8 was (partially) designed to be backwards compatible with 7-bit ASCII.
Thus, all 7-bit ASCII is valid UTF-8.
UTF-8 Encoding
Since, at least as of 2003, Unicode fit into 21 bits, UTF-8 was designed to support at most 21 bits of information.
This is done as described in the following table:
# of bytes # bits for code point 1st code point last code point byte 1 byte 2 byte 3 byte 4
1 7 U+000000 U+00007F 0xxxxxxx
2 11 U+000080 U+0007FF 110xxxxx 10xxxxxx
3 16 U+000800 U+00FFFF 1110xxxx 10xxxxxx 10xxxxxx
4 21 U+010000 U+10FFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
*/
package utf8s