go-utf8/README.md

241 lines
8.7 KiB
Markdown
Raw Normal View History

2022-07-18 23:30:32 +00:00
# go-utf8
2018-07-02 06:04:04 +00:00
2022-07-19 04:19:50 +00:00
Package **utf8** implements encoding and decoding of UTF-8, for the Go programming language.
2018-07-02 06:04:04 +00:00
2022-07-19 04:19:50 +00:00
This package is meant to be a replacement for Go's built-in `"unicode/utf8"` package.
2018-07-02 06:04:04 +00:00
## Documention
2023-08-16 12:23:58 +00:00
Online documentation, which includes examples, can be found at: http://godoc.org/sourcecode.social/reiver/go-utf8
2018-07-02 06:04:04 +00:00
2023-08-16 12:23:58 +00:00
[![GoDoc](https://godoc.org/sourcecode.social/reiver/go-utf8?status.svg)](https://godoc.org/sourcecode.social/reiver/go-utf8)
2018-07-02 06:04:04 +00:00
2022-07-19 04:19:50 +00:00
## Reading a Single UTF-8 Character
2018-07-02 06:04:04 +00:00
2022-07-19 04:19:50 +00:00
This is the simplest way of reading a single UTF-8 character.
2018-07-02 06:04:04 +00:00
```go
var reader io.Reader
// ...
2022-07-18 23:36:02 +00:00
r, n, err := utf8.ReadRune(reader)
2018-07-02 06:04:04 +00:00
```
2022-07-19 04:19:50 +00:00
## Write a Single UTF-8 Character
This is the simplest way of writing a single UTF-8 character.
2018-07-02 06:04:04 +00:00
```go
var writer io.Writer
// ...
var r rune
// ...
2022-07-18 23:36:02 +00:00
n, err := utf8.WriteRune(w, r)
2018-07-02 06:04:04 +00:00
```
2022-07-19 04:19:50 +00:00
## io.RuneReader
This is how you can create an `io.RuneReader`:
2018-07-11 03:43:45 +00:00
```go
var reader io.Reader
// ...
2022-07-19 04:19:50 +00:00
var runeReader io.RuneReader = utf8.RuneReaderWrap(reader)
2018-07-11 03:43:45 +00:00
// ...
r, n, err := runeReader.ReadRune()
```
2022-07-19 04:19:50 +00:00
## io.RuneScanner
This is how you can create an `io.RuneScanner`:
2018-07-11 04:01:30 +00:00
```go
var reader io.Reader
// ...
2022-07-19 04:19:50 +00:00
var runeScanner io.RuneScanner := utf8.RuneScannerWrap(reader)
2018-07-11 04:01:30 +00:00
// ...
r, n, err := runeScanner.ReadRune()
// ...
err = runeScanner.UnreadRune()
```
2022-07-19 04:19:50 +00:00
## UTF-8
UTF-8 is a variable length encoding of Unicode.
An encoding of a single Unicode code point can be from 1 to 4 bytes longs.
Some examples of UTF-8 encoding of Unicode code points are:
2023-08-16 12:36:21 +00:00
<table>
<thead>
<tr>
<td colspan="4">UTF-8 encoding</td>
<td rowspan="2">value</td>
<td rowspan="2">code point</td>
<td rowspan="2">decimal</td>
<td rowspan="2">binary</td>
<td rowspan="2">name</td>
</tr>
<tr>
<td>byte 1</td>
<td>byte 2</td>
<td>byte 3</td>
<td>byte 4</td>
</tr>
</thead>
<tbody>
2023-08-16 12:39:41 +00:00
<tr>
<td><code>0b0,1000001</code></td>
<td><code></code></td>
<td><code></code></td>
<td><code></code></td>
<td></td>
<td></td>
<td></td>
<td><code></code></td>
<td></td>
</tr>
<tr>
<td><code>0b0,1110010</code></td>
<td><code></code></td>
<td><code></code></td>
<td><code></code></td>
<td></td>
<td></td>
<td></td>
<td><code></code></td>
<td></td>
</tr>
<tr>
<td><code>0b110,00010</code></td>
<td><code>0b10,100001</code></td>
<td><code></code></td>
<td><code></code></td>
<td></td>
<td></td>
<td></td>
<td><code></code></td>
<td></td>
</tr>
<tr>
<td><code>0b110,11011</code></td>
<td><code>0b10,110101</code></td>
<td><code></code></td>
<td><code></code></td>
<td></td>
<td></td>
<td></td>
<td><code></code></td>
<td></td>
</tr>
<tr>
<td><code></code></td>
<td><code></code></td>
<td><code></code></td>
<td><code></code></td>
<td></td>
<td></td>
<td></td>
<td><code></code></td>
<td></td>
</tr>
<tr>
<td><code></code></td>
<td><code></code></td>
<td><code></code></td>
<td><code></code></td>
<td></td>
<td></td>
<td></td>
<td><code></code></td>
<td></td>
</tr>
<tr>
<td><code></code></td>
<td><code></code></td>
<td><code></code></td>
<td><code></code></td>
<td></td>
<td></td>
<td></td>
<td><code></code></td>
<td></td>
</tr>
<tr>
<td><code></code></td>
<td><code></code></td>
<td><code></code></td>
<td><code></code></td>
<td></td>
<td></td>
<td></td>
<td><code></code></td>
<td></td>
</tr>
2023-08-16 12:36:21 +00:00
</tbody>
</table>
2022-07-19 04:19:50 +00:00
```
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
2023-08-16 12:28:52 +00:00
| UTF-8 encoding | | | | | |
┣━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┫ | | | | |
| byte 1 ┋ byte 2 ┋ byte 3 ┋ byte 4 | value | code point | decimal | binary | name |
2022-07-19 04:19:50 +00:00
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 0b0,1000001 ┊ ┊ ┊ │ A │ U+0041 │ 65 │ 0b0000,0000,0100,0001 │ LATIN CAPITAL LETTER A │
2023-08-16 12:28:52 +00:00
├-------------┼-------------┼-------------┼-------------┼-------┼------------┼---------┼----------------------------┼----------------------------------┤
2022-07-19 04:19:50 +00:00
│ 0b0,1110010 ┊ ┊ ┊ │ r │ U+0072 │ 114 │ 0b0000,0000,0111,0010 │ LATIN SMALL LETTER R │
2023-08-16 12:28:52 +00:00
├-------------┼-------------┼-------------┼-------------┼-------┼------------┼---------┼----------------------------┼----------------------------------┤
2022-07-19 04:19:50 +00:00
│ 0b110,00010 ┊ 0b10,100001 ┊ ┊ │ ¡ │ U+00A1 │ 161 │ 0b0000,0000,1010,0001 │ INVERTED EXCLAMATION MARK │
2023-08-16 12:28:52 +00:00
├-------------┼-------------┼-------------┼-------------┼-------┼------------┼---------┼----------------------------┼----------------------------------┤
2022-07-19 04:19:50 +00:00
│ 0b110,11011 ┊ 0b10,110101 ┊ ┊ │ ۵ │ U+06F5 │ 1781 │ 0b0000,0110,1111,0101 │ EXTENDED ARABIC-INDIC DIGIT FIVE │
2023-08-16 12:28:52 +00:00
├-------------┼-------------┼-------------┼-------------┼-------┼------------┼---------┼----------------------------┼----------------------------------┤
2022-07-19 04:19:50 +00:00
│ 0b1110,0010 ┊ 0b10,000000 ┊ 0b10,110001 ┊ │ ‱ │ U+2031 │ 8241 │ 0b0010,0000,0011,0001 │ PER TEN THOUSAND SIGN │
2023-08-16 12:28:52 +00:00
├-------------┼-------------┼-------------┼-------------┼-------┼------------┼---------┼----------------------------┼----------------------------------┤
2022-07-19 04:19:50 +00:00
│ 0b1110,0010 ┊ 0b10,001001 ┊ 0b10,100001 ┊ │ ≡ │ U+2261 │ 8801 │ 0b0010,0010,0110,0001 │ IDENTICAL TO │
2023-08-16 12:28:52 +00:00
├-------------┼-------------┼-------------┼-------------┼-------┼------------┼---------┼----------------------------┼----------------------------------┤
2022-07-19 04:19:50 +00:00
│ 0b11110,000 ┊ 0b10,010000 ┊ 0b10,001111 ┊ 0b10,010101 │ 𐏕 │ U+000103D5 │ 66517 │ 0b0001,0000,0011,1101,0101 │ OLD PERSIAN NUMBER HUNDRED │
2023-08-16 12:28:52 +00:00
├-------------┼-------------┼-------------┼-------------┼-------┼------------┼---------┼----------------------------┼----------------------------------┤
2022-07-19 04:19:50 +00:00
│ 0b11110,000 ┊ 0b10,011111 ┊ 0b10,011001 ┊ 0b10,000010 │ 🙂 │ U+0001F642 │ 128578 │ 0b0001,1111,0110,0100,0010 │ SLIGHTLY SMILING FACE │
2023-08-16 12:28:52 +00:00
└-------------┴-------------┴-------------┴-------------┴-------┴------------┴---------┴----------------------------┴----------------------------------┘
2022-07-19 04:19:50 +00:00
```
## UTF-8 Versus ASCII
UTF-8 was (partially) designed to be backwards compatible with 7-bit ASCII.
Thus, all 7-bit ASCII is valid UTF-8.
## UTF-8 Encoding
Since, at least as of 2003, Unicode fits into 21 bits, and thus UTF-8 was designed to support at most 21 bits of information.
This is done as described in the following table:
2023-08-16 12:31:48 +00:00
| # of bytes | # bits for code point | 1st code point | last code point | byte 1 | byte 2 | byte 3 | byte 4 |
|------------|-----------------------|----------------|------------------|------------|------------|------------|------------|
| 1 | 7 | U+000000 | U+00007F | `0xxxxxxx` | | | |
| 2 | 11 | U+000080 | U+0007FF | `110xxxxx` | `10xxxxxx` | | |
| 3 | 16 | U+000800 | U+00FFFF | `1110xxxx` | `10xxxxxx` | `10xxxxxx` | |
| 4 | 21 | U+010000 | U+10FFFF | `11110xxx` | `10xxxxxx` | `10xxxxxx` | `10xxxxxx` |
2022-07-19 04:19:50 +00:00
```