Unicode and Amharic text

One letter, one number

Predict the length of Selam written in Ge'ez letters.

print(len("ሰላም"))

3

If you said 6, you split each fidel into its consonant and vowel (ሰ is s + ä). If you said 9, you'd heard that Amharic takes more space. It does, but not in characters; that's the next section.

Almost every script people write today, Ge'ez included, is part of one standard, UnicodeUnicodeThe world standard that gives each character of almost every script in use (Ge'ez, Latin, Arabic, emoji, and more) its own number., which gives each character its own number, its code pointcode pointThe number Unicode gives a character. ord() gives it; chr() turns it back into the character. Written U+1230 in hex.. To Python, a charactercharacterTo Python, one code point: usually one letter, digit, space, or symbol. A string is a sequence of them. is one code point, and a string is a sequence of code points. ord() gives a character's code point; chr() goes the other way.

print(ord("ሰ"))
print(hex(ord("ሰ")))
print(chr(0x1200))
start = 0x1230
print("".join([chr(n) for n in range(start, start + 7)]))

Unicode charts write code points in hexadecimal (base 16), so ሰ is U+1230. hex() shows a number in hex, and in code you type a hex number with 0x in front: 0x1230 is the same number as 4656. The Ge'ez letters (Unicode calls them Ethiopic) start at U+1200 with ሀ. The last line prints the seven code points from U+1230: the ሰ family, in fidel chart order. Change start to 0x1208 for the ለ family.

Characters are not bytes

A computer saves and sends text as bytes, not characters. Predict:

print(len("ሰላም".encode()))

9

A bytebyteThe unit computers save and send data in: a number from 0 to 255. is a number from 0 to 255, the unit of files, memory, and networks. Turning characters into bytes needs a rule, an encodingencodingA rule for turning characters into bytes and back again. UTF-8 is the one almost everything uses.. The one almost everything uses is UTF-8UTF-8The most common encoding. English letters and digits take 1 byte each; Ge'ez letters take 3.: English letters take 1 byte each, Ge'ez letters take 3. .encode() gives the bytes (in UTF-8 unless you name another encoding), and .decode() turns them back into a string.

data = "ሰላም".encode()
print(data)
print(len("Selam".encode()))
print(data.decode())

The b means bytes. Each \x and the two hex digits after it are one byte, so \xe1\x88\xb0 is ሰ. What matters: a limit counted in bytes, like a 30-byte database field, fits 30 English letters but only 10 Ge'ez letters, whatever len() says about the string.

Amharic sentences and order

Amharic ends a sentence with ። (አራት ነጥብ, four dots), not .. They're different characters, so split on ።. Predict the result:

print("ሰላም ነው። ቡና ደርሷል።".split("።"))

['ሰላም ነው', ' ቡና ደርሷል', '']

If you picked the tidy list, you expected split to clean up. It only cuts. Strip each piece and keep only the ones that aren't empty (an empty string is falsy, from lesson 2.4), with a comprehension from lesson 4.7:

text = "ሰላም ነው። ቡና ደርሷል።"
sentences = [s.strip() for s in text.split("።") if s.strip()]
print(sentences)
print(len(sentences))
print(text.split("."))

The last line splits on . and finds nothing to cut.

sorted() compares strings code point by code point. The Ethiopic block is laid out in the traditional fidel order, so ሀ comes before ለ, and ለ before መ:

words = ["መጽሐፍ", "ለምለም", "ሀገር"]
print(sorted(words))

That's all code point order promises. It isn't a full dictionary sort, so don't rely on it where the exact order matters.

Ge'ez numerals

Ge'ez has its own numerals: ፩ ፪ ፫ for 1, 2, 3. In lesson 3.5 you checked isdigit() before calling int(). Predict:

print("፩".isdigit())

True

So isdigit() says yes. But int() reads only decimal digits: 0 to 9, and digits from scripts that write numbers the same place-value way, like Arabic-Indic ٣. Ge'ez numerals don't work that way, so ፩ isn't one:

GEEZ is a constant (the capitals say so), and you use it like any other dict:

GEEZ = {"፩": 1, "፪": 2, "፫": 3, "፬": 4, "፭": 5, "፮": 6, "፯": 7, "፰": 8, "፱": 9}
print(GEEZ["፫"] + GEEZ["፬"])

This covers one to nine. Ge'ez has no zero: tens and hundreds have their own signs (፲ is ten, ፳ twenty, ፻ a hundred), so bigger numbers need more than a lookup.

When one letter is two code points

In other scripts, one letter on screen can be more than one code point. é can be one code point, or e followed by an accent code point that joins onto it. Inside a string, \u and four hex digits write a character by its code point:

import unicodedata
one = "\u00e9"
two = "e\u0301"
print(len(one), len(two))
print(one == two)
print(unicodedata.normalize("NFC", two) == one)
print(len("👍🏽"))

Both look like é, but == compares code points. unicodedata.normalize("NFC", ...) rewrites text into one standard form so they compare equal. A thumbs-up with a skin tone is 2 code points too. Just remember: len() counts code points, not what the eye sees.