Strings, bytes, encodings and formatting

In Python 3, str is a sequence of Unicode code points and bytes is a sequence of integers from 0 to 255. They never mix silently: "a" + b"a" is a TypeError. Text becomes bytes by encoding (text.encode("utf-8")) and bytes become text by decoding (data.decode("utf-8")) — and the encoding must be the same on both sides.

UTF-8 uses one byte for ASCII and two to four for everything else, so len(text) (code points) and len(text.encode()) (bytes) differ as soon as the text leaves ASCII. A database column or a protocol field limited to N bytes is not limited to N characters.

1def run():2    words = ["cafe", "café", "€5", "🙂"]3    sizes = {w: [len(w), len(w.encode("utf-8"))] for w in words}4    try:5        mixed = "a" + b"a"6    except TypeError as e:7        mixed = f"TypeError: {e}"8    return {"chars_and_bytes": sizes,9            "cafe_bytes": list("café".encode("utf-8")),10            "mixing": mixed}

When decoding goes wrong

Decoding UTF-8 bytes as Latin-1 never fails — every byte is a valid Latin-1 character — it just produces mojibake like café. Decoding bytes that are not valid UTF-8 as UTF-8 raises UnicodeDecodeError. The errors= argument chooses the policy: "strict" (the default) raises, "replace" substitutes U+FFFD (�), "ignore" drops the bad bytes. Pick one on purpose; "ignore" loses data silently.

The same applies to files: open(path, encoding="utf-8"). Without encoding=, Python uses the platform's preferred encoding, which is not UTF-8 everywhere, so the same code can read a file correctly on one machine and garble it on another.

Comparing text that looks the same

"é" can be stored as one code point (U+00E9) or as "e" followed by a combining accent (U+0301). They render identically and compare unequal. unicodedata.normalize("NFC", s) brings both to one form. For case-insensitive comparison, casefold() is the thorough version of lower(): it maps the German ß to ss, for instance.

1import unicodedata2 3 4def run():5    data = "café".encode("utf-8")6    broken = b"caf\xe9"  # Latin-1 bytes, not valid UTF-87    try:8        broken.decode("utf-8")9        strict = "decoded"10    except UnicodeDecodeError as e:11        strict = f"UnicodeDecodeError: {e}"12 13    composed = "é"         # é as one code point14    decomposed = "é"      # e + combining acute accent15    nfc = unicodedata.normalize("NFC", decomposed)16    return {17        "utf8_read_as_latin1": data.decode("latin-1"),18        "strict": strict,19        "replace": broken.decode("utf-8", errors="replace"),20        "ignore": broken.decode("utf-8", errors="ignore"),21        "lengths": [len(composed), len(decomposed)],22        "equal_raw": composed == decomposed,23        "equal_after_nfc": composed == nfc,24        "lower_matches": "Straße".lower() == "STRASSE".lower(),25        "casefold_matches": "Straße".casefold() == "STRASSE".casefold(),26    }

Formatting with f-strings

An f-string field is {expression!conversion:format_spec}. !r uses repr, and {x=} prints the expression's text as well as its value — handy for debugging. The spec after : controls layout:

1def run():2    price = 1234.53    ratio = 0.45674    name = "Ana"5    n = 2556    return [7        f"[{name:>6}] [{name:<6}] [{name:^7}]",8        f"{price:.2f} | {price:,.2f} | {price:010.1f}",9        f"{ratio:.1%}",10        f"{n:x} {n:b} {n:#06x}",11        f"{name!r} {name=}",12    ]

To build a long string from many pieces, collect them in a list and "".join(pieces) once, rather than growing a string with += in a loop, which can copy the string over and over.

Common mistakes

• Measuring a byte limit with len(text). Accented letters, most non-Latin scripts and emoji take more than one byte each.
• Slicing encoded bytes and decoding the slice. A cut through the middle of a multi-byte character makes the decode fail.
• Opening text files without encoding="utf-8" and getting different results on different machines.
• Comparing user-entered text without normalising it first.

What interviewers ask

• What is the difference between str and bytes? Code points against raw bytes; encode and decode convert, always with a named encoding.
• Why can len() of a string and of its UTF-8 encoding differ? UTF-8 is variable width: 1 byte for ASCII, up to 4 for other code points.
• What is mojibake, and how do you fix it? Bytes decoded with the wrong encoding; decode with the encoding they were written in, and declare encodings explicitly at every boundary.