Lesson 14. The Standard Library

Mission link: Every dependency is a decision someone will have to maintain. A large share of small dependencies exist because the person reaching for one did not know what already shipped with the interpreter.
Primary source: The Python Standard Library
Prerequisites: Lesson 8, Lesson 12, Lesson 13

Warm-up

1. ▢ Lesson 12 asked which shape a bundle of data deserves. Which shape holds a fixed, closed set of named values, such as three order states?

2. ▢ What does print(0.1 + 0.2) show, and what does that rule out?

Know this

The thesis is one sentence: search the standard library before adding a dependency, and before writing a loop. What follows is the recognition set, not a reference. The point is to know a module exists so you can look it up.

pathlib

from pathlib import Path

config = Path("~/.config/app.toml").expanduser()
if config.is_file():
    text = config.read_text(encoding="utf-8")

for path in Path("logs").rglob("*.log"):
    print(path.stem, path.stat().st_size)

out = Path("build") / "report" / "index.html"
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(html, encoding="utf-8")

/ joins, read_text and write_text remove the two-line with for the common case, and rglob replaces os.walk. Always pass encoding, because the default is platform-dependent and a file written on one machine is then unreadable on another.

collections

from collections import defaultdict, Counter, deque

by_country = defaultdict(list)
for order in orders:
    by_country[order.country].append(order)     # no key check

counts = Counter(word.lower() for word in words)
counts.most_common(3)

recent = deque(maxlen=100)                      # drops the oldest, no bookkeeping

One trap: reading a missing key from a defaultdict creates it. if "FR" in by_country is safe; if by_country["FR"] adds an empty list and grows the dict during a read. deque matters because list.pop(0) shifts every remaining element, so a queue built on a list is quadratic and a deque is not.

functools

from functools import cache, cached_property, partial, wraps

@cache                                # unbounded memoisation, since 3.9
def parse_schema(path):
    return json.loads(Path(path).read_text(encoding="utf-8"))

cache is lru_cache(maxsize=None) under a clearer name. Three things to know before using it: arguments must be hashable, nothing is ever evicted, and parse_schema.cache_clear() exists and is what tests need. Caching a method caches self too, which keeps every instance alive forever; cached_property is the per-instance version.

partial(func, timeout=5) builds a callable with an argument pre-supplied, which is the readable alternative to a lambda in a callback. @wraps on a decorator copies the wrapped function's name and docstring, without which tracebacks and help output name your wrapper instead.

enum

from enum import Enum, StrEnum, auto

class State(Enum):
    PENDING = auto()
    SHIPPED = auto()
    CANCELLED = auto()

if order.state is State.SHIPPED:      # identity, and a typo is an AttributeError
    ...

Three string constants let state == "shiped" be silently false forever. An Enum makes that an AttributeError, gives the set a name, makes it iterable and countable, and prints as State.SHIPPED in a traceback. Compare with is, since members are singletons.

StrEnum, from Python 3.11, makes members also be real strings, which is what a value stored in a database or sent as JSON usually needs. Without it, use Enum with explicit string values and convert at the boundary with State("shipped").

datetime and zoneinfo

from datetime import datetime, timezone
from zoneinfo import ZoneInfo

now = datetime.now(timezone.utc)                     # aware
local = now.astimezone(ZoneInfo("Europe/Berlin"))    # aware, correct across DST

A datetime with no tzinfo is naive: it names a wall-clock reading with no fact about which clock. Naive and aware objects cannot be compared or subtracted, and the moment two machines are involved, naive values are guesses. Two rules cover most production code: store and compute in UTC, convert to a zone only for display. datetime.utcnow() is deprecated since 3.12 precisely because it returns a naive value that looks like UTC; datetime.now(timezone.utc) is the replacement. zoneinfo, from Python 3.9, reads the system tz database, so daylight-saving transitions are handled instead of approximated by a fixed offset.

logging, in the four lines that matter

import logging
log = logging.getLogger(__name__)     # one logger per module, named by lesson 13

log.info("imported %s orders from %s", count, path)   # lazy formatting
log.exception("import failed")                        # inside an except: adds the traceback

Pass the values as arguments rather than formatting them in, so the string is only built if that level is enabled. Do not call basicConfig in library code: configuring handlers is the application's job, and a library that does it fights whatever the application chose.

subprocess

result = subprocess.run(
    ["git", "rev-parse", "HEAD"],
    capture_output=True, text=True, check=True,
)
commit = result.stdout.strip()

A list of arguments needs no quoting and cannot be injected into. shell=True with an interpolated string is the injection, and check=True is what makes a failure raise instead of being ignored.

How to search

docs.python.org has a module index, and the library reference is organised by problem, not alphabetically. Before writing more than about ten lines of a general-purpose utility, read the section heading that covers it. csv, sqlite3, statistics, textwrap, difflib, shutil, uuid, hashlib, hmac, ipaddress, urllib.parse, http.server and tomllib are all in there, and each one is a dependency someone else installed.

Practice

1. ▢ Rewrite with pathlib.

   import os
   base = os.path.dirname(os.path.abspath(__file__))
   target = os.path.join(base, "data", "out.json")
   os.makedirs(os.path.dirname(target), exist_ok=True)
   with open(target, "w") as f:
       f.write(payload)
   

from pathlib import Path
target = Path(__file__).resolve().parent / "data" / "out.json"
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(payload, encoding="utf-8")

2. ▢ Find the bug.

   from collections import defaultdict
   
   totals = defaultdict(int)
   for order in orders:
       totals[order.country] += order.amount
   
   if totals["FR"]:
       print("France:", totals["FR"])
   print(len(totals), "countries")
   

3. ▢ For each pair, say which is right and why.

   • a) random.choice(alphabet) or secrets.choice(alphabet) for a password-reset token
   • b) float("19.99") or Decimal("19.99") for a price
   • c) datetime.now() or datetime.now(timezone.utc) for a record's creation time
   • d) list.pop(0) or deque.popleft() for a work queue

4. ▢ Replace the string constants, and say what the new version catches.

   STATUS_PENDING = "pending"
   STATUS_SHIPPED = "shipped"
   
   def advance(order):
       if order.status == STATUS_PENDING:
           order.status = STATUS_SHIPPED
   

from enum import StrEnum

class Status(StrEnum):
    PENDING = "pending"
    SHIPPED = "shipped"

def advance(order):
    if order.status is Status.PENDING:
        order.status = Status.SHIPPED

5. ▢ Why is this logging call worse than it looks, and what does it do inside an except block?

   log.error("failed to import {} orders: {}".format(count, exc))
   

6. ▢ A colleague adds a dependency to flatten nested lists, another to retry a function, and another to parse an ISO timestamp. For each, name what could replace it.

Real-world reps

• [ ] Open the module index and read the names. Not the pages: the names. Twenty minutes, once, and it changes what you reach for.
• [ ] Take your project's dependency list and, for each small one, spend two minutes finding what in the standard library overlaps it. Keep the ones that survive.
• [ ] Convert one file that uses os.path to pathlib, and add the encoding argument everywhere a file is opened.
• [ ] Replace one group of related string constants with an Enum and let the type checker or the tests tell you where the comparisons were wrong.
• [ ] Tomorrow: find a naive datetime in code you own and follow it to where it is compared or stored. Decide whether anyone knows which clock it read.

Going further

• The Python Standard Library: organised by problem, which is how to search it
• pathlib, collections, functools, enum: the four with the highest ratio of code deleted to page length
• logging HOWTO: levels, handlers, and why a library configures nothing
• zoneinfo: the system time-zone database, and what an aware datetime buys
• subprocess: the security notes on shell=True are the part to read
• Resources

Not landing? Reread the primary source at the top, since this lesson compresses it and compression is where understanding leaks. Check the glossary for any term that felt slippery.

If the lesson itself is unclear rather than the material, that is a defect: open an issue.