← All handbooks

Python Handbook

Python end to end — the language core, the things that trip people up (mutability, closures, the GIL), and the data stack (NumPy, pandas) plus the data-engineering toolkit you actually ship with.

01 Types & data structures

Everything is an object. Variables are names bound to objects, not boxes. Assignment never copies.

int, float, complex, bool, str, bytes, None
x = 10; y = x        # both names -> same int object

nums = [1,2,3]; nums.append(4); nums[-1]       # 4
d = {"a":1}; d.get("b", 0)                    # 0, no KeyError
s = {1,2} | {2,3}                            # {1,2,3}
a, *rest = [1,2,3,4]                          # a=1, rest=[2,3,4]

02 Control flow & comprehensions

for i, v in enumerate(items):
    if v > 0: ...
    elif v == 0: continue
    else: break

match cmd:                       # structural pattern matching (3.10+)
    case {"type": "sql", "q": q}: run(q)
    case _: pass

# comprehensions — the pythonic transform
squares = [n*n for n in nums if n%2==0]
by_id   = {u.id: u for u in users}
uniq    = {w.lower() for w in words}
gen     = (n*n for n in nums)      # lazy generator, no list built

03 Functions

def load(path, *parts, sep=",", **opts):
    # *parts -> tuple of extra positionals; **opts -> dict of extra kwargs
    return ...

load("a.csv", "b.csv", sep="|", encoding="utf8")
total = lambda a, b: a + b            # small anonymous fn

• Args bind by position then keyword. Use * to force keyword-only: def f(a, *, b); / to force positional-only.
• Functions are first-class: pass them, return them, store in dicts (dispatch tables).
• Type-hint the signature; it documents and enables tooling.

04 OOP & dataclasses

from dataclasses import dataclass

@dataclass
class Order:
    id: int
    amount: float
    status: str = "pending"
    def with_tax(self, rate=0.1) -> float:
        return self.amount * (1+rate)

• @dataclass auto-generates __init__, __repr__, __eq__. Use frozen=True for immutable value objects.
• Dunder methods customize behavior: __len__, __iter__, __getitem__, __enter__/__exit__, __call__.
• @property exposes a method as an attribute; @classmethod/@staticmethod for alt constructors/utilities.
• Prefer composition over deep inheritance. Use ABC/Protocol for interfaces.

05 Errors & context managers

try:
    data = json.loads(raw)
except json.JSONDecodeError as e:
    log.warning("bad json: %s", e); data = {}
else:
    process(data)                  # ran only if no exception
finally:
    cleanup()                      # always runs

with open(path) as f:           # context manager: auto-close
    text = f.read()

• Catch specific exceptions, never bare except:. Raise with context: raise ValueError(...) from e.
• EAFP (“easier to ask forgiveness”) is pythonic: try the op, handle the exception — vs LBYL pre-checks.
• Build your own context manager with @contextmanager (yield) for setup/teardown (timers, DB sessions, temp dirs).

06 Iterators & generators

def read_batches(rows, n):
    batch = []
    for r in rows:
        batch.append(r)
        if len(batch) == n:
            yield batch          # pause & emit; resumes here next call
            batch = []
    if batch: yield batch

• Generators are lazy — they stream values one at a time, so you can process data larger than memory.
• itertools: chain, islice, groupby, product, count, accumulate. yield from delegates to a sub-generator.

07 Decorators & closures

import functools, time
def timed(fn):
    @functools.wraps(fn)
    def wrap(*a, **k):
        t=time.perf_counter(); r=fn(*a,**k)
        log.info("%s %.3fs", fn.__name__, time.perf_counter()-t)
        return r
    return wrap

@timed
def etl(): ...

A closure captures variables from its enclosing scope. A decorator is a function that wraps another to add behavior (logging, caching via @functools.lru_cache, retries, auth). Use @wraps to preserve the wrapped function’s metadata.

08 Type hints

from typing import Optional, Iterable
def total(rows: Iterable[dict], key: str) -> float:
    return sum(r[key] for r in rows)

price: float | None = None   # 3.10+ union syntax

Hints are optional and not enforced at runtime, but power editors, refactors, and mypy/pyright static checks. Hint public function signatures and dataclasses; skip obvious locals. Pydantic uses them for validation.

09 Modules, venv & pip

# isolate every project
python -m venv .venv && source .venv/bin/activate
pip install pandas numpy
pip freeze > requirements.txt        # or use uv / poetry / pip-tools

• A module is a .py file; a package is a folder. Import with from pkg.mod import thing.
• if __name__ == "__main__": guards script entry so imports don’t run it.
• Never install into system Python — one virtual env per project; pin versions for reproducible builds.

10 Concurrency & the GIL

import asyncio, aiohttp
async def fetch(s, u):
    async with s.get(u) as r: return await r.json()
async def main(urls):
    async with aiohttp.ClientSession() as s:
        return await asyncio.gather(*[fetch(s,u) for u in urls])

The GIL (Global Interpreter Lock) lets only one thread run Python bytecode at a time — so threads don’t speed up CPU work. NumPy/pandas release the GIL in C, so they parallelize fine. (3.13 ships an experimental free-threaded build.)

11 Standard library highlights

• collections: defaultdict, Counter, deque, namedtuple.
• itertools, functools (reduce, lru_cache, partial).
• pathlib.Path for filesystem; datetime/zoneinfo for time; json, csv, re, logging, subprocess, os/sys.

from collections import Counter
Counter("banana").most_common(2)    # [('a',3),('n',2)]
from pathlib import Path
for p in Path("data").glob("*.csv"): ...

12 NumPy

import numpy as np
a = np.array([[1,2],[3,4]])
a.shape, a.dtype                 # (2,2), int64
a * 2                            # vectorized, no loop
a[a > 2]                         # boolean mask
a.sum(axis=0); a.mean()

NumPy gives a typed, contiguous ndarray with vectorized C operations — orders of magnitude faster than Python loops. Key ideas: broadcasting (shape-compatible elementwise ops), views vs copies (slices are views), and axis-wise reductions. It’s the substrate under pandas, scikit-learn, and PyTorch.

13 pandas core

import pandas as pd
df = pd.read_csv("sales.csv", parse_dates=["date"])
df.head(); df.info(); df.describe()
df["amount"]                     # Series (a column)
df[["date","amount"]]            # DataFrame (cols)
df.loc[df.amount > 100, "region"]  # label-based + mask
df.iloc[0:5]                    # positional

• Series = 1-D labeled array; DataFrame = 2-D table of Series sharing an index.
• Select with .loc (labels) / .iloc (positions). Filter with boolean masks.
• Set proper dtypes (parse_dates, category) — memory + speed.

14 pandas transform

# group-aggregate (split-apply-combine)
g = (df.groupby("region")
       .agg(rev=("amount","sum"), n=("id","count"))
       .reset_index())

# join, reshape, time
m = df.merge(cust, on="customer_id", how="left")
pivot = df.pivot_table("amount", "product", "region", aggfunc="sum")
df.assign(month=df.date.dt.to_period("M"))
df.set_index("date").resample("D").sum()
df["roll7"] = df.amount.rolling(7).mean()

15 Data-engineering Python

• Extract: requests/httpx for APIs (paginate, retry, backoff), SQLAlchemy for DBs.
• Files/formats: pyarrow + Parquet (columnar, compressed) over CSV; fsspec/s3fs for cloud storage.
• Transform at scale: Polars (Rust, lazy), DuckDB (SQL on files), PySpark.
• Validate: Pydantic / Great Expectations / pandera for schema + quality checks.
• Orchestrate: Airflow / Dagster / Prefect DAGs; make tasks idempotent and parameterized by run date.

from sqlalchemy import create_engine
eng = create_engine("postgresql+psycopg://user:pw@host/db")
df = pd.read_sql("select * from orders where day = %(d)s", eng, params={"d": run_date})
df.to_parquet(f"s3://lake/orders/{run_date}.parquet")

16 Testing & logging

# test_etl.py
import pytest
def test_with_tax():
    assert Order(1, 100).with_tax(0.1) == 110

@pytest.mark.parametrize("a,b,exp", [(1,1,2),(2,3,5)])
def test_add(a,b,exp): assert a+b == exp

• pytest: plain assert, fixtures for setup, parametrize for tables of cases, monkeypatch/mock for boundaries.
• logging not print: configure once, log structured messages with levels; never log secrets.

17 Performance & idioms

• Profile before optimizing: cProfile, timeit, line profilers.
• Vectorize (NumPy/pandas); prefer comprehensions/built-ins (sum, map) over manual loops; use generators to stream.
• Cache pure functions (lru_cache); use set/dict for O(1) membership; batch I/O and DB calls.
• PEP 8 + ruff/black for style; small pure functions; explicit over clever.

18 Interview drills

• list vs tuple vs set vs dict — mutability, ordering, use.
• Mutable default argument bug; is vs ==; shallow vs deep copy.
• What is the GIL; threads vs processes vs asyncio; when each.
• Generators vs lists; what yield does; lazy evaluation.
• Decorators — write a timing/retry decorator with @wraps.
• pandas: vectorize a row loop; groupby.agg; merge types; why apply is slow.