ProjekinAja. Software Engineering Stage 2 · Craft Lesson 2.9
● Stage 2 · Lesson 2.9

Dependencies, Packaging, and the Supply Chain

Every dependency is a hire you can't interview — virtualenvs, lockfiles, semver's promises and lies, and why "pip install succeeded but import fails".

⏱ ~18 min read 📦 Packaging 🔐 Supply-chain risk

🎯 By the end you'll understand

How Python finds an interpreter and packages (PATH, sys.prefix, sys.path), why virtualenvs exist and what they physically do, how pyproject.toml and lockfiles encode intent vs. reality, what semver's social contract actually promises (and routinely breaks), how transitive dependencies multiply your attack surface, and how to make reproducibility a discipline rather than a prayer. You'll also wire TraceLab into a proper package with pinned, audited dependencies.

KEY INSIGHTEvery dependency is a hire you can't interview

When you write pip install requests, you are not fetching a library — you are hiring every maintainer, and every one of their maintainers, to run code inside your production process.

There's a build that lives rent-free in every senior engineer's memory: the one where the code worked on every developer's laptop, in CI, and in staging — and then silently broke in production. Not a logic error. Not a missing config. The dependency resolver picked a different transitive version in the production Docker image, and that version had a behavioral regression at a corner case your tests didn't cover. Nobody noticed for three weeks because the failure was quiet: the output was slightly wrong, not absent.

🔑 The reproducibility gap

Reproducibility is not the default state of Python projects — it is a discipline that requires deliberate tooling. pip install with loose constraints resolves to whatever is newest right now. Run it again in six months and you get something different. A lockfile captures the exact resolution; a virtualenv isolates it from the system. Without both, your "working" build is a snapshot of a moment in time that you cannot recover.

HOW IT WORKSInterpreters and PATH resolution

When you type python train.py in a shell, the shell does not know where Python is — it searches $PATH, a colon-separated list of directories checked left to right for an executable named python. (If this is unfamiliar, Lesson 1.4 covers the shell's lookup model in full.)

shelldiscovery
# which interpreter are you actually running?
$ which python3
/usr/bin/python3                  # system Python — danger zone

$ which python3     # after activating a venv
/home/user/tracelab/.venv/bin/python3   # isolated copy

$ python3 -c "import sys; print(sys.prefix)"
/home/user/tracelab/.venv         # venv root: this is where packages live

The interpreter you execute determines which packages you can import. Each interpreter has a sys.prefix — the root from which it looks for packages. On a fresh machine with no virtualenv, sys.prefix is /usr or wherever your system Python lives. Install packages there and you pollute the system Python for every other tool on the machine — including OS utilities that depend on specific versions of Python libraries. This is the root problem virtualenvs solve.

📖 Why "pip install succeeded but import fails"

The classic support question. You installed the package — why can't Python find it? Almost always: pip and the interpreter are different. pip install foo installs into wherever pip itself targets. If your shell resolves pip to one environment and python to another (a misconfigured PATH, or a pip from the system and a python from pyenv), you install into one prefix and import from a different one. The fix is always the same: use python -m pip install foo, which guarantees pip uses the same interpreter you're running under.

CORE CONCEPTVirtual environments: what they actually are

A virtual environment is not a container, a VM, or any kind of isolation at the OS level. It is a directory with a Python binary (or a symlink to one) and a site-packages/ folder, plus a handful of scripts that manipulate PATH. That's it.

.venv/ bin/python3 symlink → system Python lib/pythonX.Y/site-packages/ your packages live here pyvenv.cfg records home interpreter path source bin/activate $PATH prepended .venv/bin before everything else python → .venv/bin/python3 sees only .venv/site-packages deactivate reverses the PATH change — nothing else changes. The venv directory still exists; only your shell's PATH was modified.
A virtualenv is a directory and a PATH trick, not OS-level isolation. The Python binary is usually a symlink to the system interpreter — only site-packages is private to the environment.
shellcreate and activate
# standard idiom — works on Python 3.3+
$ python3 -m venv .venv
$ source .venv/bin/activate          # bash/zsh
(.venv) $ python -m pip install --upgrade pip
(.venv) $ python -m pip install numpy pandas

# commit requirements / pyproject.toml, NOT the .venv folder
$ echo ".venv/" >> .gitignore

The conventional name is .venv (leading dot keeps it hidden, and most tooling expects it). Never commit the .venv/ directory — it contains absolute paths and machine-specific compiled extensions. Commit the specification of what the venv should contain; anyone can recreate the directory from that specification.

💡 uv: the modern alternative

uv (by Astral, 2024) is a Rust-based drop-in for pip + venv that is dramatically faster and handles lockfiles natively. uv venv, uv pip install, uv lock — same conceptual model, better implementation. For new projects in 2025+, it is the pragmatic default. The concepts in this lesson apply to both; the incantations differ slightly.

IN PRACTICEpyproject.toml and the two-layer model

Python packaging has had an ugly history of competing standards: setup.py, setup.cfg, requirements.txt. PEP 518 (2016) and PEP 621 (2020) converged on pyproject.toml as the single source of truth. Here's the model that matters:

1

pyproject.toml — the intent

Declares what your project needs, with intentionally loose constraints. requests>=2.28 means "any version 2.28 or newer is fine." This file is committed and human-authored. It is for expressing policy, not locking reality.

2

lockfile — the resolved reality

Records the exact versions the solver picked for every package, including all transitive dependencies. uv.lock, poetry.lock, or requirements-lock.txt depending on tool. This file is also committed — it is what enables a collaborator to reproduce your exact environment. Never generate it by hand; always generate it from the resolver.

pyproject.tomlminimal real example
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "tracelab"
version = "0.1.0"
requires-python = ">=3.11"
dependencies = [
    "click>=8.1",       # CLI framework — lower-bound only
    "rich>=13.0",       # pretty terminal output
    "httpx>=0.27",      # for future remote logging
]

[project.optional-dependencies]
dev = ["pytest>=8.0", "ruff"]

[project.scripts]
tracelab = "tracelab.cli:main"  # `tracelab` CLI entrypoint

Notice: lower bounds, not upper bounds. Upper-pinning (requests<3) is a sign of trauma — something broke and someone added a ceiling rather than fixing the root cause. It prevents your users from getting security patches. Leave ceilings to the lockfile and your testing discipline.

⚠ requirements.txt is not a lockfile

A requirements.txt generated with pip freeze > requirements.txt captures versions but not hashes, not dependency provenance, and not a re-resolvable graph. It is better than nothing and worse than a proper lockfile. If you're using pip without uv or Poetry, use pip-tools (pip-compile + pip-sync) to get a genuine two-layer model.

CORE CONCEPTSemver's social contract — and its routine violation

Semantic Versioning (semver) encodes a promise in a version number: MAJOR.MINOR.PATCH. PATCH means bug fixes only, no API changes. MINOR means new features, backwards-compatible. MAJOR means you broke something intentionally. The entire resolver ecosystem depends on this contract — it is what allows requests>=2.28 to work correctly at all.

semantic versioning — what each field promises 2 MAJOR — breaking change . 28 MINOR — new feature, compat . 3 PATCH — bug fix only pip's ~= 2.28 means: 2.28 ≤ v < 3.0
Semver's promise: you can safely upgrade within a MAJOR version. In practice, minor version bumps routinely contain quiet breaking changes — particularly in pre-1.0 libraries where the contract is "we promise nothing."

The contract breaks in a handful of reliably recurring ways. Pre-1.0 libraries (0.x.y) explicitly exempt themselves — any release can break anything. Popular libraries sometimes release MINOR bumps with behavioral changes that are technically "not breaking" by their definition but are breaking for your code. And critically: the contract covers the public API only. If you import a private submodule (from requests.packages.urllib3._collections import …), that is not covered, and it can change in a PATCH release. This happens constantly in data-science codebases that dig into sklearn or torch internals.

💡 The tilde and caret operators

~=2.28 (compatible release) means "2.28 or newer, but not 3.0" — i.e., accepts MINOR bumps, rejects MAJOR. ^1.2.3 (caret, Poetry/npm syntax) means "1.2.3 or newer, not 2.0.0". Both are shorthand for expressing "I trust the semver contract within this MAJOR." They're reasonable starting points; the lockfile is where you settle the actual version.

SEE ITDependency resolver simulator

This is where theory meets the war story. You are building TraceLab v1, and you need six packages. The resolver must find a set of versions that satisfies all constraints — including the constraints your dependencies have on their dependencies (transitive dependencies). Watch what the solver picks, and notice when "success" still hides a vulnerability.

⚙ Interactive · Dependency resolver

Adjust version constraints for your direct dependencies. The solver resolves the full transitive graph and reports success, conflict, or silent vulnerability. Then try pinning — and watch over-pinning cause a deadlock.

Pick constraints above, then click Resolve.

The key lesson from the simulator: "resolved successfully" and "resolved safely" are not the same thing. The solver's job is satisfying version constraints — it has no opinions about CVEs. A transitive dependency you never explicitly requested can carry a known vulnerability that your lockfile captures and ships. pip audit and uv audit scan the lockfile against advisory databases; run them in CI as a non-optional step.

⚠ Transitive dependency explosion

Install boto3 (the AWS SDK) and you pull in botocore, s3transfer, jmespath, and urllib3. Install langchain and you pull in 60+ packages. Every one of those is a potential vulnerability, an accidental API dependency, or a future conflict. pip show --verbose httpx | grep Requires is humbling the first time you run it on a real project. Minimize your direct dependencies aggressively; every package you add is a hire you'll have to manage.

WEAKNESSSupply-chain risk: the dependency is the attack surface

In 2022, the maintainer of colors and faker — packages with hundreds of millions of weekly downloads — deliberately introduced infinite loops as a protest. In 2021, the ua-parser-js account was compromised and a malicious version published that exfiltrated environment variables. In 2022, peacenotwar was injected as a dependency of node-ipc and wiped files on machines with Russian or Belarusian locale settings. None of these required a flaw in the consuming code — just a successful package install.

The Python ecosystem has had its own incidents. Typosquatting (reqeusts, colourama, pycrypt0) places malicious packages at names close to popular ones, relying on developer typos. Dependency confusion attacks work by registering a public package on PyPI with the same name as an internal package but with a higher version number — the resolver picks the highest matching version, which is now the attacker-controlled public package.

🔑 Minimum viable supply-chain hygiene

For a project like TraceLab that may run in production or CI environments: 1) commit a lockfile with hashes (uv lock or pip-compile --generate-hashes). 2) Run pip audit or uv audit in CI — it checks every locked package against the OSV database. 3) Review what you're adding before you add it: check the author, the source repo, the commit history, the download count. 4) Restrict registry lookups to a known-good index when running in production. None of these are paranoia — they're the minimum bar the industry moved to after 2021.

Hash pinning is the strongest guarantee. A lockfile with hashes (httpx==0.27.0 --hash=sha256:…) means pip will reject the package if it doesn't match — even if the version string was forged or the package was replaced on PyPI after your last resolve. pip install --require-hashes -r requirements-lock.txt enforces this. It is the difference between "I resolved this once" and "I will install exactly what I resolved."

TRACELABTraceLab v1 — the real package

Stage 2's milestone is converting the v0 script into a real Python package. This lesson's contribution is the dependency and packaging layer. Here is the directory layout and the key files:

tracelab/ (v1 src layout)Stage 2 milestone
tracelab/
├── pyproject.toml         # the intent layer
├── uv.lock                # the reality layer (commit this)
├── .python-version        # pins interpreter for pyenv / uv
├── .gitignore             # excludes .venv/, __pycache__, *.egg-info
└── src/
    └── tracelab/
        ├── __init__.py
        ├── cli.py         # click entrypoints: init, log, compare
        ├── store.py       # replaces the raw CSV open() calls
        └── py.typed       # PEP 561 marker: this package ships types
pyproject.tomlv1 — real dependencies
[project]
name = "tracelab"
version = "0.1.0"
requires-python = ">=3.11"
dependencies = [
    "click>=8.1",
    "rich>=13.7",
]

[project.optional-dependencies]
dev = [
    "pytest>=8.0",
    "ruff>=0.4",
    "mypy>=1.9",
    "pip-audit>=2.7",
]

[project.scripts]
tracelab = "tracelab.cli:main"

[tool.ruff.lint]
select = ["E", "F", "I"]  # pycodestyle errors + pyflakes + isort

The workflow after cloning is now three commands: uv sync --dev (creates venv, installs exact locked versions), uv run pytest (runs tests in the venv), uv run pip-audit (checks for known vulnerabilities). This is the entire "set up the project from scratch" story for a new contributor — and because the lockfile is committed, they get the exact same environment you tested with.

💡 The src/ layout is intentional

Putting package code in src/tracelab/ rather than tracelab/ at the root prevents a subtle import hazard: if you run pytest from the project root without installing, Python adds the current directory to sys.path, and import tracelab finds the source directory rather than the installed package. With the src/ layout, only the installed version is importable — which is what you actually want to test against. It's a small discipline that eliminates a category of "works on my machine" bugs.

The remaining debts from v0's table: the CSV encoding issue and comma-injection problem are now handled in store.py using Python's csv module with explicit UTF-8. The O(n) scan is addressed with an indexed lookup (Lesson 1.6's principle applied). No locking yet — that's a Stage 3 debt. No formal schema — Stage 4. One layer at a time.

WEAKNESSWhere this bites

✗ Committing no lockfile and trusting "it builds in CI"

CI runs pip install -r requirements.txt with loose constraints, gets today's newest compatible versions, and reports green. A month later, a transitive dependency releases a regression. CI is now on the new version; your production image was built two weeks ago with the old. Debugging requires reconstructing exactly what was installed at the time of the production build — information that no longer exists unless you committed a lockfile or built from a pinned image. Reproducibility is not retroactive. The lockfile must be committed before you need it.

✗ Over-pinning everything to exact versions in pyproject.toml

The instinct after getting burned is to pin everything: requests==2.31.0. This breaks your library for anyone who also needs requests but at a different version — you've just made your package unmergeable with half the ecosystem. The rule: applications pin (in the lockfile), libraries constrain loosely (in pyproject.toml). A library that pins exact versions is hostile to its users. Exact pinning belongs in the lockfile, not in the declared dependencies.

✗ Assuming "no known vulnerabilities today" means safe forever

A CVE published tomorrow against a package you locked six months ago is now in your production environment — and you won't know unless you run pip-audit against the lockfile again. The lock tells you what you have; only continuous auditing tells you if what you have is still safe. Set up a weekly CI job that runs pip-audit against the current lockfile and fails loudly. Several widely-used packages (Pillow, cryptography, urllib3) have had security advisories in the last two years — this is not theoretical.

KEY INSIGHTHow this connects

  • The Shell Is a Programming Language (1.4) — PATH resolution, shell environment, and how activate works are all shell mechanics. If the venv activation logic felt opaque, that lesson gives you the full model.
  • Debugging as a Science (2.8) — dependency failures produce specific, diagnosable error patterns: ModuleNotFoundError, ImportError, ResolutionImpossible. The systematic debugging approach from the previous lesson applies directly.
  • Code Review: Humans, Egos, and AI-Generated Diffs (2.10) — next up: a reviewer's first question on any PR that adds a dependency is "why this package, why not the standard library, and have you audited it?" You'll practice that question from both sides.

Packaging and dependencies are where the "software that survives contact with time" part of Stage 2 becomes concrete. Code that works today but is tied to an untracked, unreproducible environment is not software — it's a lucky moment. The lockfile is how you make the moment repeatable.

GLOSSARYKey terms

Virtual environment (venv)
An isolated Python installation consisting of a directory with its own site-packages/ and a PATH manipulation that makes a specific interpreter + package set the active one. Not OS-level isolation — just directory + PATH.
sys.prefix
The root path from which the active Python interpreter resolves its standard library and installed packages. Checking sys.prefix tells you which environment a running interpreter belongs to.
pyproject.toml
The PEP 517/518/621 standard file for declaring a Python project's build system, metadata, and dependency constraints. The intent layer: expresses what is needed, not exactly what is installed.
Lockfile
A generated file (e.g., uv.lock, poetry.lock) that records the exact versions and hashes of every package, including transitive dependencies, that the resolver selected. Committed to version control; enables exact environment reproduction.
Semantic versioning (semver)
The MAJOR.MINOR.PATCH convention: PATCH = bug fix, MINOR = backwards-compatible feature, MAJOR = breaking change. A social contract among maintainers, not a technical guarantee — violations are common, especially pre-1.0.
Transitive dependency
A package you didn't directly declare but that gets installed because one of your direct dependencies requires it. Most security vulnerabilities and version conflicts arrive through transitive dependencies.
Hash pinning
Recording the cryptographic hash (sha256) of each locked package alongside its version. Pip with --require-hashes rejects any package whose hash doesn't match — protecting against post-resolution tampering or supply-chain substitution.
src/ layout
The convention of placing a project's importable package under src/packagename/ rather than directly at the repo root. Prevents the local source tree from shadowing the installed package during testing, eliminating a class of "works locally, breaks in CI" bugs.

ACTIVE RECALLCheck your understanding

Answer in your own words first, then reveal.

1You run pip install httpx and then python -c "import httpx" fails with ModuleNotFoundError. What is the most likely cause, and what command confirms it?›

The pip executable and the python executable are resolving to different environments — pip installed into one interpreter's site-packages and python reads a different one. Confirm with which pip, which python, and python -c "import sys; print(sys.prefix)". The fix: always use python -m pip install httpx, which guarantees pip targets the same interpreter you're invoking.

2What is the difference between pyproject.toml's [project] dependencies and a lockfile? Why do you need both?›

pyproject.toml expresses intent: loose constraints that allow a range of versions. The lockfile records reality: the exact versions (and hashes) the resolver actually selected, including all transitive dependencies. You need both because intent is for humans and flexibility (a contributor with different existing packages can still satisfy your constraints), while the lockfile is for machines and reproducibility (anyone who installs from the lockfile gets byte-for-byte the same environment you tested with).

3Your package declares requests>=2.28. A user installs it and gets requests 2.31.0. Four months later you add a feature, they upgrade, and now requests 2.32.0 is installed. What semver promise means this should work — and what are two ways the promise can break?›

Semver promises that a MINOR version bump (2.28 → 2.32, same MAJOR=2) is backwards-compatible — no existing API should break. It can break in two common ways: (1) the maintainer introduced a behavioral change they considered non-breaking because no API signature changed, but your code depended on the old behavior; (2) you're importing a private submodule or implementation detail that isn't part of the public API contract and changed freely between versions. Semver is a social contract, not a technical guarantee.

4Why is the src/ layout recommended for Python packages, and what bug does it prevent?›

Without src/, running pytest from the project root causes Python to add the current directory to sys.path, making import tracelab resolve to the local source tree rather than the installed package. This means you test against code that hasn't been built/installed, potentially hiding errors that only appear after packaging. With src/tracelab/, the package is only importable after installation, so your tests always run against what a real user would install — a class of "works on my machine" bugs is eliminated.

Next: reviewing code — human and AI-generated — without rubber-stamping →