interzone-mirror

A package mirror that holds only what interzone and matrix-docker-ansible-deploy install, on Debian, Ubuntu, Raspbian and Arch Linux. It is meant for servers that lose international connectivity: host it where they can reach it, and point interzone at it. Alongside it, Forgejo carries the playbook and its roles, and zot every container image they run.

To run your own, follow SETUP.md. This page explains how each part works.

One host serves all three of interzone's mirror settings:

interzone setting	URL	carries
OS package mirror	https://mirror.example/	/debian, /debian-security, /ubuntu, /ubuntu-ports, /raspbian, /archlinux
Docker package mirror	https://mirror.example/docker	a copy of download.docker.com/linux: debian, ubuntu, raspbian
PyPI mirror	https://mirror.example/pypi/simple	Ansible, passlib and their dependencies

How it stays small

A full mirror of these distros runs to terabytes. This one is a few GB:

• Indexes are copied untouched. That covers Release/InRelease, Packages and core.db. Upstream signatures still verify, so servers keep checking them as usual.

• Packages are only those needed. For each release and CPU, the distro's own package manager runs in a container and resolves, as if nothing were installed, what interzone installs:

  • Docker CE,
  • python3-docker,
  • fuse,
  • openssl,
  • git,
  • the time sync daemon,
  • and so on: see NEEDS in sync/izmirror.py.

  It also resolves the packages a server already has (Debian's required and important packages, base on Arch). Installing one of them can pull an upgrade of the others, so they are carried too. Every dependency is carried, at the version the copied indexes name.

• Anything else gets a 404. A server can still apt update / pacman -Sy, because all the indexes are there. It can install and upgrade everything interzone needs, but not arbitrary packages. Add your own packages under [extra] in mirror.toml.

APT servers fetch indexes by hash (by-hash/SHA256/…). Files nothing refers to any more are kept for keep_days, so a server holding the previous indexes can still download what they name.

Use

./sync.sh                 # copy indexes, resolve, download, publish (~30 min the first time)
docker compose up -d web  # serve data/mirror through Traefik (labels in compose.yaml)

• Re-sync daily. interzone flags security indexes over 14 days old. For example:

  0 3 * * * /srv/interzone-mirror/sync.sh >>/var/log/interzone-mirror.log 2>&1
  

  A sync builds the new indexes aside and switches over only once every package they name is downloaded. The mirror stays usable while it runs.

• Put TLS in front. Use a reverse proxy (Caddy, Traefik, nginx) with a certificate for the host name, or change nginx.conf. Servers verify package signatures either way, but HTTPS hides what they download.

• Choose what to carry in mirror.toml:

  • releases,
  • CPUs (amd64/arm64; Arch is x86_64 only),
  • Raspbian,
  • the Python versions the PyPI index serves,
  • and the upstream URLs.

  Point the upstreams at mirrors near the sync host for speed. Any mirror with the same layout works.

• The sync host needs internet access. Behind a proxy, export https_proxy/http_proxy before ./sync.sh; if the proxy listens on the host's localhost, sync.sh puts the sync on the host network (SYNC_NETWORK=host).

• The sync host and the serving host can differ. The serving host needs no internet: sync outside, then rsync -a --delete-after data/mirror/ inward.

Git mirrors and images

The images job keeps the rest of what interzone's blackout mode uses in step with matrix-docker-ansible-deploy:

• Git: every pull mirror on Forgejo (FORGEJO_URL) re-syncs every 6 hours. Roles that requirements.yml adds get a new mirror at <forgejo>/<org>/<repo>, which is where interzone points github.com role URLs.
• Images: the playbook and its roles are checked out at the mirror's current commit, and images/list_mdad_images.py lists every image they can run on amd64 and arm64. Each tag the registry (REGISTRY, zot) lacks is copied for every architecture to <registry>/<path>, the upstream host dropped, which is what matrix_container_global_registry_prefix_override expects. Moving tags (latest, main) are copied again when upstream changes. Failures are retried on the next run; data/images/state.json records the last commit and what failed.

It needs three files in secrets/ (kept out of git, mode 600): forgejo-token (an admin token, to change mirror settings and create mirrors), zot-user and zot-password.

docker compose --profile sync run --rm images   # once
docker compose up -d scheduler                  # images, then ./sync.sh, every 6 hours
docker compose logs -f scheduler

The scheduler drives the host's Docker through its socket, and sees this directory at the same path, so run compose from here.

Test

DATA=./data test/offline.sh            # every synced system this CPU can run
DATA=./data test/offline.sh debian-trixie-amd64 archlinux-x86_64

Each distro's container is put on an --internal Docker network whose only other member is the mirror. It points its sources at the mirror, runs a full upgrade, then installs interzone's packages. Docker's and the distro's signatures are checked. A client on each configured Python version then runs pip install ansible passlib against the PyPI index.

More

• SETUP.md: setting up every part, the scheduler, freeze.sh, proxy.sh, and interzone's settings
• IPFS.md: the IPFS entry node and publish
• examples/: Forgejo and zot as this mirror runs them
• LICENSE: MIT

Layout

compose.yaml       web (nginx) plus the "sync" profile: plan → resolve-apt, resolve-archlinux → fetch
mirror.toml        what to carry
sync/izmirror.py   plan: copy indexes to data/staging, write data/plan/<system>/
                   fetch: download packages, build the PyPI index, publish, prune
resolve/resolve.sh runs apt-get --print-uris / pacman -Sp in debian and archlinux containers
data/mirror        what web serves