Abstract

This PEP describes a mirroring infrastructure for PyPI.

PEP Withdrawal

The main PyPI web service was moved behind the Fastly caching CDN in May 2013: https://mail.python.org/pipermail/distutils-sig/2013-May/020848.html

Subsequently, this arrangement was formalised as an in-kind sponsorship with the PSF, and the PSF has also taken on the task of risk management in the event that that sponsorship arrangement were to ever cease.

The download statistics that were previously provided directly on PyPI, are now published indirectly via Google Big Query: https://packaging.python.org/guides/analyzing-pypi-package-downloads/

Accordingly, the mirroring proposal described in this PEP is no longer required, and has been marked as Withdrawn.

Rationale

PyPI is hosting over 6000 projects and is used on a daily basis by people to build applications. Especially systems like easy_install and zc.buildout make intensive usage of PyPI.

For people making intensive use of PyPI, it can act as a single point of failure. People have started to set up some mirrors, both private and public. Those mirrors are active mirrors, which means that they are browsing PyPI to get synced.

In order to make the system more reliable, this PEP describes:

• the mirror listing and registering at PyPI
• the pages a public mirror should maintain. These pages will be used by PyPI, in order to get hit counts and the last modified date.
• how a mirror should synchronize with PyPI
• how a client can implement a fail-over mechanism

Mirror listing and registering

People that wants to mirror PyPI make a proposal on catalog-SIG. When a mirror is proposed on the mailing list, it is manually added in a mirror list in the PyPI application after it has been checked to be compliant with the mirroring rules.

The mirror list is provided as a list of host names of the form

X.pypi.python.org

The values of X are the sequence a,b,c,…,aa,ab,… a.pypi.python.org is the master server; the mirrors start with b. A CNAME record last.pypi.python.org points to the last host name. Mirror operators should use a static address, and report planned changes to that address in advance to distutils-sig.

The new mirror also appears at http://pypi.python.org/mirrors which is a human-readable page that gives the list of mirrors. This page also explains how to register a new mirror.

Mirror Authenticity

With a distributed mirroring system, clients may want to verify that the mirrored copies are authentic. There are multiple threats to consider:

1. the central index may get compromised
2. the central index is assumed to be trusted, but the mirrors might be tampered.
3. a man in the middle between the central index and the end user, or between a mirror and the end user might tamper with datagrams.

This specification only deals with the second threat. Some provisions are made to detect man-in-the-middle attacks. To detect the first attack, package authors need to sign their packages using PGP keys, so that users verify that the package comes from the author they trust.

The central index provides a DSA key at the URL /serverkey, in the PEM format as generated by “openssl dsa -pubout” (i.e. RFC 3280 SubjectPublicKeyInfo, with the algorithm 1.3.14.3.2.12). This URL must not be mirrored, and clients must fetch the official serverkey from PyPI directly, or use the copy that came with the PyPI client software. Mirrors should still download the key, to detect a key rollover.

For each package, a mirrored signature is provided at /serversig/<package>. This is the DSA signature of the parallel URL /simple/<package>, in DER form, using SHA-1 with DSA (i.e. as a RFC 3279 Dsa-Sig-Value, created by algorithm 1.2.840.10040.4.3)

Clients using a mirror need to perform the following steps to verify a package:

1. download the /simple page, and compute its SHA-1 hash
2. compute the DSA signature of that hash
3. download the corresponding /serversig, and compare it (byte-for-byte) with the value computed in step 2.
4. compute and verify (against the /simple page) the MD-5 hashes of all files they download from the mirror.

An implementation of the verification algorithm is available from https://svn.python.org/packages/trunk/pypi/tools/verify.py

Verification is not needed when downloading from central index, and should be avoided to reduce the computation overhead.

About once a year, the key will be replaced with a new one. Mirrors will have to re-fetch all /serversig pages. Clients using mirrors need to find a trusted copy of the new server key. One way to obtain one is to download it from https://pypi.python.org/serverkey. To detect man-in-the-middle attacks, clients need to verify the SSL server certificate, which will be signed by the CACert authority.

Special pages a mirror needs to provide

A mirror is a subset copy of PyPI, so it provides the same structure by copying it.

• simple: rest version of the package index
• packages: packages, stored by Python version, and letters
• serversig: signatures for the simple pages

It also needs to provide two specific elements:

• last-modified
• local-stats

# package,filename,useragent,count
zc.buildout,zc.buildout-1.6.0.tgz,MyAgent,142
...

How a mirror should synchronize with PyPI

A mirroring protocol called Simple Index was described and implemented by Martin v. Loewis and Jim Fulton, based on how easy_install works. This section synthesizes it and gives a few relevant links, plus a small part about User-Agent.

>>> import socket
>>> socket.gethostbyname_ex('last.pypi.python.org')[0]
'h.pypi.python.org'

References

Acknowledgments

Georg Brandl.

Copyright

This document has been placed in the public domain.