Mirroring on demand
Note: see Creating a mirror instead for an alternative in which packages are synchronized periodically, which is the method normally used by public mirrors.
Since 70-100Gb could be a bit too much for mirroring Parabola repos, you can make your web server fetch, cache and serve packages on users' demand.
The basic procedure would be like this:
- Maintain repo databases updated. This means don't cache them at all or cache them for as much as a day.
- Make your web server proxy package and package signatures (or anything but *.db) and cache them in a way that the second query will be served locally.
Feel free to add instructions for your favourite web servers here.
Using Nginx
Being a reverse proxy, Nginx can be configured to proxy Parabola repos while filling a local cache. Here's an explained snippet:
# You can redirect queries to any number of mirrors to distribute the load
# as long as they serve the 'repo.parabolagnulinux.org' subdomain
upstream parabola {
# Our repo IP
server 80.87.131.252;
# Other mirror IPs!
}
# The mirror
# It serves two subdomains:
# * The actual mirror, and
# * A bogus main repo subdomain so this server can act as another mirror
# upstreamĀ ;)
server {
listen 80;
server_name mirror.example.org repo.parabolagnulinux.org;
root /srv/http/mirror.example.org;
autoindex on;
# Databases are never cached
# Queries are proxied directly to upstream
# We trick upstream into serving the main repo subdomain
location ~ \.db$ {
expires 1h;
proxy_pass http://parabola;
proxy_set_header Host repo.parabolagnulinux.org;
}
# Get and cache everything else
# The error_page directive means 'go fetch anything that can't be found here'
location / {
expires 7d;
error_page 403 404 = @get;
}
# Bogus location that redirects queries
# Pass it to the parabola upstream
# We trick upstream into serving the main repo subdomain
# Store the files in this format
# Give them 644 permissions
location @get {
proxy_pass http://parabola;
proxy_set_header Host repo.parabolagnulinux.org;
proxy_store /srv/http/mirror.example.org$request_uri;
proxy_store_access user:rw group:r all:r;
}
}
Cleaning up
You can cleanup old packages using the db-cleanup utility from our dbscripts. Clone it from https://projects.parabolagnulinux.org/dbscripts.git, edit the config file to suit your needs and run db-cleanup every once in a while (we do it weekly).