Changes between Initial Version and Version 1 of Mirroring

Timestamp:

Mar 1, 2016, 1:41:03 PM (8 years ago)

Author:

ryandesign (Ryan Carsten Schmidt)

Comment:

add Mirroring page

Mirroring

@@ +1 @@
+= Mirroring
+
+Thank you for your interest in providing a public MacPorts mirror server to join our existing [wiki:Mirrors list of mirrors]. Making MacPorts content available in more locations makes MacPorts faster.
+
+As of March 2016 these instructions are preliminary. If you find inaccuracies or have other comments or questions, please let us know by contacting [https://lists.macosforge.org/mailman/listinfo/macports-mgr portmgr].
+
+== Contact us
+
+First, please contact [https://lists.macosforge.org/mailman/listinfo/macports-mgr portmgr] to let us know you would like to create a mirror. Please tell us the following information:
+
+* The name and web page URL of the organization providing the mirror, so that we can credit you properly.
+* The city, state/province and [wikipedia:ISO_3166-1_alpha-2 ISO 3166-1 alpha-2 country code] where your mirror is located and the [wikipedia:International_Air_Transport_Association_airport_code IATA airport code] of the closest airport. The airport code and country code will become your MacPorts mirror name.
+* Your mirror's existing hostname. We will create new hostnames abc.xy.distfiles.macports.org, abc.xy.packages.macports.org and abc.xy.rsync.macports.org (where "abc" is your airport code and "xy" is your country code) `CNAME`d to your existing hostname.
+* The email address of the server's administrator. We will list this on our mirrors page and MacPorts users might report problems to this address.
+* Your mirror's upstream Internet connection speed and whether or not your mirror supports IPv6. We will list this information on our mirrors page.
+
+We will review your request to become a mirror and will let you know if there are any problems. Meanwhile you can follow the rest of these instructions to get your mirror ready. We reserve the right to refuse a request to become a mirror for any reason, or to remove existing mirrors from use at any time and without notice.
+
+If in the future you wish to temporariliy or permanently discontinue providing your mirror, please let us know so that we can make the appropriate changes.
+
+== Requirements
+
+Running a MacPorts mirror will use a possibly significant amount of bandwidth. Our primary server in California, for example, serves around 60GB of data to users per day. MacPorts chooses the closest mirror, so how much traffic your mirror sees depends on the number of MacPorts users for whom your mirror is the closest. Ensure you have the permission of your network provider to run such a mirror, and that this amount of traffic will not be a problem for you.
+
+You'll need a server with at least 1TB of available disk space on a case-sensitive filesystem. You'll need a web server on port 80 such as Apache httpd or nginx on which you can configure new virtual hosts, and the rsync daemon running on port 873. The server should have a hostname and a static IPv4 address. IPv6 access is welcomed but is not required. The server does not need to be running OS X and does not need to have MacPorts installed.
+
+== Rsync modules
+
+The MacPorts files are distributed in four rsync modules. You should usually mirror all of them.
+
+* **packages** contains precompiled archives of each MacPorts port for several OS X versions. This is the largest module, around 600GB. Mirroring this module will provide the greatest benefit to MacPorts users near you since it lets them automatically receive these large files more quickly.
+* **distfiles** contains the source code tarballs from which the precompiled archives were created. There are some ports that cannot legally be distributed as binaries, so those must build from these source files on the user's computer. This is the second-largest module, around 100GB. **Note:** this module used to be called ''mpdistfiles''. If you are mirroring ''mpdistfiles'', please switch to mirroring ''distfiles''. If your local module is called ''mpdistfiles'', please rename it to ''distfiles'' and let us know so that we can update mirror_sites.tcl.
+* **release** contains the latest released version of the MacPorts base source code which users update by running `sudo port selfupdate`, as well as the collection of Portfiles—the text files MacPorts reads that tell it how to install each port—which users update by running `sudo port sync`. Users near you can manually configure MacPorts to user your mirror for faster speed. This small module is around 500MB.
+* **trunk** contains the latest development version of the MacPorts base source code. This small module is around 20MB.
+
+== Get the files
+
+You'll get the files from our server via [https://rsync.samba.org/ rsync]. You should use the latest version of rsync.
+
+For your initial sync, you might speed up your transfer by using a nearby existing MacPorts mirror server. For subsequent scheduled syncing, use the master rsync server, rsync-origin.macports.org. 
+
+**Note:** If you are syncing from rsync.macports.org, distfiles.macports.org, packages.macports.org, or a specific IP address, please switch to rsync-origin.macports.org.
+
+There is a sample script below for mirroring these modules via rsync. You don't have to use that script if you already have other arrangements for syncing, but please use the following `rsync` flags, for the following reasons:
+
+||=`rsync` flag        =||=Purpose =
+|| `--compress`, `-z`   || File data is compressed during transfer. The server won't re-compress files that are already compressed.
+|| `--delete-delay`     || Files which were removed on the server are removed from the mirror after the transfer is finished. This option is similar to but more efficient than `--delete-after`.
+|| `--hard-links`, `-H` || Hard links are preserved.
+|| `--links`, `-l`      || Symbolic links (symlinks) are preserved.
+|| `--perms`, `-p`      || File and directory permissions are preserved.
+|| `--recursive`, `-r`  || Directories are copied recursively.
+|| `--timeout=600`      || If no data is transferred for 10 minutes, the connection is dropped.
+|| `--times`, `-t`      || Modification times are preserved.
+
+== Stay up to date
+
+After the initial sync is complete, keep your mirror up to date by periodically re-running the sync script. You can do this using cron, launchd, systemd, or another scheduler. A sample cron configuration is provided below.
+
+The sample configuration syncs every hour. It is not necessary to sync that often, but MacPorts developers commit dozens of port updates each day, so the more frequently you sync, the more likely your mirror is to have the content users are requesting. Ensure your mirror syncs at least once daily.
+
+== Rsync server
+
+MacPorts retrieves the contents of the ''release'' and ''trunk'' modules via rsync. MacPorts comes preconfigured to use the server rsync.macports.org but users can edit their configuration to use your mirror instead. In addition, other mirror administrators may wish to copy your ''packages'' and ''distfiles'' modules. So you should run an rsync daemon to republish the modules you synced.
+
+The MacPorts rsync modules have generic names, so we suggest you put our modules in a "macports" folder. We would prefer you not rename our modules.
+
+**Note:** If you are providing the ''release'' or ''trunk'' modules under nonstandard names, please edit your rsyncd configuration to provide the module under both the nonstandard name and the standard name, and edit the [wiki:Mirrors Mirrors] page to reflect the standard name. If you are providing the ''packages'' or ''distfiles'' modules under nonstandard names, please edit your rsyncd configuration to provide the module under the standard name, edit the [wiki:Mirrors Mirrors] page to reflect the change, and let us know so that we can update mirror_sites.tcl and/or archive_sites.tcl.
+
+There is a sample rsyncd configuration below.
+
+== Web server
+
+MacPorts retrieves files from the ''packages'' and ''distfiles'' modules via http. MacPorts automatically downloads these files from whichever mirror is closest. So you should run a web server to make these modules available.
+
+The contents of the ''packages'' module should be made available directly under !http://abc.xy.packages.macports.org/ and the contents of the ''distfiles'' module should be made available directly under !http://abc.xy.distfiles.macports.org/. It is not necessary to provide the contents of the ''release'' or ''trunk'' modules via http.
+
+**Note:** If you are currently providing the contents of the ''packages'' or ''distfiles'' modules in a subdirectory of your MacPorts mirror hostname, or any other hostname, you should reconfigure your web server to provide the contents directly under the MacPorts mirror hostname, and let us know so we can update archive_sites.tcl and/or mirror_sites.tcl. You should configure HTTP 301 redirects from the previously-used URLs to the new ones.
+
+Directory listings should be turned on. URL spelling correction and multiviews must be turned off. There are sample web server configurations below.
+
+== Add mirror to MacPorts
+
+Once your server is synced and ready to accept MacPorts traffic, let us know and we will edit archive_sites.tcl and mirror_sites.tcl to add the URLs of your MacPorts mirror hostnames. Also edit the [wiki:Mirrors Mirrors] page to add information about your server, or we can add it for you.
+
+== Sample scripts and configuration files
+
+=== Sample rsync update script
+
+This sample script can be used to mirror the MacPorts rsync modules.
+
+For your initial sync of ''packages'' and ''distfiles'', you can possibly make your transfer faster by uncommenting the `RSYNC_URL_FORMAT` line for a mirror server closer to you, but note that currently not all servers mirror all modules, and not all servers follow our module naming suggestions. For subsequent scheduled syncs, use the origin server.
+
+{{{
+#!/bin/sh
+
+set -euo pipefail
+
+RSYNC_DIR="/var/www/html/macports"
+
+RSYNC_MODULES="packages distfiles release trunk"
+
+RSYNC_URL_FORMAT="rsync://rsync-origin.macports.org/%s"
+#RSYNC_URL_FORMAT="rsync://cjj.kr.rsync.macports.org/macports/%s"
+#RSYNC_URL_FORMAT="rsync://fco.it.rsync.macports.org/macports-%s"
+#RSYNC_URL_FORMAT="rsync://jnb.za.rsync.macports.org/macports/%s"
+#RSYNC_URL_FORMAT="rsync://jog.id.rsync.macports.org/macports/%s"
+#RSYNC_URL_FORMAT="rsync://lil.fr.rsync.macports.org/%s"
+#RSYNC_URL_FORMAT="rsync://mse.uk.rsync.macports.org/%s.macports.org"
+#RSYNC_URL_FORMAT="rsync://nou.nc.rsync.macports.org/macports/%s.macports.org"
+#RSYNC_URL_FORMAT="rsync://nue.de.rsync.macports.org/macports/%s"
+#RSYNC_URL_FORMAT="rsync://osl.no.rsync.macports.org/%s"
+#RSYNC_URL_FORMAT="rsync://sea.us.rsync.macports.org/%s"
+#RSYNC_URL_FORMAT="rsync://sjc.us.rsync.macports.org/%s"
+
+RSYNC=rsync
+RSYNC_ARGS="--compress --delete-delay --hard-links --links --no-motd --perms --recursive --stats --timeout=600 --times"
+if [ -t 1 ]; then
+    RSYNC_ARGS="$RSYNC_ARGS --info=progress2"
+fi
+
+for RSYNC_MODULE in $RSYNC_MODULES; do
+    RSYNC_URL="$(printf "$RSYNC_URL_FORMAT" "$RSYNC_MODULE")"
+    echo "Syncing $RSYNC_MODULE from $RSYNC_URL"
+    $RSYNC $RSYNC_ARGS "$RSYNC_URL" "$RSYNC_DIR/$RSYNC_MODULE"
+    echo
+done
+}}}
+
+=== Sample cron configuration
+
+This sample cron configuration runs the above sync script every hour at 30 minutes past the hour:
+
+{{{
+30 * * * * /usr/local/bin/macports-rsync.sh 2>&1 >> /var/log/macports-rsync.log
+}}}
+
+So that not all mirrors are trying to connect to the origin server at the same time, please change "30" to a random number between 0 and 59.
+
+=== Sample Apache httpd configuration
+
+This sample configuration can be used to serve MacPorts files with Apache httpd.
+
+{{{
+<Directory /var/www/html/macports>
+    Order allow,deny
+    Allow from all
+    AllowOverride None
+    Options +Indexes -FollowSymLinks -MultiViews
+    IndexOptions Charset=UTF-8 FancyIndexing IconsAreLinks HTMLTable NameWidth=* SuppressDescription
+    AddType application/macbinhex-40 .hqx
+    AddType application/octet-stream .bin .exe .rmd160
+    AddType application/x-7z-compressed .7z
+    AddType application/x-apple-diskimage .dmg .image .img .smi
+    AddType application/x-bzip2 .bz2 .tbz .tbz2
+    AddType application/x-compress .Z
+    AddType application/x-gzip .gz .tgz
+    AddType application/x-iso9660-image .iso
+    AddType application/x-lzip .lz
+    AddType application/x-lzma .lzma .tlz
+    AddType application/x-rar-compressed .rar
+    AddType application/x-stuffit .sit
+    AddType application/x-stuffitx .sitx
+    AddType application/x-tar .tar
+    AddType application/x-tcl .tcl
+    AddType application/x-xz .txz .xz
+    AddType application/zip .zip
+</Directory>
+<VirtualHost *:80>
+    ServerName abc.xy.distfiles.macports.org
+    ServerAlias distfiles.macports.org
+    DocumentRoot /var/www/html/macports/distfiles
+    UseCanonicalName On
+    <IfModule mod_speling.c>
+        CheckSpelling Off
+    </IfModule>
+
+    Header set "Cache-Control" "max-age=2592000, public"
+    <LocationMatch "/$">
+        Header set "Cache-Control" "max-age=300, public"
+    </LocationMatch>
+    <LocationMatch "^/ports\.tar\.gz$">
+        Header set "Cache-Control" "max-age=1800, public"
+    </LocationMatch>
+</VirtualHost>
+<VirtualHost *:80>
+    ServerName abc.xy.packages.macports.org
+    ServerAlias packages.macports.org
+    DocumentRoot /var/www/html/macports/packages
+    UseCanonicalName On
+    <IfModule mod_speling.c>
+        CheckSpelling Off
+    </IfModule>
+
+    Header set "Cache-Control" "max-age=2592000, public"
+    <LocationMatch "/$">
+        Header set "Cache-Control" "max-age=300, public"
+    </LocationMatch>
+</VirtualHost>
+}}}
+
+If you are serving MacPorts files from subdirectories on any of your hostnames, use `Redirect` directives to redirect those requests to their new locations. For example, these directives show how to redirect requests for !http://hostname/pub/macports/packages/... to !http://abc.xy.packages.macports.org/... and requests for !http://hostname/pub/macports/distfiles/... to !http://abc.xy.distfiles.macports.org/...:
+
+{{{
+    Redirect 301 /pub/macports/packages http://abc.xy.packages.macports.org
+    Redirect 301 /pub/macports/distfiles http://abc.xy.distfiles.macports.org
+}}}
+
+Place these directives inside the `VirtualHost` directives for each hostname from which you were serving the files at the old URLs, or use an include file.
+
+=== Sample nginx configuration
+
+This sample configuration can be used to serve MacPorts files with nginx.
+
+{{{
+types {
+    application/macbinhex-40 hqx;
+    application/octet-stream bin exe rmd160;
+    application/x-7z-compressed 7z;
+    application/x-apple-diskimage dmg image img smi;
+    application/x-bzip2 bz2 tbz tbz2;
+    application/x-compress Z;
+    application/x-gzip gz tgz;
+    application/x-iso9660-image iso;
+    application/x-lzip lz;
+    application/x-lzma lzma tlz;
+    application/x-rar-compressed rar;
+    application/x-stuffit sit;
+    application/x-stuffitx sitx;
+    application/x-tar tar;
+    application/x-tcl tcl;
+    application/x-xz txz xz;
+    application/zip zip;
+}
+server {
+    listen 80;
+    listen [::]:80;
+    server_name abc.xy.distfiles.macports.org distfiles.macports.org;
+    root /var/www/html/macports/distfiles;
+    server_name_in_redirect on;
+    autoindex on;
+
+    add_header Cache-Control "max-age=2592000, public";
+    location ~ /$ {
+        add_header Cache-Control "max-age=300, public";
+    }
+    location = /ports.tar.gz {
+        add_header Cache-Control "max-age=1800, public";
+    }
+}
+server {
+    listen 80;
+    listen [::]:80;
+    server_name abc.xy.packages.macports.org packages.macports.org;
+    root /var/www/html/macports/packages;
+    server_name_in_redirect on;
+    autoindex on;
+
+    add_header Cache-Control "max-age=2592000, public";
+    location ~ /$ {
+        add_header Cache-Control "max-age=300, public";
+    }
+}
+}}}
+
+If you are serving MacPorts files from subdirectories on any of your hostnames, use `return` directives inside a `location` directive to redirect those requests to their new locations. For example, these directives show how to redirect requests for !http://hostname/pub/macports/packages/... to !http://abc.xy.packages.macports.org/... and requests for !http://hostname/pub/macports/distfiles/... to !http://abc.xy.distfiles.macports.org/...:
+
+{{{
+    location ^~ /pub/macports/packages(/.*)?$ {
+        return 301 http://localhost.packages.macports.org$1;
+    }
+    location ^~ /pub/macports/distfiles(/.*)?$ {
+        return 301 http://localhost.distfiles.macports.org$1;
+    }
+}}}
+
+Place these directives inside the `server` directives for each hostname from which you were serving the files at the old URLs, or use an include file.
+
+=== Sample rsyncd configuration
+
+{{{
+TODO
+}}}