argh

3 files changed, 80 insertions, 24 deletions

diff --git a/sbodl b/sbodl
index 7dd35f4..def7c7e 100755
--- a/sbodl
+++ b/sbodl
@@ -90,7 +90,7 @@ for dl in $DL; do
 	else
 		wget $WGETARGS $EXTRAWGETARGS "$dl" || die "Download failed"
 		if [ -e "$FILE" ]; then
-			mv "$FILE" "$CACHEDIR"
+			mv -b "$FILE" "$CACHEDIR"
 			ln -s "$CACHEDIR/$FILE" "$FILE"
 		fi
 	fi
@@ -105,6 +105,5 @@ for dl in $DL; do
 	else
 		echo "WARN: can't find downloaded file $FILE"
 	fi
-	echo
 	shift
 done
diff --git a/sbosrcarch b/sbosrcarch
index f5da6d4..0b32cb9 100755
--- a/sbosrcarch
+++ b/sbosrcarch
@@ -1192,8 +1192,8 @@ sub handle_info_file {
 		my $filename = url_to_filename($url);
 		print ": $url\n";
 
-		if(exists($url_rewrite_hacks{$category/$prgnam})) {
-			$url = $url_rewrite_hacks{$category/$prgnam}->($url);
+		if(exists($url_rewrite_hacks{'$category/$prgnam'})) {
+			$url = $url_rewrite_hacks{'$category/$prgnam'}->($url);
 		}
 
 		if(already_exists($filename, $category, $prgnam, $md5)) {
diff --git a/sbosrcarch.faq b/sbosrcarch.faq
index 19b30ac..88b0e2c 100644
--- a/sbosrcarch.faq
+++ b/sbosrcarch.faq
@@ -32,12 +32,12 @@ A: The archive server is operated by Darren Austin, aka "Tadgy"
    archive healthy.
 
    The best way to contact us is using an IRC client to connect to
-   Freenode and join the ##slackware or ##slackbuilds channel.
+   Freenode and join the ##slackware or #slackbuilds channel.
 
    We can also be reached by email:
 
    B. Watson <yalhcru@gmail.com>
-   TODO: make sure Tadgy is OK with his email being here!
+   Darren Austin <mirrors (at) slackware.uk>
 
    Please read this entire FAQ before asking us questions. Chances are,
    you'll find the answer here. If not, or if the answer isn't clear
@@ -95,10 +95,10 @@ A: Several answers here:
      you'll see a list of build name directories. Each of these will
      contain the source file(s) for the build.
 
-     Example: you can't download the source to system/atari800, so
-     you go to the by-name page, click on "system", then "atari800".
-     There you'll see the file you wanted, atari800-3.1.0.tar.gz (unless
-     it's been updated since I wrote this).
+     Example: you can't download the source to system/atari800
+     from its original URL, so you go to the by-name page, click on
+     "system", then "atari800".  There you'll see the file you wanted,
+     atari800-3.1.0.tar.gz (unless it's been updated since I wrote this).
 
    - With a download tool like wget or curl. You could do this using the
      same by-name tree as you would for manual lookups, but it's better to
@@ -244,8 +244,7 @@ A: Multiple answers:
    can remove and blacklist it. It is not our intention to violate
    anyone's license.
 
-Q: Why does the status page say the by-name tree is missing 4 files, but
-   the by-md5sum tree is 100% complete?
+Q: Why do some of the by-name directories have filenames ending in ".x86_64"?
 
 A: This is due to a design flaw in the archive structure. We assumed that
    download filenames would either be unique within an .info file, or else
@@ -258,12 +257,14 @@ A: This is due to a design flaw in the archive structure. We assumed that
       DOWNLOAD_x86_64="https://www.perforce.com/downloads/perforce/r18.1/bin.linux26x86_64/p4"
 
    Notice that both URLs end in "/p4". The directory parts of the URL are
-   different, but the filenames are the same.
+   different, but the filenames are the same. In the archive, the 32-bit
+   download will be called "p4" and the 64-bit one will be "p4.x86_64".
 
    The archive script successfully downloads these files and stores them
    in the by-md5 tree in the correct directories. But when it tries to
    store them in the by-name tree, it's trying to save two files in the
-   same directory with the same name. The second one overwrites the first.
+   same directory with the same name. If it didn't use a different name,
+   the second one would overwrite the first.
 
    The current list of builds affected by this is:
 
@@ -272,10 +273,6 @@ A: This is due to a design flaw in the archive structure. We assumed that
       development/p4d
       libraries/p4api
 
-   Eventually this will get more-or-less fixed: for these builds (and only
-   these), there will probably be separate x86/ and x86_64/ subdirectories
-   (e.g. development/p4/x86/p4).
-
 Q: I'm a SlackBuild maintainer, and the download URL for one of my builds
    has disappeared. Can I use the archive URL as the DOWNLOAD in my .info
    file?
@@ -299,12 +296,64 @@ A: Yes, but only as a temporary measure or a last resort.
    every source file to have two working URLs: the original plus the
    sbosrcarch one.
 
+Q: I'm a SlackBuild maintainer, and one of my builds keeps showing up
+   on the sbosrcarch STATUS as missing. How can I prevent this?
+
+   This usually happens for one of these reasons:
+
+   1. You made a mistake in your submission. Double-check the DOWNLOAD URL(s)
+      and MD5SUM(s) in the .info file. If they're wrong, resubmit your build.
+
+   2. The filename in the download URL is "unversioned", meaning the version
+      number isn't part of the filename (e.g. "thingy-latest.tar.gz"). At
+      some point after you last updated your .info file, but before the
+      SBo public update, the file changed on the server. Actually, this
+      occasionally happens even for files that have the version number
+      in the filename: upstream makes a mistake (leave a file out of the
+      tarball for instance) and a day or so later, they fix it without
+      changing the version number. When the archiver downloads the file,
+      it checks the md5sum against your .info file and sees a mismatch,
+      so it won't archive the file.
+
+   3. Upstream made a new release after you updated your build, but before
+      the SBo public update, and they removed the old version from their
+      server (or, possibly, moved it to a different location like /archives/
+      or /old-versions/). When the archiver tries to download the file, it
+      gets a '404 Not Found' error.
+
+   For (2) and (3), the problem is really the same: the web is a moving
+   target. Your download URLs and their md5sums were valid, but they got
+   changed on the server sometime after you submitted your build.
+
+   The solution is the same for both: find somewhere else to host your
+   source downloads. Either use your own web or ftp server if you have
+   one, or ask on the mailing list and someone will probably volunteer
+   to host it for you. Once you have the file(s) hosted somewhere,
+   update your .info file to point to the new location.
+
+   Before you do this, make sure the license allows you to: if it
+   doesn't allow redistribution, you can't host the download somewhere
+   else... and neither can we, so the build should be added to the
+   sbosrcarch blacklist (let us know if this is the case).
+
+   4. The file on the server is 'protected', because the server checks
+      the HTTP Referer and/or User-agent fields in the request. Typically
+      this means the download will work when using a browser, but will
+      fail when using wget or curl. Usually when this happens, one of
+      the sbosrcarch operators will manually download the file and add
+      it to the archive within a day or two. If not, let us know and
+      we'll get to it ASAP. Again, check the license of the download
+      file: if redistribution is not allowed, it should be added to the
+      blacklist and not kept in the archive.
+
 Q: How do I create my own archive?
 
 A: Two choices:
 
-   - Mirror the directory the usual way, using wget or rsync. Using
-     rsync is better!
+   - Mirror the directory the usual way, with rsync. Using wget
+     would be possible, but it would use about twice the bandwidth and
+     storage. This is because rsync supports hard links, which sbosrcarch
+     makes extensive use of.
 
    - Get a copy of the sbosrcarch script and run it on your web server.
      This will be more work on your part, but your archive will be
@@ -313,10 +362,6 @@ A: Two choices:
 
      The script lives here:
 
-     http://urchlay.naptime.net/repos/sbostuff/plain/sbosrcarch
-
-     ...or, it'd be better to use git:
-
      git clone git://urchlay.naptime.net/sbostuff.git
 
      It's written in perl, and has extensive documentation. Run it as
@@ -327,3 +372,15 @@ A: Two choices:
      entry in it) and I'd like it to include all the archives eventually.
      Also I'm pretty good at troubleshooting, if you're having problems
      with the script.
+
+Q: How much disk space will I need for my archive mirror/instance?
+
+A: Currently (2018-06-26), the archive is 93GB. The by-name and by-md5 trees
+   also seem to be 93GB apiece, but that's because hardlinks are used between
+   the two trees.
+
+   If you're using the sbosrcarch script to create your archive, you can
+   run a smaller (incomplete) archive. The config file (sbosrcarch.conf)
+   has a "maxfilemegs" setting. Any file larger that this, won't be
+   downloaded and archived. You can also blacklist builds (or whole
+   categories) to save space.