diff options
| -rwxr-xr-x | sbodl | 3 | ||||
| -rwxr-xr-x | sbosrcarch | 4 | ||||
| -rw-r--r-- | sbosrcarch.faq | 97 |
3 files changed, 80 insertions, 24 deletions
@@ -90,7 +90,7 @@ for dl in $DL; do else wget $WGETARGS $EXTRAWGETARGS "$dl" || die "Download failed" if [ -e "$FILE" ]; then - mv "$FILE" "$CACHEDIR" + mv -b "$FILE" "$CACHEDIR" ln -s "$CACHEDIR/$FILE" "$FILE" fi fi @@ -105,6 +105,5 @@ for dl in $DL; do else echo "WARN: can't find downloaded file $FILE" fi - echo shift done @@ -1192,8 +1192,8 @@ sub handle_info_file { my $filename = url_to_filename($url); print ": $url\n"; - if(exists($url_rewrite_hacks{$category/$prgnam})) { - $url = $url_rewrite_hacks{$category/$prgnam}->($url); + if(exists($url_rewrite_hacks{'$category/$prgnam'})) { + $url = $url_rewrite_hacks{'$category/$prgnam'}->($url); } if(already_exists($filename, $category, $prgnam, $md5)) { diff --git a/sbosrcarch.faq b/sbosrcarch.faq index 19b30ac..88b0e2c 100644 --- a/sbosrcarch.faq +++ b/sbosrcarch.faq @@ -32,12 +32,12 @@ A: The archive server is operated by Darren Austin, aka "Tadgy" archive healthy. The best way to contact us is using an IRC client to connect to - Freenode and join the ##slackware or ##slackbuilds channel. + Freenode and join the ##slackware or #slackbuilds channel. We can also be reached by email: B. Watson <yalhcru@gmail.com> - TODO: make sure Tadgy is OK with his email being here! + Darren Austin <mirrors (at) slackware.uk> Please read this entire FAQ before asking us questions. Chances are, you'll find the answer here. If not, or if the answer isn't clear @@ -95,10 +95,10 @@ A: Several answers here: you'll see a list of build name directories. Each of these will contain the source file(s) for the build. - Example: you can't download the source to system/atari800, so - you go to the by-name page, click on "system", then "atari800". - There you'll see the file you wanted, atari800-3.1.0.tar.gz (unless - it's been updated since I wrote this). + Example: you can't download the source to system/atari800 + from its original URL, so you go to the by-name page, click on + "system", then "atari800". There you'll see the file you wanted, + atari800-3.1.0.tar.gz (unless it's been updated since I wrote this). - With a download tool like wget or curl. You could do this using the same by-name tree as you would for manual lookups, but it's better to @@ -244,8 +244,7 @@ A: Multiple answers: can remove and blacklist it. It is not our intention to violate anyone's license. -Q: Why does the status page say the by-name tree is missing 4 files, but - the by-md5sum tree is 100% complete? +Q: Why do some of the by-name directories have filenames ending in ".x86_64"? A: This is due to a design flaw in the archive structure. We assumed that download filenames would either be unique within an .info file, or else @@ -258,12 +257,14 @@ A: This is due to a design flaw in the archive structure. We assumed that DOWNLOAD_x86_64="https://www.perforce.com/downloads/perforce/r18.1/bin.linux26x86_64/p4" Notice that both URLs end in "/p4". The directory parts of the URL are - different, but the filenames are the same. + different, but the filenames are the same. In the archive, the 32-bit + download will be called "p4" and the 64-bit one will be "p4.x86_64". The archive script successfully downloads these files and stores them in the by-md5 tree in the correct directories. But when it tries to store them in the by-name tree, it's trying to save two files in the - same directory with the same name. The second one overwrites the first. + same directory with the same name. If it didn't use a different name, + the second one would overwrite the first. The current list of builds affected by this is: @@ -272,10 +273,6 @@ A: This is due to a design flaw in the archive structure. We assumed that development/p4d libraries/p4api - Eventually this will get more-or-less fixed: for these builds (and only - these), there will probably be separate x86/ and x86_64/ subdirectories - (e.g. development/p4/x86/p4). - Q: I'm a SlackBuild maintainer, and the download URL for one of my builds has disappeared. Can I use the archive URL as the DOWNLOAD in my .info file? @@ -299,12 +296,64 @@ A: Yes, but only as a temporary measure or a last resort. every source file to have two working URLs: the original plus the sbosrcarch one. +Q: I'm a SlackBuild maintainer, and one of my builds keeps showing up + on the sbosrcarch STATUS as missing. How can I prevent this? + + This usually happens for one of these reasons: + + 1. You made a mistake in your submission. Double-check the DOWNLOAD URL(s) + and MD5SUM(s) in the .info file. If they're wrong, resubmit your build. + + 2. The filename in the download URL is "unversioned", meaning the version + number isn't part of the filename (e.g. "thingy-latest.tar.gz"). At + some point after you last updated your .info file, but before the + SBo public update, the file changed on the server. Actually, this + occasionally happens even for files that have the version number + in the filename: upstream makes a mistake (leave a file out of the + tarball for instance) and a day or so later, they fix it without + changing the version number. When the archiver downloads the file, + it checks the md5sum against your .info file and sees a mismatch, + so it won't archive the file. + + 3. Upstream made a new release after you updated your build, but before + the SBo public update, and they removed the old version from their + server (or, possibly, moved it to a different location like /archives/ + or /old-versions/). When the archiver tries to download the file, it + gets a '404 Not Found' error. + + For (2) and (3), the problem is really the same: the web is a moving + target. Your download URLs and their md5sums were valid, but they got + changed on the server sometime after you submitted your build. + + The solution is the same for both: find somewhere else to host your + source downloads. Either use your own web or ftp server if you have + one, or ask on the mailing list and someone will probably volunteer + to host it for you. Once you have the file(s) hosted somewhere, + update your .info file to point to the new location. + + Before you do this, make sure the license allows you to: if it + doesn't allow redistribution, you can't host the download somewhere + else... and neither can we, so the build should be added to the + sbosrcarch blacklist (let us know if this is the case). + + 4. The file on the server is 'protected', because the server checks + the HTTP Referer and/or User-agent fields in the request. Typically + this means the download will work when using a browser, but will + fail when using wget or curl. Usually when this happens, one of + the sbosrcarch operators will manually download the file and add + it to the archive within a day or two. If not, let us know and + we'll get to it ASAP. Again, check the license of the download + file: if redistribution is not allowed, it should be added to the + blacklist and not kept in the archive. + Q: How do I create my own archive? A: Two choices: - - Mirror the directory the usual way, using wget or rsync. Using - rsync is better! + - Mirror the directory the usual way, with rsync. Using wget + would be possible, but it would use about twice the bandwidth and + storage. This is because rsync supports hard links, which sbosrcarch + makes extensive use of. - Get a copy of the sbosrcarch script and run it on your web server. This will be more work on your part, but your archive will be @@ -313,10 +362,6 @@ A: Two choices: The script lives here: - http://urchlay.naptime.net/repos/sbostuff/plain/sbosrcarch - - ...or, it'd be better to use git: - git clone git://urchlay.naptime.net/sbostuff.git It's written in perl, and has extensive documentation. Run it as @@ -327,3 +372,15 @@ A: Two choices: entry in it) and I'd like it to include all the archives eventually. Also I'm pretty good at troubleshooting, if you're having problems with the script. + +Q: How much disk space will I need for my archive mirror/instance? + +A: Currently (2018-06-26), the archive is 93GB. The by-name and by-md5 trees + also seem to be 93GB apiece, but that's because hardlinks are used between + the two trees. + + If you're using the sbosrcarch script to create your archive, you can + run a smaller (incomplete) archive. The config file (sbosrcarch.conf) + has a "maxfilemegs" setting. Any file larger that this, won't be + downloaded and archived. You can also blacklist builds (or whole + categories) to save space. |