Snarkpit Forum Index » General Banter » Dead links!
Dead links!
1
Post Reply
Quote
 Re: Dead links!
Posted by eukara on Mon Mar 21st at 11:48pm 2022

List of 'missing' maps

When scraping snarkpit.net in its current form I went through and noted every single map that I failed to download in any capacity (checking for .zip, .rar, .exe, .tar and .gz archives)

NOTE: It lists some recent maps too, which do have 'working' links but are behind some trickery that didn't allow me to download it. Things like Google Drive URLs or external landing pages on TWHL and ModDB. While those are not 'dead' they'd be good to mirror locally as well.

I did not count maps that had no mirrors listed. Although it seems some used to have download links at some point. archive.org suggests that some of that broke during the 2009 update?

The count of maps that are marked as missing is as high as 1508, which implies that 42% of maps are missing...

Anyway, hope this helps in any way.



Quote
 Re: Dead links!
Posted by Oskar Potatis on Tue Mar 22nd at 12:04am 2022

If someone tries to fix those downloads...

I'm slowly working on restoring dead downloads at TWHL.info (halfway done). I have a list of links I've used to find maps, WADs, models etc. that I plan to sort through, but some of the more useful downloads and sites have been:
https://aghl.ru/files/maps/all_maps.zip
https://www.moddb.com/mods/twhl-maps-and-mods/addons/half-life-dm-gold-multiplayer-maps-century-megapack1
https://www.17buddies.rocks/17b2/index.php (careful with this one, they have a HUGE number of maps but the .res files are often wrong and the map downloads are often missing files or including extra files that are already included with the relevant mod)
https://www.17buddies.rocks/17b2/View/Wads/Gam/1/All/Pag/1/textures_Half-Life.html (use the lower of the two search boxes to search for WADs)
https://ds-servers.com/ (great place to search for missing sounds, models, skies etc.)
http://www.half-life.fr/cs/
This one has lots of HLDM maps but no way to list them so try adding mapname.bsp to the URL (remember to download the .txt, the overview and any resources needed including *t.mdl models and player models if it's a teamplay map): http://www.maps.ltkclan.org/CUSTOM_CONTENT/maps/
Also http://scmapdb.com/ https://www.moddb.com/ https://gamebanana.com/ but those go without saying

Remember that there sometimes are multiple maps with the same name by different authors.



Quote
 Re: Dead links!
Posted by Xylemon on Tue Mar 22nd at 6:17am 2022

Awesome work and info Oskar! One thing I wanted to add is that we should also run direct urls through archive.org since there is always a chance it got crawled. For instance, this map had a direct URL that was captured:

https://www.snarkpit.net/index.php?s=maps&map=1486

If we run the original URL like so:

https://web.archive.org/web/0/http://homepages.nildram.co.uk/~henleyb/ph_machine_02.zip

It'll find the first capture:

https://web.archive.org/web/20060323224619/http://homepages.nildram.co.uk/~henleyb/ph_machine_02.zip

This can be automated too and then curate the results. We can also add the archive.org link as a mirror as well.




Quote
 Re: Dead links!
Posted by Riven on Wed Mar 23rd at 2:29am 2022

This is phenomenal work!

If you are checking these links, selecting the dead link button on a map's detail page adds it to a separate list that makes it easy for admin on the site to replace those links. It's the little triangle button shown below.



Unlike in the early days, we have plenty of hosting space, so it would be really great to get those maps added back for good to the site.

Without opening each individual map though, what ought to be the process to validate them before reinstating them with their records?


Blog: www.playingarchitecture.net
LinkedIn: Eric Lancon
Twitter:@Riven202



Quote
 Re: Dead links!
Posted by eukara on Thu Mar 24th at 7:34am 2022

Quoting Riven
Without opening each individual map though, what ought to be the process to validate them before reinstating them with their records?


If you're talking about 'what should qualify'...
Trying to avoid repacks if possible (most repackers, like 17buddies, will pollute the archive with their own README) but if that's the only thing we can find they should probably be marked as such.

As in how everyone should be to contribute?
Well considering the volume of maps - this is a huge undertaking. I thought about running curl and just spamming the 'report missing link' php request but I'm not sure that'll help any of you administrators get them fixed either - you'd still have dig through archive.org or alternative sites for 1500+ files, many of which we don't know the filename of either if they weren't direct-links.

If I could snap my fingers, I'd have a site up and running where people could select from a list of maps considered missing and submit a working URL, which could then get checked by an administrator. Crowd-sourcing this would be the answer to the volume of missing files.

This weekend I may have some more things to show in that regard. Will keep you all posted.




1
Post Reply