I am writing you a PHP script right now, which will let you upload a sitemap, press go, and then check each link for a 404 (obviously I can't test it for you since it's a local site, you said.) It will output (one per line) the URL to the dead pages only. I could also make it show dead ones in RED and good links in GREEN. Let me know soon, before I finish the script. I'll post it as an answer after I'm done. If possible, can you upload your sitemap to Pastebin or something? It will help greatly.

also, if you want, you can simply provide the php code and I can paste it into a text file and upload - it sounds like this would be really helpful for alot of webmasters to be able to edit slightly as well to do the same thing! I don't know much PHP, but if there is anything you need or that I can help with, let me know!

I am going to make the source public for the community when it's done. Shouldn't be too much longer. Wait, check each page from that sitemap for dead links within that page (1 page to another) like open each page FROM the sitemap, scan the whole page for dead links? Or just find dead links from the sitemap itself?

the sitemap is dynamically generated and will be 100% correct, the problem is the individual pages within that sitemap will then have tons of dead links, so it would need to recursively scan each page within the sitemap - tried to clarify the question a bit, as I was worried you had interpreted it the other way. Basically whatever tool would need to spider all anchors on each page and scan those, probably ALOT more complex :P - I understand if you don't want to try to reach for that one :)

Hmm, yes I misunderstood. I'm still going to work on this, though it will take a bit longer. It will take quite a while to run, and use a good bit of memory. Though I'll even make it send a text file with the dead links to download, rather than output all of them on the page. I'm thinking hierarchical like "Sitemap link: (new line and indent) [dead link]" and list all dead links under the page it's contained in. Or is it better to list all unique dead links in one shot?

ya, its definitely alot more complex needing to recursively search each - some pages may have more than 1 dead link as well (though not frequently) - as for computing power, I would probably run this locally on my gaming pc (quad core I7, 4.9 Ghz, 32gb ram, 10Mb/s down, 5Mb/s up) - those specs should be hefty enough?

It won't necessary require that much CPU power at all, it just takes a while to run because it has to use HTTP to open each and every page, and scan that. Since it's PHP, it has to be left running and you'll know it's done only when my script prompts you to download the output. Best advised to set up your domain as "127.0.0.1" in the Windows Hosts file and use a local copy to scan.

hello

considering about 10k pages or so, each with around 250 average internal links... roughly 2.5 million requests

not sure how long each would take, but assuming 1 second each, thats like 25 days

That's pretty crazy. But I think cURL is faster than that, so I don't know. You'll have to experiment. I mean, it's a learning experience for me too. I've never made a tool to spider and check entire websites for 404's. Hopefully it works, and you get your site fixed. I'm at about 90 lines of code right now.

awesome, fortunately, I havn't hit the button that will break it yet, we get very minimal traffic, and once we switch to this new method, its going to go to hell

on a side note, we currently outsource alot of the tougher programming tasks (really tough javascript/PHP), but the guy we use here is a bit flaky, ill talk to the boss here about possibly checking into ionfish

"fortunately, I havn't hit the button that will break it yet" That sounds like the Big Red Shiny Button. Be careful. And make backups.

unfortunately, I have to get to a meeting with our data entry guy here in about 10 minutes, but I should be able to check in on this as needed etc - if you want, you can email me directly as well at [email protected] - I generally respond to emails within minutes if I am anywhere remotely close to a PC

I changed it so that you enter the URL of a sitemap rather than upload the file, because then you don't have to download it in the first place. And perhaps I'll adapt it into a tool on our site, but very unlikely because fetching "remote URLs" from "strangers" can be very, VERY dangerous. And thanks, I'll email you now so you have my email.

gotchya, ya, there are definitely security loopholes

we have one of those in our site that is a huge hole that lets any user load any url they want into our server, I just have been lazy to go fix it up, one of these days ><

plus xml sitemaps can be -slightly- different in formatting, ours uses exact google specs, but some might not

The script I'm writing only validates it's "valid XML" and will just extract raw URLs from it to test. It should be flexible for all.

ahh, ok

I would just worry about the server load it would put on your servers honestly

if you did find a way to securely host it haha

My servers? Don't worry about that. I could securely host it, but it would likely be a paid service. Someone could submit a sitemap like yours, and use like 500 megs of bandwidth a minute.

or somebody malicious could open 500 copies at once

Or try to remotely include a shell script and hack their way into the server? Or worse...

probably best to offer it as a one time paid service, they send you the XML URL, you send them the results :)

Use it to spam small websites so that they go offline.

ya, there are plenty of nasty uses for something that can brute force that well

But yeah, that could work. Not a priority, big project to be released Saturday.

I saw, some pretty snazzy screenshots

Thanks. But it's much better. Those are old snaps.

its nice to see people doing hosting outside of godaddy and the big boys

We have a lot of other stuff that's not on the site, mainly because it's private work for individuals and businesses. (Bank websites, etc.)

gotchya, I used to host a ton of gaming/ventrilo servers myself

Easy money -- Minecraft servers.

we have a really hefty project coming up we might try to get somebody to do - basically our main site can't run php at all, so we used an old domain to host all of our php scripts called from our main site, but we want to move that 2nd php site to a subdomain of the main site

so like every piece of code on the main site will need to change, not hard, just tedious lol

as for minecraft, I never saw the point of that game

I never played it, just that setting up a few headless linux boxes to run the game costs almost nothing.

ya, I imagine, its not exactly a high-maintenance server id imagine

And people pay good money for it.

thats the part I don't get haha :P

20 bucks a month for like, 8 slots, static ip.

anyways, ive got to get going to that meeting, ill hopefully be able to jump back on once I am back at the office

Sure, talk to you later. You can email me too, I sent a mail to you.

Hopefully you got it.

ya, ventrilo servers are similar, about 5-10 for 8 slots, and its voice only - easy easy money

I did

Ok, so far, it downloads the remote file and tells whether it's valid or not, and maybe 20% done development. So in a few hours? Bah. PHP is fun. Time is not.

Now, it extracts all URLs from the XML file, and opens them all up and searches yet again for all the links inside the pages themselves. Almost done!

90% done!!! It's working perfectly, just have to do some formatting and make it not timeout for massive sitemaps like yours.

Alright, a few bugs. I have to still make it ignore duplicate links, and not fail with #links like that.