Archiving images, dead links

45

Q: Restoring Bill Thurston's figures in an answer

The question Floating polyhedra with fair equilibria was answered by Bill Thurston, but his (typically amazing) answer contains links to 7 figures, all now broken. I don't know how to retrieve those images from the past, but if someone does know how and is willing to spend the time & effort, it w...

In connection with the above question, archiving MO content was mentioned a few times.

I wish posts/answers like this would be archived by archive.org (with deep links). Unfortunately it's not the case, it seems. — Dima Pasechnik Jun 29 at 8:18

@DimaPasechnik: Sometimes archive.org does provide the Dropbox images: see for example Thurston's Is there an underlying explanation for the magical powers of the Schwarzian derivative?, Can we add two matrices by performing an operation on their eigenvalues & eigenvectors?, A random walk on random lines, and Volumes of n-balls: what is so special about n=5?. Unfortunately this did not happen for the OP's list of 16 posts. — jeq Jun 29 at 14:23

whether or not archive.org has crawled an MO page appears to be a random event; I have over 300 images in MO answers and about 50 of those have been copied to archive.org; perhaps there is a way to instruct archive.org to systematically copy the whole MO site, but probably this has to be arranged at some higher level; I am now instructing archive.org "by hand" to copy all of these 300 images, one-by-one: I have not found out how to do this in batch. — Carlo Beenakker Jun 29 at 15:21

IMHO it looks like a good idea if Mathoverflow (and perhaps other stackexchange sites) join archive-it.org (putting on Wayback machine sufficiently highly scored questions and answer looks like a good idea) — Dima Pasechnik Jun 29 at 22:36

There is a separate thread about this: Archiving of MathOverflow threads

13

Q: Archiving of MathOverflow threads

We've had some discussion of adding an archiving feature to MathOverflow on the private moderators list, and I thought I should report on the current situation. (You might also read earlier threads Long-term archiving of MathOverflow, Should top mathoverflow threads get DOIs and be permanently ar...

discussion

Archive-It was also mentioned in another topic: tea.mathoverflow.net deprecated

10

A: tea.mathoverflow.net deprecated

I would really hope we can find a more permanent solution for this content of archival value; perhaps via Archive.org using the "Archive-It" functionality ? I could even imagine using this service in order to preserve other MO content for posterity. Some relevant points from the FAQ: What is Arc...

However, it seems that discussion here is mainly about archiving external link to images (and maybe more generally other sources) that are outside MO (outside SE network).

I have recently looked for a few dead images.

S. Carnahan edited his answer after I pointed in a comment that a dead link is available in the Internet Archive: mathoverflow.net/posts/21671/revisions The geometry behind the ICM 2010 Logo

I have edited the question Finding 3 dimentional B-spline controll points from given array of points from spline solution?

For this one I only left a comment:

Links in the post seem to be dead, here are links to the Wayback Machine: bunny.jpg and bunnyDelaunay.jpg. — Martin Sleziak yesterday

dead-link images I have tried search for images after seeing a comment from YCor:

It might be useful to chase for links to figures, maybe starting from most upvoted answers, beyond the case of Thurston, so as to anticipate on possible missing figures in the future. — YCor Jul 1 at 19:03

I did at least some experimenting with jpg files in MO posts. (Of course, if somebody decides to do this regularly, other extensions should be considered, too.)

Search for urls containing the string jpg returns 2001 posts.

I get similar number of posts when I search for links with jpg ...

...or links with .jpg using SEDE. (There are certainly a few false positives.)

For the purposes of extracting links, query which only returns body might be more suitable: data.stackexchange.com/mathoverflow/revision/1075311/1328325/…

If somebody wants to check the links which were added recently, it might be reasonable to restrict posts by LastActivityDate: data.stackexchange.com/mathoverflow/revision/1075315/1328339/… data.stackexchange.com/mathoverflow/revision/1075319/1328343/…

Anyway, I tried the query with the jpg extension: data.stackexchange.com/mathoverflow/revision/1075311/1328325/…

I downloaded the results and extracted urls ending with jpg. And then I omitted the ones on i.stack.imgur.com - since those pictures should exist indefinitely, at least in theory.

Here is the list of files that I got: pastebin.com/7NRv0B5c

I have tried to download them using wget.

I had to omit one of the links manually - I am not sure why the wget always stopped this one: icm2010.org.in/wp-content/icmfiles/otherimages/icm_top_logo.jpg. But this image has been edited away in the meantime.

It takes up about 50 MB of space. And in the log it's possible to check which files were not found. (Or possibly, if I try to do the same thing in a few months, I would see which of the files that I downloaded this time no longer exist.)

Of course, some pictures might be protected by no-robots. Some pages might be down temporarily. This is definitely not 100% reliable.

Doing something like this is about as far as my programming skills go. I suppose that some people would be able to automate this or make some web applications that check all new and edited posts on MO for links.

Anyway, from this experiment we can see at least that there are not that many pictures with direct links on MO. (By which I mean that they were not uploaded using the editor.)

Naturally, if somebody wants to really use this, they should also chekc for other formats.

It would be much bigger undertaking if somebody wanted to save all pdfs linked from MO: mathoverflow.net/search?q=url%3A%22%2Apdf%2A%22

SEDE: data.stackexchange.com/mathoverflow/revision/1075306/1328319/… data.stackexchange.com/mathoverflow/revision/1075311/1328325/…

MathOverflow

Participants