Sorry the index has been so outdated now. I am rewriting a0 version 3 recently and it is almost finished. The newer version will be efficient enough for me to afford maintaining another index source (my plan is to include AoPS). It also eliminates the impact of sup/subscripts in some cases, e.g., definite, indefinite integrals and sums. I also expect it to generate better search results because of refined relevance model. I will publish more updated index with v3 soon.

@Sil Let me know if you have collected some AoPS data using your crawler script, the crawling takes quite some time and I have not started yet. So it will be very helpful if you have saved something on disk.

@WeiZhong I think I have some still stored, let me check it.

Ok I have a huge portion of files crawled from High School math forum category (the one with most posts), however it is quite big and lots of files, i will try to put it into the zip file. By the way do you need the .html files, or just the .json files are enough? I assume the .html are just preview files and I can remove those, right?

By the way, I've found some updates made to the crawler which have not been pushed to the github, did it now (github.com/TheSil/search-engine/commit/…).

Anyway put the json files to drive.google.com/open?id=1_cI1NvUwz0YJMWBPuHbBucOKLICrOV4Z - check it out. It seems to contains category 6 data from April 22 2019 through July 22 2006 (by inspecting the posts crawled earliest/latest). Oldest post in this category seems to be around 2003, so it should be quick to crawl those, plus other categories need to be crawled as well.