@WeiZhong Hey, just looking at journals format at arxiv and it seems easy to get it in latex format automatically, and so is "detexing" it and use it for indexing. I don't remember, was there any legal (or any other) reason why not indexing the arxiv? It would be super useful for research level mathematics, if I have enough time on my hands I could do a dirty crawler for that, but only of course if it will be used (The AoPS also ended up unused)

I recall that you discussed this before: chat.stackexchange.com/transcript/46148/2019/6/12

Oh thanks, I knew I remembered something about it somewhere, so "arxiv has data available too, but it has license to exclude re-distribution." is the answer

Also I am not sure if indexing counts as re-distribution

This is probably the question for somebody who knows more about copyright issues and various licenses than I do.

Could contact them directly I guess, unless @WeiZhong already did

Certainly the arXiv license do not prevent it from being indexed by Google (and Google Scholar).

Good point...

I think that for some time I saw results from AoPS in Approach0 searches. Or was it only in some unofficial/testing version? (It's also possible that I remember it wrong.)

Yes, it was there twice, first only on test server, then I believe even on official site for a moment, but it was removed for some reason (I think due to resources or something like that)

AoPS was mentioned several times in this room. This starred messages mentions also MathOveflow.

And the site never really had a much more support for more sources (it is quite MSE centered). Something like selecting sources would be nice.

"If things go well, I can further include AoPS and MathOverflow source as index" So I guess things didn't go well :D

Too sad, I was ready to contribute (I think you did Martin) once it was on paypal or something more user-friendly, but it seems he gave up on setting it up

Nowhere to be found on official site :o

And one more thing related to approach0, it seems that questions are not automatically indexed (I know in past it used to be once a week or something like that). I can find posts from april that cannot be found...

Or maybe I've just picked wrong ones which do not show in search results for some reason

@Sil Hi Sil, I always appreciate your AoPS crawler script. I could index it but current search engine is not efficient enough to run larger index while maintaining good query process time, unless more machines are deployed. I am working on improvement of the core engine recently, it is a slow progress, but I would love to add AoPS source once I feel the engine is cost effective to have more data indexed. After all I have to pay the bill for all the server cost.

I would rather wait until an efficient and satisfying core engine comes out, before I add a lot of data. Also I am busy with some personal issues so the progress is slow.

Since you are the developer member of Approach0 private repo, you can see I am rewriting some core functionalities, and I believe these rewrites will improve the efficiency and accuracy of approach0 to the next level.

@Sil As for the donation option, Martin's contribution has already supported Approach0 for one month, later I decide to use my own money to keep Approach0 free, mostly because I do not feel satisfied about the current core engine, there are new ideas that I am excited and now being explored.

One fact good to know, the current version is fast partially due to it cost 2 machines to distributively host MSE index, and because the current model can search subexpressions of query, it is much costly than the version you see one year early, so the VSP I pay each month personally is 50$ just for hosting MSE. It should be quite obviously that given my limited time and personal budget, I would rather wait for a more efficient search engine before I put arxiv, mathoverflow, AoPS there.

@Sil Yes, I did not do automatically index. I could do that, but currently I really have to deal with some personal issue, and I am paying much more attention to the design of a more efficient engine, so sorry, I have to adjust to the best I can.

If anyone is interested to host a new A0 instance that does index automatically (maybe more index source included), there is an utility repository I've written to help you automate things (but not everything), it is quite time consuming and requires patients to check every step runs correctly, and it is really a burden for me right now to do each week.

BTW, the mockup donation UI is already there: link, feel free to try clicking the Fibonacci Sponsor button and see the faked payment process.