@Sil have you started over crawling AoPS again? I am afraid my VPS does not have enough memory or at least it may be the cause of the server fault last time (it is weird I couldn't find kern.log last time but after stopping the crawlers the search program never got killed again). So I am afraid I can help little on crawling this time. I wish your network connection is much faster than what I have at home.

I have not, but I can start it (it takes couple of days to fetch whole category 6 i think)

Ok started crawling category 6

by the way the colored signs ([94m etc) do not work in windows console

might want to replace it with something like pypi.org/project/colorama

Btw, @WeiZhong maybe this is a bug? When searching this: approach0.xyz/search/… , the two top results are different MSE posts, but approach shows same text for them, but the text really belong only to the top one....

Even the title does not match, it says Family of irreducible polynomials for both, but one of them should be "Irreducible polynomials?"

@Sil Cool. I may need your corpus later to update official index :-)

@Sil I don't think I want that, adding another dependency for visual looking does not offset the potential maintenance cost.

We can remove color in AoPS crawler if you want.

Nah it does not really matter that much

But talking abou dependencies

@Sil I am not sure if it is because MSE redirects crawler request to duplicate question. That is the only possibility I can think of.

it would be nice to have requirements.txt file somewhere near crawlers so that anyone can easily download them using pip install -r requirements.txt

@Sil Agreed, you can add that if you want, I will pull. For now it is not open source, literally one or two may benefit.

Why not make it open source by the way? You can choose suitable license if you want to avoid "stealing"

Btw that seems strange handling of duplicates, does not seem like this should be wanted...

@Sil I really do not know. One time I want to mine the value myself, because I personally know several startups/people tell me they want this, and searching exam questions is a huge market in Asian I have to admit (many companies are already doing it, without hiding the way they help students avoid thinking), I just think may be there is chance I can start a business to utilize this project one day.

Sometimes I also want to share the code because I love doing open source.

Maybe separate to repositories accordingly, I guess only a part of the repo does the actual trick, like the indexer or something, idk...

The crawlers/web seems to be a good canditate to move out, it is already in demo folder :D

This is what I have right? The old open repo is still there.

I mean the components of approach, not old version of approach

Sure, if you want make, a pull request with only crawler/web folder to the public repo, I will pull.

Nah I dont, i am too lazy for that

just giving ideas :)

But you never know, sometimes i get bored and i do stuff

I found there is a few people interested at participating this project when it is still open source, you are one of the few, so that makes me a little tired to improve the old repo too.

But I am sure if I merge your recent changes to the old repo, more people will recognize your contributions to A0 since it is publicly available.

I dont really care too much about that :) It is enough that changes are in deployed version. The changes i mentiond are more to handle the private/public issues for you, as you said you like doing open source.

And yea it is a shame you don't have more contributors, not sure why.

For this kind of non-mainstream project, unlike some popular framework code, it is hard to get sponsored for our efforts (and its value is not so obvious, plus many people tend to avoid math), so I believe keeping private and inviting people to contribute will serve my (or our) current interests.

Is the money really the issue? I mean, it was mentioned in past, just make paypal contribution available and people will donate :)

Or maybe not paypal, I am not sure what are the alrenatives, i personally like paypal so that i do not expose my credit card number to anyone

I will open donation link later, now there is not much user base (80 IP/day at best?).

yea but you mentioned your vps cant do that and that, and has low memory etc

you could perhaps afford better one

Sure, I also plan to give paypal link among others.

i and use should scale with content, and advertisement. The aops although it adds new content, i guess we cannot openly share this with AoPS community (i doubt they would like the crawling we do:D), but for example if mathoverflow would be added, you would have them

Right, but the low-end VPS is sufficient for 90 IPs a day at least for now. So I will wait a while to consider add donation, because when I do that, I probably will blur a few search results and reveal only to sponsors/developers just to recognize their help.

Of course for professional mathematicians it would be best if they could search stuff in various released journals, that would be ultimate :o

But indexing journals would be brutal, they are pretty long compared to the current MSE/aops content, it would be challenge, and its just too much of it imo...

It is always easy to add another SE-like data source (just need to modify MSE crawler I think). I want to wait and see if users are happy with v3 improvements and I feel more comfortable to ask for donations when I see more people using it.

Then perhaps announce it on MSE meta?

That is a way you can get feedback

@Sil Not bad idea, keep a few days waiting, just want to make sure it does not crash that often as before.

right :D

and maybe the AoPS crawler will finish by then

although it is just one category

I can guarantee you already that one feedback you will get is unability to filter sources, people use approach often to find duplicates on MSE, for that is it good to enable only MSE posts

i wonder if in order to support the filtering, will the crawlers need update?

(seems not, you already have an mse/aops prefix there)

@Sil Right, I can kinda predict that too :P, but progress takes time. Looking back, this year is already a huge jump forward for Approach0, the version before cannot even find common subexpressions in the query.

yea i dont know what that means, dont forget to include examples when you announce it on mse :D

@Sil Maybe... That also saves my pocket, I like your idea.

@Sil Not necessarily true. The engine can only use default field from JSONs and digest other fields if they present.

Remember it is JSON, very powerful :)

yea, well it still has url in it, where you can see artofproblemsolving.com etc

shouldn't be problem to categorize based on that

Right

I have to have dinner now, chat you latter.

Sure, later

It is interesting. If you login SE, visiting this link (math.stackexchange.com/questions/3330069/…) goes to 3330069, however if you use incognito mode, you will be directed to duplicate post.

wow

didnt know that :)

I think this is a policy made by SE so to improve quality of search engine results shown for that site.

maybe the behavior can be altered by disabling redirects or something

Since user uses google can find the original duplicate which presumably has more content (because it is not locked)

The redirection page itself may not contain the actual content, so I guess you will need to login to see the locked post.

:S

As explained there if one wants to disable the redirect one can add ?noredirect=1 to the URL.

@SirCumference hi.

I've tried it and it works

@Sil that is very helpful.

@YuvrajSingh... Hi

I have a question, I do not know where I am making mistake.@Sil

But I still wonder whether we need to crawl data from duplicate posts, usually no useful data there.

If you can guide I will be more happy.@Sil

@WeiZhong I think so, because often there are lots of duplicates with interesting content, and they are marked as duplicate years later

@Sil OK, I will add that parameter into crawler when I get time.

@YuvrajSingh... Something related to searching? Anyway I am about to go for lunch now, but ask here

No, it is related to mathematics.?

@Sil a quick one.

@YuvrajSingh... if it is a math question, maybe you can just post on the main site?

@Sil I think your avatar looks like Euler, that's why people chasing you to ask question :P

Just a guidance.

Not complete answer.

10^(11^(12^13)) mod 11 this was the question.

Aha, it is indeed a question for Euler. I cannot help you.

Use Euler's equation. Haha

Now I can justify it natural, I was thinking for some better and reasonable approach.

@WeiZhong I have to solve it without any equation.

Because I haven, t study them yet.

10 = -1 mod 11, so 10^(11^k)=(-1)^(11^k)=-1=10 mod 11 (since the exponent is odd)

I got seriously question on k.

How do I define k.

He just say, it is a guess.

@Sil

One random suggestion, would it be possible to print the score of search result match somewhere near to it? That way we would have at least some idea how the results compare to each other (at least from approach perspective).

Also, over time this might give us an idea of some possible threshold that could separate relevant results with trash results (if I am correct, there is now always same number of results, regardless of the actual score).

Another random suggestion, if there was a way that user could mark result as a good match vs bad match, this gould be later useful input also for having idea for the threshold, as well as a way to report some potential issue

@Sil Well, this is something which moderators can do.

To be more precise, moderators can request this from the CMs. There are only a few parts of help-center which can be edited directly by the mods: What parts of the help center can site moderators edit?

I have reminded this to the mods some time ago:

Maybe at some point we should bother them again.

In the meantime, there are probably still some improvements that can be done to the faq-proposed post How to search on this site?.

I have mentioned some possible improvements here: chat.stackexchange.com/transcript/46148/2018/12/18

Some further suggestions:

1. Would it be good to add to the question sort of index/toc linking to the answers?

2. Should we add an answer with other math-aware search engines? (I am only aware of SearchOnMath.)

I think that Approach0 deserves a separate answer. It was created specifically with Mathematics in mind. (Although eventually probably more sites will be added - AFAIK so far only AoPS.)

3. In the answer about Approach0 I specifically mentioned the difference between "+\infty" and "\infty. Perhaps also difference between searching for |x| and \left|x\right| should be mentioned.

This was discussed here: chat.stackexchange.com/transcript/46148/2019/12/5

@MartinSleziak Those are good suggestions, I am not sure about SearchOnMath, I've tried it once and didn't find what i needed, but why not add it, its value is given by votes it gets anyway.

I was thinking about oeis.org, but the meta post seems to be only abou searching on this site, so no i guess

I should probably post a few links here to make comparison between Approach0 and SearchOnMath easier.

Right

maybe one think that is missing in the meta post is reference to this very chat room?

oh nevermind, i see it in comments of the question

It is also mentioned in the top-voted answer:

> Although the "searching" chatroom has a more general purpose, quite a lot of the conversation in that room was devoted to Approach0.

It looks like it has all the important stuff

I have added a new answer with other search engines. (Although the only one I know is SearchOnMath. There used to be uniquation, but it does not exist anymore.)

Is somebody has a bit of spare time, you can edit the answer about Approach0 to mention that |x| and \left|x\right| returns different results.

The link in that answer seems to be broken

I am not really sure why this link is rendered wrong in the answer: "search for $\sum_{k=1}^n \frac1{k(k+1)}$".

Hm it cannot search directly \sum_{k=1}^n \frac1{k(k+1)} or $\sum_{k=1}^n \frac1{k(k+1)}$? The {} seems extraneous

Anyway now I see why it couldnt find anything before, i just entered standard latex/mathjax input

The link you given is still broken though, it has extraneous dollar at the end, i think it should be searchonmath.com/… ?

I see what you are struggling with, maybe problem is with nested () ? In "(searchonmath.com/…;

@MartinSleziak I've modified the url in the anwer if you dont mind, now it seems to look and work correctly

Thanks!

I am not really sure what I was doing wrong. (I have copy-pasted an url from the browser which seemed to work.)

your url for some reason had ( and ) in it, mine did not, unsure why

Is SearchOnMath able to recognize basic subsitutions? like k for n, etc?

Does this work at least in chat?

Test: $\sum_{k=1}^n \frac1{k(k+1)}$: [Approach0](approach0.xyz/search/?q=$\sum_{k=1}^n \frac1{k(k+1)}$), [SearchOnMath](searchonmath.com/result?query=$\sum_{k=1}^n \frac1{k(k+1)}$)