Also, I have just added functionality to crawl comments (but only those statically displayed on a post, not those fetched by clicking "show N more comments"): github.com/approach0/search-engine/commit/…
Thank you. Ideally if we have enough computers, we can distributively crawl the entire MSE within hours. But sadly I am using one of my personal computer doing this, so if there is a bug in crawler script, it is really time-consuming to fix and update the entire corpus.
Since this is a modification to core ranking formula, this change will apply to any query. Please let me know if you observe other cases that do not work well after this change. If this change has side effect, I may consider to replace or reinvent the current ranking formula entirely.
@Workaholic Good question, first, unlisted TeX command (those starts with a backslash) such as \limits and \displaystyle in Approach0 parser will be omitted by this rule: github.com/approach0/search-engine/blob/master/tex-parser/… . Thus will have no impact on search results.
I can tell you that the most popular sites in China about math Q&A is some mobile APPs that help student search homework questions. (to my knowledge, they are using deep learning to process photos uploaded by students) And these APPs are getting extremely popular in China
@Workaholic Some of you probably do not know that StackExchange is not easily accessible in China because it is using JQuery (a Javascript library) hosted on googleapis.com which is blocked in China.
@Workaholic Good suggestion. I also want to add this and show users query history from other users. I am pretty busy these days, but will definitely implement this feature in the next version.
@MartinSleziak Before I was releasing the first version, I advertised many places on the Web. Which brings in a lot traffic at first (AFAIK, initial traffic comes notably from Reddit: reddit.com/user/ga6840). Just now, I logged onto server to have a look on recent query log, the number is about 3k since sep/23, so the traffic goes down a lot. And most recent log interval records that ~100 queries between the day of 11 and 14, so it is about 25 queries/day.
Update: After evaluating this issue in depth, I conclude that this workaround will add if conditions into hot code, which can impede parallelism thus reduce efficiency. So sacrificing overall efficiency to eliminate one bad hit is not a good choice. Therefore, I choose not to fix this problem. I would love to explain this decision in detail upon request.
@MartinSleziak Yes, the search result is expected to be the same using your examples. This is because Approach0 parses a TeX into "operator tree" in which changing the order of subscript and super script or enclosed a TeX bracket (i.e. "{}") does not affect the "operator tree" structure. So many times when I say "structurally relevant", I mean the operator tree representation of query is the same or subtree of the operator tree representation of another expression.
@MartinSleziak Right, so I have modified that EDIT, I agree that those users who have search problem should stay here and ask, but to push appraoch0 forward and get developer involved, bugs and feature request should be posted on GitHub in my opinion.
That is not strictly a bug, it is because I put a restriction on the number of document to be searched in one posting list (not trying to be very technical). I have pushed my new code to GitHub (see commit github.com/approach0/search-engine/commit/…), this change gets rid of that restriction and now that relevant post is shown as the 2nd search result.