« first day (1172 days earlier)      last day (1891 days later) » 

00:00
test 2: [SearchOnMath](searchonmath.com/result?query=$\sum_{k=1}^n \frac1{k(k+1)}$)
@Sil I would hope so. We can try your link with j instead of k: search for $\sum_{j=1}^n \frac1{j(j+1)}$.
Sil
Sil
Seems to work
In that case, already the first page of results shows both $\sum_{j=1}^n \frac{1}{j(j+1)}$ and $\sum_{j=1}^n \frac{1}{k(k+1)}$.
IIRC Approach0 has higher preference for exact match, but searches also for expressions with other variables. Perhaps it's similar here.
Sil
Sil
hm is that engine still maintained? pages indexed? it seems that latest year mentioned on the website is 2015
You mean SerachOnMath?
Sil
Sil
yea
it looks like abandonware a bit, trying to search for stuff added like 1 or 2 years ago and i cannot find it
00:09
Flavio Gonzaga made posts about the site on meta and on main in January/February.
Since they were advertising it here, I suppose it worked at least at the time.
Sil
Sil
well, simple example, this post
2
A: Is $x^j+x^k+2$ irreducible whenever $j+k$ is odd?

Sil Can we use the bound of the absolute values of the roots and that the constant coefficient is prime to show that $f(x)$ must be irreducible in $\mathbb Q[x]$ Yes, that is indeed exactly how we can prove the irreducibility of $x^j+x^k+2$, by more closely inspecting its (complex!) roots. Fir...

nothing...
oh my bad...
the irreducible keyword was an issue, not sure why, but without it, it finds it
I don't think SearchOnMath support keywords and multiple formulas - as Approach0 does. But take my words with grain of salt - I haven't really played around with SearchOnMath that much.
I should get some sleep, it is already past midnight in my part of the world. See you later!
Sil
Sil
yea here too
see you
BTW uniquation was also mentioned in a few posts on main and on meta.
But it seems to be gone:
Oct 19 '18 at 16:57, by Wei Zhong
@MartinSleziak Right, uniquation is down for a long time.
Sil
Sil
00:31
@Wei
nevermind :)
 
7 hours later…
07:36
Good news, Linode has agreed to sponsor my hosting cost for one year.
07:49
@MartinSleziak I have just added that bit. See math.meta.stackexchange.com/questions/29265/…
07:59
@Sil My guess is they separate searching process into mathematical phase textual phase (as they state at the bottom of their search result page). And it seems they filter "irreducible" first and then filter that formula based on previous filtered results. Because there are too many documents having "irreducible" occurred, and the first phase returns a limited number of results, you may not find your post after applying that formula keyword.
@MartinSleziak I think they do, at least they claimed here: "SearchOnMath is able to search for mathematical formulas and/or text" (searchonmath.com/about).
Thanks for an edit. Maybe when I have a bit of time, I'll add at least one example with links to that bullet point. (One to search with |x| and one with search \left|x\right|.)
@MartinSleziak I think I have chatted with uniquation creator Denis Rystsov (twitter.com/rystsov also rystsov.info/2009/05/01/uniquation.html) via email years ago, as far as I remember, he did give up.
Quoted here: "This project was a part of my thesis. After I graduated I had to abandon it because I didn’t know how to make money with it and I didn’t have spare time to evolve it. "
@WeiZhong What I meant that in approach you can search for two formulas at the same time. I am not sure whether it is possible in SearchOnMath.
Sounds a little doom for a new comer in this field :P I do have spare time due to doing PhD. research full time, but I admit not too many people pay attention to math-aware search and it may not be sustainable with current amount of users.
@MartinSleziak Got it. Yes, I am not sure on that.
08:24
@Sil You can see the score if you use browser's inspect tool, ahead of each DOM element of search result, there is a hidden <span> element with class "score", that is the corresponding result score.
Although I am not sure if it is better to show result scores. I imagine for most people, seeing a score around result is not intuitive, common search engines don't do that too.
Sil
Sil
"After I graduated I had to abandon it because I didn’t know how to make money with it and I didn’t have spare time to evolve it. " Why not open source it then...
"Good news, Linode has agreed to sponsor my hosting cost for one year." Does it mean better vps? :)
In any way, that is a great news
Even assuming you find a perfect threshold, is that really necessary to cut results under that score? I am open to discuss this. But I think hard code a threshold is not very robust because the way TF-IDF scoring works: The score in theory only makes sense to compare between results of same query.
@Sil Not really, it covers the cost of current 4 nodes for one year (5$ / 1 node / year).
Sil
Sil
Well what other search engines do is that they either cut it or first show you relevnant results and then state that following results may not be relevant or something
Although I can upgrade the server at the cost of hosting fewer months given fixed budget, but no need to do so since there are not that many of users.
Sil
Sil
i do not know of engine that will just show you anything even if does not match at all :)
08:35
@Sil I will think about it. The way I understand text scoring part is that they are derived from Odds ratio or likelihood ratio and by thinking query q is fixed, something like P(R | d,q) = P(R|d)
I may not be just applying a threshold, I think they use other strategies, I will keep a note and think about it.
Sil
Sil
interesting trick with the hidden score, im looking at some scores right now
what is kind of interesting
if you search for $x^{n-1}+2x^{n-2}+\dots+(n-1)x+n$
lots of found result, high score etc
then on page 5: approach0.xyz/search/…
it seems to finally decrease, but then i can see the result which is imo quite a good match, or should be
I've taken a not about TF-IDF derivation, here is the note: drive.google.com/file/d/1gTK3C7Ijjih3UV73Rii3VgYydi-g91cX/…. In case you are interested.
Sil
Sil
it contains f(x)=x^{p-1}+2x^{p-2}+3x^{p-3}+...+p, shouldn't that be almost exact match?
like, why it has worse score than the one above it which seems to be far from match :o
@Sil Because it uses p instead of n? And there are many p so my math scoring sums a much lower score? I am not sure, I will look into it locally to find out.
Sil
Sil
or third result in p=6, it contains x^{p-1}+2x^{p-2}+...+p
after some seamingly irrelevant matches
its just random observation, its not critical
08:45
Ah, $f(x)=x^{p-1}+2x^{p-2}+3x^{p-3}+...+p$ and $x^{n-1}+2x^{n-2}+\dots+(n-1)x+n$ are not really exact match.
Although at first sight I did not catch the difference.
Sil
Sil
oh i see, number of terms shown matter right?
There are a few differences, the left has f(x)=, and the right does not have 3x^{p-3}, also they failed to match symbol on polynomials.
Sil
Sil
f(x)= matters?
symbol on polynomials, what is that? you mean dots?
oh you mean p and n
@Sil Yes, basically approach0 is as dumb as I am in Math knowledge sense. It only test scores by structure/symbol differences.
Sil
Sil
i am used to that it is irrelevant :)
i thought it somehow searches "substrings"
okay, now ti makes sense
it also means that when searching sums, one should try different numbers of terms ;)
08:50
I mean x to p-3 is the highest term before dots, while the other one, x to n-2 is the highest term before dots.
I understand your confusion :P.
Sil
Sil
anyway this was really bad example to test score on it, basically even the results on last page are quite relevant :D
No, it is a good example showing the current relevance scoring is not perfect.
As you see, if we have a little math knowledge we will be able to think the mentioned link is actually quite relevant.
Sil
Sil
well in a sense, all results have score of > 150
in other query i try, 150 seems to be nice threshold
i will check these once now and then :)
i know you mentioned some score of -5 or something, i mean there must be some threshold below which it is just ridiculous to show
Haha, you can try to find a threshold, if you find something let me know.
@Sil But like I said, the score is relative (dependent on query), it is a conditional probability ratio.
Oh, I am wrong, this is about math formula scoring, so it is not TF-IDF.
Okay, let me show you how the math scoring looks like in detail...
Sil
Sil
well if it is relative then it says it all
although idk, score should be at least > 0 or something :)
08:59
A most recent published paper, if you are interested to read :P (drive.google.com/open?id=1QjKVpgsTAIMLqrIDhdDOHvDa7sLvoxq7), it is depending on the document formula length.
I think it is more detailed than what I can explain here in a few words. There is the exact scoring formula I use to score a math match.
@Sil It is hard to design a perfect score like that. Just like the most bottom result which you think is irrelevant, but it still share some matched parts.
Sil
Sil
might read it later :) although is that document length or math formula length?
Oh right, for math score it is the document math formula length.
Sil
Sil
what is document math formula :D
like, does the length of MSE post influence score of a formula in it? even if the formula is the same? like having one post with 10000 characters and one with 100 charaters, but both sharing a formula
say you search x^2 and it returns a result containing x^2+1, the latter is a document formula, and its length is defined as the number leaves of its tree representation.
Sil
Sil
but if i search in "blablablablablablabla $x^2$" versus in "blablablablablablablablablablablablablablablablablablablabla $x^2$", the score should be the same?
09:05
is blabla a text keyword or math keyword?
Sil
Sil
text
If it is text, it is text scoring, i.e., tf-idf, more specifically Okapi-BM25
So they are not the same because they are different keyword in texture sense.
text and math uses different scoring formulas, and their overall score is summed to get final score (in a high level)
Sil
Sil
hm, so even if i search just the $x^2$
then i get different scores, right?
No, in that sense you have an empty text inverted list, you do not have text score contributed to overall score.
Sil
Sil
interesting, it is suspicious to someone who does not understand how it works, but not saying it is good or bad
oh good then
i see, the inverted list
one day i will google what is that
09:10
So in texture sense bla is totally irrelevant with blabla (if you are talking in TF-IDF without synonym embedding).
just like an empty text word is irrelevant to blabla.
Sil
Sil
that terminology seems strange, i always thought this is regular index, mapping from words/terms to the content :o
@Sil Yes, inverted index is the core of most search engines.
Sil
Sil
like if we have index at the end of document
Inverted index is indeed mapping from words/terms to content, what content really means is the document IDs in a list.
So when you merge (e.g. AND or OR) the inverted lists of all your queried terms, you get document IDs that are relevant.
There is an (open) text book that I think you will be interested if you want to understand details: nlp.stanford.edu/IR-book
Sil
Sil
i found good examples on wiki
Inverted index
Word Documents
the Document 1, Document 3, Document 4, Document 5, Document 7
cow Document 2, Document 3, Document 4
says Document 5
moo Document 7
Forward Index
Document Words
Document 1 the,cow,says,moo
Document 2 the,cat,and,the,hat
Document 3 the,dish,ran,away,with,the,spoon
:)
Sil
Sil
i guess what i always imagined under term "index" was really the inverted index
the forward index seems to me just tagging
nice, i will add them to my to-read list :)
it is a (non-reverted) priority list though...
the index mainly contains inverted lists and dictionary of the terms, there are other indices such as compressed document content indexed by id, they are used in a later stage when you highlight results.
@Sil Take your time, no worry, compared to many posts on MSE, it is easy to understand. I am sure you will grasp the idea very soon.
Sil
Sil
btw category] 6, [before] 2018-01-10 03:54:25, [after] 1999-12-22 10:05:26
about 8 more days i guess is needed
If later you want to inspect scores, you can paste this into your inspect tool console (e.g., under console tab in Chrome): $('span.score').each(function (i,r) { console.log(r) });
It list all scores of this page.
Sil
Sil
very nice, much better than clicking on each separately :)
09:27
@Sil Sorry, you mean it starts from 1-10? Is it introduced from splitting the topic?
Sil
Sil
no it starts from yesterdays date, it is just where it is currently
i am surprised it did not hit any error during parsing yet :o
So is there a problem? i am sorry i don't understand
Sil
Sil
nope, just stating it will need about 8 more days
Oh, I get it, you mean it is too slow?
Sil
Sil
nah, just stating, i mean it could be faster but it is kind of intentional due to # sleep to avoid over-frequent request.
time.sleep(0.6)
although delay between two requests seems to be more abou 2 seconds than 0.6
09:31
Ah, it is actually faster than I expected.
Sil
Sil
okay nevermind, more like 1 second
So it crawls from oldest to newest?
Sil
Sil
from newest to oldest
[94m
[category] 6, [before] 2018-01-04 13:29:47, [after] 1999-12-22 10:05:26

[curl] /m/community/ajax.php
[curl] /community/c6h543864
[curl] /community/c6h1569661
[curl] /community/c6h1569215
[curl] /community/c6h1569675
[curl] /community/c6h1567723
[curl] /community/c6h1569651
[curl] /community/c6h1569202
[curl] /community/c6h1559623
[curl] /community/c6h1569245
[curl] /community/c6h1072756

[category] 6, [before] 2018-01-04 10:50:19, [after] 1999-12-22 10:05:26

[curl] /m/community/ajax.php
this is a current output
it works in a way you tell aops before which date you want requests, i think you cant ask "after"
so you need to iterate "before" as long as it gives you any
however other categories will stilll need to be crawled as well
i mean if you want to use them (they have lots of math too :) )
Right, I just checked, I remember it starts from the newest too.
Sil
Sil
basically it took almost exactly 24 hours to crawl 2 years of data in this category
09:38
If it is fast, why not also spend couple days to crawl 3, 4, 5 and 7.
Sil
Sil
still its funny, if you consider how fast it would be for aops to crawl it on their side from their database
we are hackers :D
And we are using a simple single script.
It is actually powerful than Google's i bet, because they just follow every link in a huge graph, we are covering everything linearly.
And I believe I heard it is not easy to crawl ajax generated content using their way. So we are good.
So specially designed crawler is very suitable for our case (when we consider only a limited data source).
Sil
Sil
yea, well until aops change their requests API :)
@Sil Sure...

« first day (1172 days earlier)      last day (1891 days later) »