In the search of a question

« first day (116 days earlier) ← previous day next day → last day (2906 days later) »

Wei Zhong

13:38

Indeed, it is because the <br> or <hr> that prohibit Approach0 from correctly crawling that post. But this issue has been fixed before. See chat.stackexchange.com/transcript/message/33944427#33944427

Martin Sleziak

So it means that if the post is craled again, it will get indexed?

Wei Zhong

The reason that this fix is not reflected on the current index is that the crawling process is painfully slow. Given my current hardware resources, it will take perhaps one month to re-crawl the entire MSE, so instead, Approach0 is doing incremental crawling right now, and new post to be crawled since that fix will not have this issue.

Martin Sleziak

question BTW when you update index, do you look at recently active posts or only at new posts?

The post I linked to was bumped 3 days ago, so perhaps it will be indexed later.

Wei Zhong

But as you see, that post is a pretty old one, so crawler output file is not updated since the crawler is busy fetching new posts. However, I just login to my server and manually re-crawled that specific post, but you will not be able to observe this change until the next time I re-index the corpus (crawler output).

Martin Sleziak

The post is old one, but the question was active (bumped) three days ago. That's why I am asking whether "incremental crawling" means new questions, or new answers or all post with new activity since the last update.

Wei Zhong

13:49

@MartinSleziak Since Approach0 is not able to really "update" index, it has to re-index all the corpus every time (it takes ~2 hour). However, even if the entire corpus is re-indexed every time, the corpus it self is crawled incrementally, i.e. only newly crawled corpus is able to overcome this bug.

Martin Sleziak

By corpus you mean "your copy" of SE posts?

Wei Zhong

Unfortunately, the crawler is fetching posts by the time of the posts being created. For example, currently it is repeatedly crawling the 1 - 500 pages of the newest questions: math.stackexchange.com/questions?page=<here from 1 to 500>&sort=newest

Martin Sleziak

I see. So even if som e question appears in sort=active math.stackexchange.com/questions?sort=active it does not get crawled (added to corpus) again. Right?

So if I understand it correctly, neither new answers to old questions nor edits of old answers gets into the corpus in this way.

Wei Zhong

Corpus is the output of crawler. Think of MSE raw post as HTML files, and corpus is a plain text json file which gets rid of ugly HTML tags so that indexer will not index these <a> <href> things.

Martin Sleziak

crawl_pages(start, end, extra_opt) is the proces which adds things to corpus, right?

Wei Zhong

13:58

@MartinSleziak Right, we can consider to improve this by crawling both most active ones and newly created ones.

Martin Sleziak

I am glad that at least I understand a bit what (which posts) the crawler handles currently.

And I am also glad that the "bug" I found is not really a bug.

Wei Zhong

This is Chinese new year, I am in my home town. The Internet is not good. So sorry if you feel I am replying too slow.

Martin Sleziak

So, happy new year, then!

And enjoy the holidays!

Wei Zhong

Thank you. Ideally if we have enough computers, we can distributively crawl the entire MSE within hours. But sadly I am using one of my personal computer doing this, so if there is a bug in crawler script, it is really time-consuming to fix and update the entire corpus.

1 hour later…

Wei Zhong

15:09

The ability to crawl both recently active and newly created posts is added to crawler script now: github.com/approach0/search-engine/commit/…