@ArnaudD. Approach0 parser treats expression symbols differently by their types or symbols. For example, both \mid and | are of type VERT, but they obviously are different symbols. In theory you can get both of them by query either of them, but the search results will prioritize exact symbol match. This is designed to differentiate many other things in general, such as equality and inequalities, variables and even integrals and sums. Supposedly we do not want too constrained on similarities.

@WeiZhong Are \lvert and \left| also in the same type? Or does \left nad \right make a difference.

if I want to find out what belongs in what type, do I look in lexer.template.l?

For example, I see there:

\\mid                                 { RET_TOK(_AUTOGEN_, VERT, WC_NORMAL_LEAF, VAR); }
\\lVert                                    { RET_TOK(Vert, VERT, WC_NORMAL_LEAF, VAR); }

So from this I get that \mid and \LVert are both of the type VERT, right?

I see there also:

 /* absolute value */
\\left[ ]*\|                                                         { return _L_VERT; }
\\right[ ]*\|                                                        { return _R_VERT; }

So \left| and \right|are of different type from VERT; namely _L_VERT and _R_VERT? Or does this do something else?

Sorry I meant to write "are \mid and \lVert of the typo VERT?" The "\LVert" was just a typo.

@MartinSleziak Exactly. When a symbol is next to \left or \right, or it is embedded with grammar implication such as \lVert, they will need to be further parsed depending on its context. It is ambiguity that makes me decide to distinguish \vert and \lvert because in set language you may have something like {x \mid x \in R} notation and for efficiency reason the parser should know it is not necessarily paired with another VERT, because in this case it is not absolute value.

While in some cases like |x-b| you want a grammar tree to have "absolute value" tag.

Of course you can still treat vertical bars in |x-b| as individual tokens, but that will lose a level of semantics and there is ambiguity to pair the vertical bars if you have expression like {x | |x|>2 }.

So in short, we want to better understand paired token such as \left and \right something, but if tokens are not explicitly paired, we do not want to create ambiguity. So my solution is just treating them differently. I admit there must be ways to be more intelligent on this, but on the other hand the parser needs to be efficient to run for millions of posts, so efficiency still matters.

The parser I am using is a LL(1) parser which generates grammar finite machine to digest TeX string from left to right without "looking back and forth", and the grammar rules should have no ambiguity. That is why it is fast.

Since some people use absolute value with \left..\right and some without them, I should search both for $\left|x+y\right|\le\left|x\right|+\left|y\right|$ and for $|x+y|\le|x|+|y|$ make sure that I find all insteances?

For now yes. And later if search engine is efficiently enough (I think it is approaching to that goal) I will add query expansion feature to help you search likely different forms of possibilities automatically in single query.

Thanks a lot for the explanation!

I'll keep in mind trying both versions when searching.

You are welcome Martin.

For example, in Arnaud D.'s search, I'd have to try both with \left..\right and without, as in the searches which are linked here: chat.stackexchange.com/transcript/message/52786924#52786924

Yes, also remind yourself a0 is in a very very early age :P

@WeiZhong Just a note, it is still pretty much work in progress, but I have updated the docker repository github.com/TheSil/docker-approach-test so that there are now two images, the build image and the deployment image.

The build image needs to be built usually just once and can be then instantiated whenever the build of approach from latest sources is needed, while the deploy image can be instantiated to run web/indexer/searchd containers (currently only documented running the web, for searchd/indexer containers i will have to deal with index persistency).

Again, it is still suited for me as a Windows user, but shouldn't be hard to tailor for linux hosts.