@RolfMertig We have discussed this topic many times over the years, but the reason to make this post now is that Arnoud mentioned Stephen is thinking about discussing this. I wanted to finally write my thoughts down and I shared this with Arnoud. Maybe it helps, probably it doesn't make a difference.

@CarlLange Exactly my point and the reason why I asked if it is a good idea. I assume no one of us has enough knowledge about the performance of regex's in JS to answer if a regex-trie performs better.

@halirutan :) I think that it's not worth it unless the performance of the highlighter is poor - is it? Seems like a reasonable speed to me.

(speaking as an ex-professional js dev 😅)

@CarlLange On the other hand, I enjoy to test such stuff. While we're at it, I have already implemented it and tested it with Kotlin (Java VM) and the performance of the Trie-Regex is about 3x faster.

@halirutan Well, definitely do test it if you're willing! It would be interesting to find out :)

@CarlLange What I did is use all symbols we have in Mathematica and build both the normal "alternative regex" and the trie-regex. Then I gave it all 7000 symbol names for a positive match and a list of 7000 dictionary words as negative match. Runtime is 500ms compared to 150ms.

@halirutan Interesting! The performance on the JVM might be quite different to JS though :)

But I'm going to keep this in mind next time I need to do so me sort of big string match in any language ;)

@CarlLange This is where you come in as I don't have much experience with JS. Is there a possibility to test this in JS?

@halirutan Yeah, I could do it. Would you upload the kotlin code you used to github gist or something, and I can translate to JS and test it out?

No promises that I can do it soon though, I am going to start hiking again tomorrow and I have some prep to do :)

@CarlLange I can prepare everything. Give me some more hours. I already rewrote the whole approach and I will push it to a new repo. It will contain a Mathematica package that lets you easily create the regex for JS

And of course it will also contain the small Kotlin tests.

@halirutan Great stuff. Looking forward to screwing around with it!

@CarlLange Alright, with non-capturing groups, I'm down to 84ms against 445ms which reduces the time to under 20%.

I have tested the new JS script for highlighting and it appears to work, although I have to say it really hard for me to work efficiently in JavaScript.