@CarlLange If you're still interested, I found out most details about the regex-problem in the meantime.

Basically, using a trie is a good thing and is done by other regex optimizers too. However, some go a step further. As you know, a trie is nothing more than a prefix tree which can also be thought of as a deterministic automaton (DFA) which reads input and goes from state to state (nodes in this case). What other programs do in addition is that the minimize the DFA once it is constructed and build a regex from that.

This merges internal pathways and gives further optimization.

I have asked a question here and the answer led me to all information I needed

So one difference between the full-blown optimization and my simple trie can be seen in this inner node which matches something with Arduino. My regex results in

udio(?:InputDevices|OutputDevices)

while the full-blown solution reduces this further by merging suffixes

udio(?:In|Out)putDevices)

The timing of the person who answered gave almost exactly the timing differences that I also found with Kotlin, but I'm not sure if his program used JS or some other language.

Wolfram Engine is free for non-commercial software development.

I can't believe I haven't heard about that before. This opens up a whole new range of use cases for WL.

(I should've said "non-production" but that distinction is not that important to me. I learned about it here, and using WL on AWS EC2 is exactly the kind of thing I might want to use it for.)

@C.E. I knew they were discussing it in live streams but I didn’t know it had actually shipped

That’s really cool

Stephen is working on a blog on this topic (I think it will be published tomorrow)

We're still working on getting final builds for the "Wolfram Engine" product

The tldr; for advanced existing users: This is a kernel-only product, so the notebook interface is not included. The frontend is included so all rendering services will work (rasterizing, of images, typesetting etc. etc. )

The licensing is for 'non-production use' (Stephen will write up the details and there are details in the licensing.txt file in the top directory of the layout

Perhaps one way to describe the licensing is an open ended (not time limited) trial version of the full Wolfram Language, sans notebook interface.

The positioning of the product emphasizes deployment into software engineering systems and interactivity with other systems, databases, and file formats (so, for example using it with the Wolfram Client Library for Python (see recent blog post on that)).

The product layout is very much what is being used to power Wolfram|Alpha and the Wolfram Cloud at the moment

@ArnoudBuzing does "non-production" include running it on an HPC for research purposes?

@ArnoudBuzing how does it differ from the Wolfram Player...? I just checked out the bundle and it looks like a whole Wolfram Player is embedded.

That seems a bit....excessive?

@ArnoudBuzing Speaking of licensing, can you offer any thoughts on this? chat.stackexchange.com/transcript/message/50332372#50332372

@halirutan That's really cool! Would you like me to implement the matching in JS?

Really useful to know - I haven't checked but I can only assume the trie version is also smaller in terms of bytes, so it's faster in a lot of ways.

@CarlLange I'm not sure it's worth the effort. Let me ask a dumb question: Could you implement a simple speed test in JS for my question number 2 of the SO post?

This is, you take the pure regex and simply match through all the positive and negative examples and time the performance.

@halirutan Sure, sounds fine

What I would like to have is a comparison of my two regexes.

OK, you got it

Awesome

@CarlLange If you don't find some things or need a different format, just ping me here.

@halirutan By "positive and negative examples", do you have a specific test set?

@CarlLange Yes, the last link in my second question

github.com/halirutan/WolframLanguage-Google-Prettify/tree/…

doh :)

symbolNames are all the positive examples and dictwords the negatives (although they contain some words which are symbols of course)

@b3m2a1 Well, and FE is embedded. There are many services that are provided by the FE. Leaving it out would have crippled the system severely.

@Szabolcs I just didn't realize the FE was 3GB I guess

uh, silly question, but what's the difference between the FrontEnd and the notebook interface?

@halirutan Is this what you were looking for? jsfiddle.net/k7spvntq/1

here, it shows both at virtually the exact same performance:

@Kuba woah I didn't know BackgroundAppearance could work for Cell too...

SetOptions[
 EvaluationNotebook[],
 DockedCells ->
  Cell[
   "holy shit", BackgroundAppearance -> FrontEnd`FileName[{"WolframAlphaClient"}, "WLInput.9.png"],
   CellMargins -> {{-3, -3}, {0, 0}},
   CellFrame -> None,
   Background -> None
   ]
 ]

@ChrisK Not much really, but the I assume they've made the WolframEngine FE refuse to actually create notebook windows.

I mean maybe not, but that'd be my guess.

@CarlLange OK, let me put the optimized regex of answer in.

ah, sorry, try from jsfiddle.net/k7spvntq/3

I must have made a mistake, because that was way faster than it should have been :)

so that shows your trie regex as a bit faster, although not really by so much that it's worth it :)

@CarlLange What is the return value of reg1.test(symbol)? A boolean?

yeah

it doesn't actually show every symbol as matching, which I assume is some escaping issue I didn't deal with

@CarlLange OK, it is so odd that there is no real difference in JS

yes, they're both missing about 200 of the symbols.

It's more likely I've made a mistake.

These are probably the ones with $

ah, not 200, it's actually way less

they match 6759 of the 6775 symbols and 29 of the 6975 words each

jsfiddle.net/k7spvntq/5

well, I actually don't think I've made a mistake at this point

I suppose it's nearly 10% faster to use the trie

test1 is the trie version, test2 is alternatives

@CarlLange Yes, I got about the same results.

@Carl OK, I think considering that it was fast enough before, we can live with a simple trie which further speeds it up.

And of course it reduces the regex size considerably.

@halirutan I agree, I think it makes sense. The byte size would be the main improvement

and it doesn't hurt to be 5% faster :)

Yes.

I'm interested in the code you used to generate the trie, could you link me to it?

@CarlLange Pretty short Kotlin implementation

github.com/halirutan/WolframLanguage-Google-Prettify/blob/…

It's all in the same repository.

very nice, thank you!

Added support for a bunch of the built-in stylesheets in my IDE (it was low-hanging fruit):

If anyone has palettes they've written for working with specific types of notebooks (where the type is based on the stylesheet it builds off) I can try to include those as toolbars for these things, too.