After the discussion around function definition syntax, I decided to write down my thoughts on the subject. Mostly so that I can gather my own thoughts on it. However, if anyone wants to read it and perhaps provide some helpful ideas, I would be very happy. I put it here: gist.github.com/lokedhs/72ce0a35dac6744c557a5a3fb18012aa
loke[m]: it's just complete garbage and has been for way, way too long
it takes an average of 131ns per character, but that's apparently way too high compared to the average of 2.1ns per character of compilation (complication obviously isn't per character, but still)
@ASCII-only it's java. And I have much more that I could change before changing everything (checking if is valid name by 'a'<=c<='z'ish things instead of iterating through a list of chars, using a switch instead of many if-elses, etc)
@ASCII-only java is actually reasonably fast I've found
1.76ms → 1.63ms just by doing slightly less stupid stuff (though i feel like i'm still doing something incredibly stupid, and i should just rewrite it entirely anyways)
@ASCII-only right, but there aren't many better alternatives other than unrolling to a switch (which is still probably the right move). though hashset is still 1.56ms
(and UTF16 is sort of annoying as BQN uses 𝕨𝕎𝕩𝕏𝕗𝔽𝕘𝔾𝕤𝕊𝕣 which are surrogate pairs in UTF16 (but this actually turns out nice for performance, as detecting them all is simply checking if the first char is 55349))
@dzaima Are you sure? Kap's tokeniser isn't slow and it's also using the JVM.
I'm not sure Java is to blame here.
@dzaima I have created a special kind of stream that works on codepoints (int's only). That way I have isolated the UTF-16 garbage to a single place. All code that deals with characters actually handles lists of integers.
Except for the formatting code, which deals with lists of strings (each string containing a grapheme cluster)
@EliasMårtenson I've been measuring this file, and have an average of 120 ns/character now. Of course that specifically won't work for APL, but any sort of measurement of ns/char on a somewhat large file would be acceptable
@dzaima (my tokenizer definitely isn't slow slow, but it's still slow by at least some standards)
@EliasMårtenson right, I kind of thought that'd be the case. do you have a separate tokenizer that could be fed some invalid KAP but valid APL for testing?
@EliasMårtenson converted to dzaima/BQN (with strings in-place of symbols), after warming up with a couple thousand runs, tokenization+complication+evaluation takes an average of 5.4ms over 1000 runs
@dzaima I see what you mean. But based on my 20 second look at the code, I think even though it's not well structured (it's written as a quick hack initially, I presume?) it seems straightforward enough that performance-wise, I would expect it to be similar to mine, which indeed is what we saw.
Mine is also written in a similar straight-forward way, with very little in the way of triks to improve parsing performance (it wa snever a goal of mine).
how about this for a test: https://dzaima.github.io/paste/#0VZpbduRADEIXpMMP@1/cpC0uqplkkk4/7LJKQoBsW5q/H/77ud8e/37P3y/J8/fj@/579PfU3zP@ff899O@d34f9/f@9TTz5@9j3pvkdWXn5O@h8v35vHO3n/o74@/899T3zLcEsy9/7Z5f1O/V3eJ7/HV27wP1jT/F93XK@tfzW9/2afSlXt0f9ff0@rVznfIv4PfwOP3t13zpnl/H7l3A0JJOLzVK@hX3H2GvOe7/YeqORsHhfsXbhG80v3vrevAtKiDZa4oAbi@8wysUnPt7PfsuaDei@J1u2Efle22DmtN@Zvnj9vvcU35Xtw0kIxXqHVFH2TzmTOM7sO7PD3gXuqbT/iMF3/ISYlCC45tC7wMl25yibczMkrPbMc9ufDP9yZrLASei/AysHJstaCok1oWhUN@KJ1ZdNu6smrffJTSa7V7lrJbiTFexaknzfmVIXCdOm1he85JWSS4nALpJk8lBtm5fag@d0dvZ0U9nkiLf4JjuTh2RxwrF1P@TbZnVOtHvrbCzZm70ILuw2bDYqi…
@dzaima (with my local changes, takes an average of 3.1ms - 0.56ms tokenization, 2.1ms compilation (times swapped now :o), 0.2ms execution (yes those don't add up, i'm timing each separately))
@dzaima How much of the time for the BQN-based tokenizer is used by the ⍋ in CharCode? Also note that Tokenize deduplicates tokens, which is probably a decent fraction of the total time. If you switch the character code lookups to Java, it might make sense to do a lot of the tokenization array-style, since it takes advantage of booleans.
@dzaima Not actually that bad then. On reflection, the array-style tokenization might be better for some things like word (number or identifier) detection, but those are probably a small portion of overall time, so it's not really worth it. Unless you have a huge codebase and can switch the whole thing to GPUs or something like that.
No real bottlenecks, so basically the whole thing is just a few times slower than a good Java implementation with the current runtime.
Which is fine. It's way better than the average interpreted language could do. It would be a pretty big stretch to say it's the future of compilation though.
@dzaima got it down to 1.2ms compilation by doing manual parsing instead of the string pattern matching for monadic/dyadic function application
@dzaima not that much of a difference outside of that example - on Marshall's compiler, 0.022→0.020ms, but that's within the realm of error and i can't be bothered to check more thoroughly
interestingly enough, this function appears to take 7.8% of time of compilation (of c.bqn)
@dzaima (yay i actually finally started profiling things)
oh, you know what, i'm stupid. Tokenization of a string involves compilation all inner block and compilation of the tokenization result only involves compiling the outermost part of it. So my separation of tokenization and compilation is extremely wrong
@dzaima (and then there's a bridge from here to matrix that i'm working on, so the above message went through 3 bridges to arrive at my local matrix homeserver :D)
