@dzaima No, Iridescence would hopefully be self-hosted. It turns out it's pretty similar to [github.com/google-research/dex-lang](Dex) but maybe less focused on array programming.
It's higher-level than BQN and I would probably use it less, but I think it's theoretically interesting and would make a good Julia substitute (in terms of language, not implementation).
@dzaima It's pretty fast to check whether all of the indices are non-negative, and you can fold it into range checking. I'm not sure mask/add/gather is any slower than plain gather anyway, since the computation should interleave well with lookups.
@dzaima Which I guess I should put on the problems page. I've definitely spent some time wondering if my indexing really works after doing some arithmetic. On the other hand, being able to use ¯1 for a special element at the end can come in handy.
my asm for addIr (add-ints-reduce): https://dzaima.github.io/paste#0rVRNT@MwED3bv2KkwmUxVZJ2aRqpaIU4LBLHvVUVcmOXBvJh7BQifv2OkzHNshJwIJfEei/zxu95LJW6sRl8@kzgl/RUPoHT7XYaZZxVzTOzshNgVcdZq13L8IuWPbgU8PTSWAWmtbC2qthw9qDZ9PbqKpndpVQrxlp5ZYZaMTLqN8qMKAlSusYymyqsnyok4Q9EuuCBHZpKkSSHJp5YV1URrpeE9R26gjNZq555nhCSeyTlbGoSWRb3NZsLiLplFMrPs089Wl3@2RcObupaW7htGgO/tVQaDb7Wpt2v4mNP8Ttvuk1fwSOwgkpX62gjXrVtODN@6wgkwsPYLiYxbAS7NWjdfauOeEy4347fmznUJn8sVVD1xNAvqfmXl/MQveN1TOsYQzNYMTg5SIxCmlNIP9F@4/aHnToqIX@Rjh06KkbrRMxEJHz9Pv6Q2psYlSDP/Gki7dERCmcgG8frq5RaMsxZoLeugDOY/7ApKrnDlkgfJL34rqRdR0GoUdTOj8F/7W0oNtwl/sGZ0…
@ngn The main loop, starting at .LBB23_4, looks optimal to me, unless I'm forgetting a better way to do sign extension. Maybe comparing rdx with a maximum value instead of decrementing rcx would fuse with the jump, but I doubt there's enough contention in scalar registers for that to matter.
Wait, it's loading a half-register and unpacking to a full register. Could be better to load a full register and unpack to two registers.
@Razetime We never compensated your for the code you wrote to get image sizes. Please accept our apologies. We will fix this shortly. How many hours do you estimate you spent on it?
@dzaima I never tried your code on Dyalog. I have tested plain +/⍳N earlier, and noted it was much faster, and I just noted how slow your numbers were.
@EliasMårtenson so? That doesn't say anything about how my system compares to yours
(of note is that +/⍳N×10 is often >10x slower than +/⍳N, so you must be careful; for 10000000, my 3MiB L3 cache runs out so it also has a performance loss there)
@dzaima No you're right. With heap allocation it takes 261 ms.
That explains things.
But I think the performance slowness didn't come from the heap allocation, but rather that I changed to code so as to allocate the iota result into a generic array where each element is boxed by a JVM object.
@EliasMårtenson (thing being, other things were usually competitive with others here. So either that thing's a beast, or something funky is going on with summing specifically)
@Adám reading in some complex json, I was amazed that this works: data.(≢field.⎕NL¯9) ⍝ Count objects under field, where data is an array of ns. So cool.
(also note that this is very much a test of the performance of ⍳, not +/ - in this you can see that +/ takes only 20% of the time of the +/⍳10000 expression)
We had ambitions to take that further, but the rewards in real applications seemed rather limited, and we had so many other interesting things we wanted to add... We may return to parallelisation again, but at the moment the trend towards more and more virtualisation means that the benefits of multi-threading single processes SEEM to be fading rather than growing.
@dzaima So as of 17.1 the Dyalog allocator only moves forward across the workspace, so that allocated memory is never in cache. In 18.0 I added a little snippet to pull it backwards if a pocket is freed near the end of the workspace ("Better cache usage" here). But this code has to be simple to avoid too much overhead, so if there are too many pockets it tends to fail.
Re 1111⌶: In fact, we are seriously considering changing the default for 1111⌶ from (all available cores) to 1 on all platforms. It is already 1 by default under AIX because that platform is typically used as a transaction processing system on machines with a very high number of cores, and using all the processors by default would be a bad thing.
I think the improvement affects microbenchmarks more than real code because a larger section of code should have more computation or some in-place operations to allow memory time to catch up, particularly if it can figure out it needs to prefetch past the end of the workspace.
@Marshall That is entirely possible. DECF's are of course much better targets because you generall have much more CPU spend per byte processed.
Paralellisation of things that do nothing and write lots of data to memory (like ⍳N) quickly get you into negative territory as the cores fight for memory bandwidth and the system tries to sort out the cache coherence (as far as I understand it).
The number of inter-connections between features and documentation pages accumulated over a few decades makes it a really interesting problem to find all the different pages you might need to update when you make a change to the interpreter.
@Razetime I'd like to ask you if a proposed payment is reasonable for you, but do you want me to do so via email (private) or here in chat (for all to see)?
@dzaima @Marshall interesting observation: if you write them in the most straightforward way, the inner loops for iota shorts, iota ints, and iota longs end up exactly the same in asm
@shapr My Swedish teacher was from Norbotten. One day she tried speaking in class using her original dialect. Nobody could understand her (none of the native Swedes either).
I had trouble understanding Stockholm Swedish at first, and I caused at least one big laugh when I said "Ska vi fara?" which is pretty much "Let's mosey pardner".
It's funny, I'm from Alabama, USA and I have zero rural accent, but after moving to Sweden I gained a rural redneck accent in a completely different language!
And then my plan is to go mountain biking, and then possibly fix the parser bug that prevents me from pushing the new function definition syntax in Kap.
@nathanrogers no. i'm not sure what to do about ffi. ngn/k is f/oss, so you can compile anything you like into the interpreter anyway, without an explicit api for ffi
@nathanrogers something like that, but they still haven't released
@nathanrogers if you believe it doesn't do what it should, you can mention it in the google group, ideally with a minimal repro, and arthur&co should take care of it
