Interesting, it surprises me a bit (in a good way!) that new people are making things with APL, I thought 99% of commercial use was legacy code from decades ago
@rak1507 So far mostly small experiments. But on the other hand, the average time from someone starts using APL until Dyalog makes real money from support contracts or other licenses is 15-20 years. One might expect the cycle times to be shorter these days, we shall see.
@rak1507 Our real competition, at least according to our own analysis, is not other array languages. It is Python, R, Matlab and other engineering workbenches. Even Excel.
@rak1507 "Real money" is a fuzzy term, not sure how to make that a real number. perhaps a useful concept would be "increase our revenue by more than 1%"?
@KamilaSzewczyk It's a little while since I looked at isolates, but a "FUTURE ERROR" is a general exception which covers the fact that a future that you were waiting to materialise is not going to arrive. Taking a closer look at your expression to see if I can guess at why you got that VALUE ERROR.
OK, big is a function from dfns, starting to get the picture now.
If you have functions that you are going to use all the time in your isolates, it is a good idea to launch the isolates from a different workspace than the bare isolate workspace.
If you were to )SAVE a workspace that contained the isolates workspace plus any code that you want to refer to, you can set isolate.Config 'workspace' 'yourws' This would give you isolates with the additional code (and perhaps data) pre-loaded.
Applications that have a lot of state often use pre-allocated isolates rather than using the operators that create new isolates on each call. That way you can manage the state of each isolate "namespace" directly. Have you read the user guide?
Recent isolate workspace also include a ll namespace containing ll.Each and ll.EachX, which reuse pre-conditioned isolates rather than creating them on the fly.
@rak1507 R and Matlab are "array languages", but it is not their array languages that makes them competitive against APL (everyone seems to agree the languages are pretty awful), it is the huge libraries of solutions, workbenches, statistical packages, etc that make many engineers pick them over APL.
@rak1507 R is very popular for statistics. My point is it is not the R "array language" that makes it popular, it is the vast collection of excellent statistical libraries.
Right yeah, that's not a surprise, definitely the same for python too, unfortunately hands down better than APL for basically any task not solved trivially in APL
But they have reached critical mass, which we remain some distance away from in APL. However, when I look at what we need to do to make Dyalog APL more competicive, it is tooling, tooling and more tooling, a package manager, documentation and training materials that we need. A more powerful or slightly more consistent array language doesn't appear very high up on the todo list.
As an APL language geek at heart, I would be very happy to see someone start to compete successfully against us on the basis of a cleaner, better array notation.
@rak1507 Wrt numpy, I once had lunch with Jim Hugunin in Seattle, back when he was a Microsoft employee working on "IronPython". He told me that numpy was "APL done right" :-).
Depends what he means by 'done right', better in terms of the array language, or in terms of the ease of use in a real program, because there it definitely wins.
@ngn Is K popular because of the notation or because of the speed?
@ngn if it is K you are referring to, I think it is hard to argue that it is a cleaner, better array notation. And anyway, the owners have decided it doesn't exist any more ;-)
Well, they replaced a bunch of stuff with real words in Q and haven't lost any customers from it, so I'd argue that it's the speed + other factors that's important, not the array notation
@ngn Arthur worked on high frequency trading in APL for many years. He got tired of waiting for the people at I.P.Sharp to do what he needed to be even faster, and started with A+ and then k, which I think of as a DSL for timeseries with a high performance database built in. So again, I would claim that it is the "specialised tool" aspect of K that has allowed it to compete against APL rather than the array notation.
@ngn Don't get me wrong: I have HUGE respect for what Arthur has done and there are also some very, very nice language features in k
the biggest issue I had with J was that the compiler toolchain for the calculator didn't support dynamic linking, and was also missing some floating point headers
@rak1507 i'm familiar with both, and i think it's very clear which is the better language (once you learn it, of course). i would like to know why you prefer apl?
@Adám yep, so my question basically boils down to: is there a highly portable, self-contained implementation of an APL-family language that doesn't use any dynamic linking? If that exists, I'm pretty confident I could convince the compiler to build it for the calculator (github.com/ndless-nspire/Ndless)
@Adám Has there been much work on GPGPU front of APL? I have a paper that I am preparing in which I run APL on Vulkan for ML inference. Do you happen to know any related work beyond Hsus PhD thesis?
@Adám ngn/apl has closures, so you can probably squeeze mutable data out of it somehow. Difficult with the static syntax though. But it does let you run native JS, and you can definitely work with objects through that.
@dzaima It has been on 249 for a while, though new people have written for the first time. I wonder if there's a minimum amount of activity required (like what makes generic avatars and userNNNNN names go away), or if the counter isn't continuously updated.
@jhaavist The thing about APL on the GPU is that (as I'm sure you know) it's very difficult to run a full APL with nested arrays and everything on the GPU, so APL programmers wouldn't usually be interested in working with a GPU-native language. APL on CPUs is fast enough for most use cases as well.
@Marshall ... for the reasons laid on this reply. I have been thinking about lightweight typing of APL programs for GPU execution. And my project was more about having an APL runtime on GPUs. Basically, you give an APL source code and parameters and it runs on the GPU.
I havent dealt with nested arrays though, since my work has been on scalars, vectors and matrices for now. But, I would imagine that with some lightweight typing you could maybe infer subprograms which you could run as independent parts.
@ngn I disagree, I am getting comparable execution times on numpy and Vulkan for workloads which take around 500ms.
@jhaavist Sometimes. What happens when the user calls Each to do computations on a hundred arrays and most are tiny but a few have millions of elements? Only known at runtime, of course.
Basically, barring a huge advancement in GPU compilers, the kind of programs an APLer (even a good one) naturally writes and the kind that run well in a GPU-native language are very different. But of course there are many fields where getting it on the GPU is very important and programmers are willing to change to a flat array style for it.
@jhaavist That's why I think Hsu's work is promising. It lets you compile specific parts of your APL code, so you can work on making performance-critical parts well-fitting for the GPU, leaving general stuff to the CPU.
That said, I wrote BQN's compiler in a flat array style. It's a much larger codebase than Co-dfns since only a small part of Co-dfns is designed to be run on the GPU. It still uses nested arrays occasionally: for example there's a fixed length-4 array for IDs of identifiers and numeric, string, and character literals to avoid repeating code.
@Marshall Good points, in the Each case you would most certainly end up waiting a lot of time. I think that in general, the question of whether you write an each loop on GPU code or whether you just submit N jobs instead is an application-specific thing.
yay got TeaVM to execute 2+2 in dzaima/BQN in JS/Chromium. Funnily enough, the only actual runtime issue after it compiled (removing •sh, file reading & similar) was in formatting the 4 to a string, the literal last step :D
@dzaima performance: +´↕10000000 executes in 160ms (regular jvm does it in 30ms and Dyalog takes 7.5ms for +/⍳10000000)
@Marshall why node? if you're running outside a browser, just use java
there's GraalVM nativeimage if you want faster startup time (it's a max of 2x slower than regular jvm, and has <10ms startup time (on a way simpler thing than BQN but still))
@dzaima which is like what i get with the regular jvm too (excluding the first run, presumably caching files from the hard drive, but node of course also has that overhead)
@Marshall executing an empty file with node is 350ms, which is already worse than java
@dzaima JDK is faster than I remembered. I was thinking like 1s, but it's about 0.3 to either run a small dzaima/BQN file or docs/bqn.js with no arguments.
Okay, I think I set up Java to use GraalVM, but now it's 0.6s!
@Marshall graalvm alone isn't enough, you need to compile with nativeimage to actually get an AOT-compiled thing (i'll see if i have any leftovers of commands i used to build dzaima/BQN with it)
iirc you needed to pass some flag so it wouldn't all fall back to a regular jvm because JComp exists (which also means that •compstart←¯1 is necessary, but you get your 99% faster startup time so)
ah, there we go: native-image --report-unsupported-elements-at-runtime -jar BQN.jar
One of my music synthesis scripts was slower though (7.5 versus 5.7 seconds). Kind of the opposite of what I'd expect, since gendocs is a lot more interpretive and I'd think that's where the JVM transpiler is best.
@dzaima Oh, most of the time for the sound scripts is going to be filters, which use scalar code (I need to specialize that code for common lengths like I did in C).
