The APL Orchard

5:15 AM

@dzaima That's basically the approach of GNU APL, and it's also criticised.

7:35 AM

@rak1507 Dyalog APL allows you to generate a hashed array for fast lookup: help.dyalog.com/18.0/#Language/I%20Beam%20Functions/…. The hashing is updated under the "obvious" operations like adding or items to the end of the array.

@ngn re Dyalog "resting on their laurels forever": I'm sorry, but this is an absurd statement. Dyalog APL has a larger full-time development team, and is consistently evolving faster, than ANY APL system in the history of the language. We added dfns, OO, real Unicode support, DotNet integration, see https://www.dyalog.com/dyalog/index.htm

These advances would appear to be the main cause of your so-called "vendor lock in".

Dyalog APL

<DuClare> Can you write an RTOS in APL?

Morten Kromberg

@DuClare I don't think I would recommend any existing APL systems for that. APL might be a good modelling language for an RTOS though.

Dyalog APL

<moon-child> RT refers to the kernel/scheduler, and I'm dubious it's practical to implement a kernel in apal

Morten Kromberg

@Wezl I'm somewhat allergic to people taking cheap shots at whether we approach our responsibility as an APL vendor ethically and passionately ;-)

ktye

7:52 AM

@KamilaSzewczyk don't tell me you never used them! or would you compute the eigenfunctions of the stokes operator in a cylinder without them?

Razetime

How do I get input using a custom prompt?

Elias Mårtenson

@MortenKromberg In my experience, pretty much every developer that has touched APL has a different opinion as to how it should be properly extended to suit them better.

RGS

@Razetime Use ⍞← to output the prompt, use ⍞ to read it in and then drop as many characters as the prompt had.

prompt ← {⍞←⍵ ⋄ ⍞↓⍨≢⍵}

Elias Mårtenson

We have one extreme being GNU APL which is very reluctant to go beyond APL2. Then along a different axis we have Dyalog that for obvious reasons want to prioritise features that are beneficial in the enterprise (and being employed by an enterprise software company myself, in the banking industry no less, I fully understand these kinds of customers.

And then we have the experimental crowd that runs in all different directions, like BQN and, yes, KAP.

Morten Kromberg

@EliasMårtenson Right, which is why I only feel safe if I have actual users of the language who will help verify our direction. The current CTO of Dyalog spend 30 years as an APL consultant before feeling qualified to take up that position.

Elias Mårtenson

8:00 AM

@MortenKromberg I would hope that all of us who implement experimental implementations will help give Dyalog an opportunity to see these wild experiments in real life.

Morten Kromberg

I sometimes wonder how many of the people driving experimental array languages have every worked on a real application in an array language. I know that some of them have.

Elias Mårtenson

And if KAP had more than a single digit number of actual users, I'm not sure I'd be as excited to work on it anymore. Imagine not being able to change something because you feel like it. The horrors.

@MortenKromberg My array background is R.

I started developing a bunch of statistics stuff in GNU APL, and then ran into some limitations. I was working on the GNU APL codebase to make some improvements, but then I wanted to change the language a bit so I decided to implement my own experimental version.

Morten Kromberg

Experimental and open source array language implementations are extremely valuable to Dyalog, in all sorts of ways. I salute you all for putting your energy into this work. I do sometimes get a bit weary of having my motives attacked by people who I suspect have very little insight into what it means to have an installed user base. As you point out, you probably do not actually WANT users of your language, it would be "horrible".

Fortunately I do not consider it to be horrible, having been a user most of my life, and getting my kicks from seeing people USE the stuff that I help to create with the team at Dyalog.

Elias Mårtenson

@MortenKromberg Oh I know. My previous job was developing the controller software as part of a new product line of mechanical locks and access card systems (I was employed by the world's largest lock manufacturer, but most of the access control solutions were developed by a very small team with me being the architect)

Seeing those access panels on door all over the world gives me a bit of a confidence boost :-)

That also led me to know exactly how painful it is when customers have problems and demands on your software :-)

I think when it comes to users of Kap, then it has more to do with the fact that I'd have to go back to implementing features based on other's requests, which is fine when I agree with them but that's not always the case :-)

Morten Kromberg

It is like any kind of dependency on you, in family or business. It can mean you need to do some work, but ultimately (IMHO) it is what gives life purpose. Of course, going off to do your own thing can be an important part of how you recharge your batteries.

I did a careful analysis, and it is clear that my life would be much simpler if I could get rid of all the customers AND all the employees :D. Speaking of which, time to prepare for the first meeting of the day. TTFN.

Razetime

8:44 AM

@RGS omgf thankyouthankyouthankyou

Morten Kromberg

9:01 AM

@Razetime This is one part of APL that I think is in need of a facelift. I think our work on "scripting" may lead us to add some more modern mechanism for taking input from the user.

rak1507

Ooh cool

Razetime

9:17 AM

I'd definitely like to see that

Morten Kromberg

Any suggestions for what you would like to see?

RGS

9:47 AM

@MortenKromberg Maybe I am overlooking something but the prompt function I just shared is more or like what I would expect and what I find useful... Perhaps that's because I am very used to Python, where we have the input function that behaves basically like that prompt one.

Bubbler

1. print to stdout without newline (current ⍞ prints to stderr) 2. input not clobbered with output (TIO)

RGS

But I am aware languages like Java and C have more sophisticated ways to read input, with string formatters and whatnot, right?

@Bubbler Ah, so I can't see ⍞ prints to stderr in the interpreter, but if I were to run a script and stdout ≠ stderr, then that's where ⍞← would end up?

Bubbler

@RGS Yeah, that's what you see on TIO

Dyalog APL

<moon-child> @RGS c has scanf, but its use is generally discouraged. Format specifiers are mostly just for output

RGS

@Bubbler Ah right, didn't even notice the "prompt: " message was on the "debug" panel.

Bubbler

9:52 AM

APL's output formatting is good enough, and not much idea for input parsing (maybe no need to worry, most languages just parse inputs manually, with the most basic tools for string to int/float)

Probably something in the lines of ⎕,←stuff might work for "print to stdout without newline", if you don't want to introduce a new symbol

(Python 2 did something similar: a print statement ending with a comma, like print 1, 2, printed things without newline.)

Kamila Szewczyk

$[=0.5;
@h=(1, 2, 3, 4);
print $h[0.5] . ' ' . $h[1.5];

I just found out that Perl has ⎕io, but better

Dyalog APL

<moon-child> most things are better than ⎕io

Elias Mårtenson

10:15 AM

The optimal solution is no ⎕IO

Kamila Szewczyk

i think perl should have access granularity too

rak1507

11:14 AM

how come namespaces take up so much space? even ⎕NS⍬ takes up 2032 bytes

Elias Mårtenson

@rak1507 Is that a problem? How many namespaces do you need?

I mean, you'd have, what? 10?

rak1507

It's not a real concern just wondering why there's so much overhead

Elias Mårtenson

I mean, a namespace is a mapping of symbols to objects, so it's a hashmap with some metadata I guess?

Hashmaps oftens have an initial size allocated. Especially in runtimes where memory allocations is very expensive (such as C).

rak1507

I was trying to do something really dumb and got a WS FULL but it wasn't a good way of doing it, just an experiment

dzaima

@EliasMårtenson (Dyalog of course has its own memory management, so that's not much of an issue)

Elias Mårtenson

11:18 AM

@dzaima True, but it may suffer from the same issue that most runtimes without a compacting collector suffer from.

dzaima

@EliasMårtenson it also has compacting

Elias Mårtenson

OK, I did not know that.

rak1507

interesting that ⎕SIZE seems to recursively traverse namespaces and keeps track of circular references

Kamila Szewczyk

11:42 AM

{g←⍺⍺⋄{1=+\1 ¯1 0['{}'⍳⍵]}⎕cr'g'} why is the shape of the return value of this thing 1 8

i expected it to be a vector, not a 1x8 matrix

I can (somewhat) fix it with ∊ / ,

dzaima

@KamilaSzewczyk example arguments?

Kamila Szewczyk

⍺⍺={⍵×2} / ⍵=⍬

dzaima

@KamilaSzewczyk ⍴{⍵×2}{g←⍺⍺⋄⎕cr'g'}⍬ already has shape 1 8

Kamila Szewczyk

why does ⎕cr return a matrix?

because of multiline functions or something?

dzaima

@KamilaSzewczyk yep

⎕vr may or may not be an acceptable alternative to ⎕cr (it surrounds with ∇s and the name, but gives a single vector)

Kamila Szewczyk

11:51 AM

i want to (somehow) parse the contents of a dfn, provided as an argument, and then execute it

i made a small testcase to make sure my program doesn't break on some weird testcases:

 f←{⍵×⍵
     ⍝ that's a nice test case?
     v←2+2 ⍝ isn't it?
     ⍵+2 ⋄ 3+3
 ⍺×⍺}

dzaima

@dzaima (⎕nr being the third alternative, because of course there are 3 different functions for getting the source of a name)

Kamila Szewczyk

this monster seems to work:

f{g←⍺⍺⋄{'}',⍨⍵/⍨1=+\1 ¯1 0['{}'⍳⍵]}↑,/'⋄',⍨¨{⍵/⍨0≠{≢⍵}¨⍵}{{⍵/⍨2∨/0,' '≠⍵}⍵/⍨~+\⍵='⍝'}¨↓⎕cr'g'}⍬

AndyS

12:08 PM

:57176298 Don't forget ⎕or (or perhaps ⎕fmt ⎕or:-)

Morten Kromberg

@dzaima Once upon a time, APL was "flat" and ⎕CR was the only solution. Later, SOME APL systems added ⎕VR (or 1 ⎕FD as I think it was in SHARP APL) to help people format code the way it would look if you asked the system to print it. Finally, nested arrays arrived ca 1983, and ⎕NR was added.

At the moment we are considering a general metadata function ⎕ATX that would hopefully become the one system function to rule them all.

But we can't get rid of the old ones because they are used in ancient code that just works.

Kamila Szewczyk

12:41 PM

I wanted to make an operator which would extend the domain of some primitives inside the function which is passed to it

 dx←{
     ns←⎕NS ⍬
     ns._Neg←{0=⍵}

     h←{k v←↓⍉2(⊢⍴⍨÷⍨∘≢,⊣)⍺ ⋄ {⍵∊k:⊃v[k⍳⍵] ⋄ ⍵}¨⍵}
     t←'~' '_Neg'
     g←⍺⍺ ⋄ c←↓⎕CR'g'
     s←{{⍵/⍨2∨/0,' '≠⍵}⍵/⍨~+\⍵='⍝'}¨c
     b←↑,/'⋄',⍨¨{⍵/⍨0≠{≢⍵}¨⍵}s
     f←{'}',⍨⍵/⍨1=+\1 ¯1 0['{}'⍳⍵]}b
     e←ns⍎(∊' '(1↓∘,,⍤0){t h ⍵}¨60⌶f)
     0=⎕NC'⍺':e ⍵ ⋄ ⍺ e ⍵
 }

that's how it looks at the moment, and for now supports only ~ domain extension

seems to handle most edge cases properly

AndyS

1:05 PM

@rak1507 I wouldn't rely too much on the result of ⎕size: it sort of returns what would be needed in a workspace if nothing of the object in question existed beforehand, and nothing is duplicated. You're better off comparing the output of ⎕wa before and after having created the namespace.

Having said that, ⎕wa is a heavy operation in that it attempts to squeeze every object to the smallest possible representation, and then compacts the workspace. 2000⌶ may be slightly less accurate, but you'll only hit a compaction if there's not a big enough free space in which to hold the result.

rak1507

Alright thanks

xpqz

1:50 PM

Is there a place where all available ]usercommands are listed?

RGS

There should be, the question really is where is it...

rak1507

 ARRAY         Compare  Edit
 CALC          Factors  FromHex  PivotTable  ToHex
 DEVOPS        DBuild  DTest
 EXPERIMENTAL  Config
 FILE          CD  Collect  Compare  Edit  Find  Open  Replace  Split  ToLarge  ToQuadTS  Touch
 FN            Align  Calls  Compare  Defs  DInput  Latest  ReorderLocals
 LINK          Add  Break  Create  Export  Expunge  GetFileName  GetItemName  Import  List  Refresh
 MSWIN         Assemblies  Caption  CopyReg  FileAssociations  GUIProps  KeyPress
 NS            ScriptUpdate  Summary  Xref

AndyS

@xpqz ]?

As in that's the best we have at the moment. That has the merit of reporting what the current interpreter on the current platform thinks is the case.

xpqz

@AndyS ah yes - nice.

Morten Kromberg

]
all alone on a line will also lead you into it

xpqz

1:57 PM

I only discovered the handy ]map by watching old webinars

AndyS

@xpqz ]map ??? news to me :-)

xpqz

Every day is a school day.

AndyS

But asking the interpreter is the best way to go .. there are some user commands that are not supported on all platforms.

xpqz

Wait, what's this? ]calendar

AndyS

A couple of releases ago we took the decision to abandon documenting each of the user commands in the User Commands User Guide, and instead put our efforts into improving ]?cmd (or better ]cmd -?).

That way the documentation team weren't for ever playing catchup (or possibly whack-a-mole depending on who you talk to :-))

dzaima

2:04 PM

@xpqz there's there's also dfns.cal :)

AndyS

oh yes ! dfns was the brain child of the late great John Scholes, and is so very useful.

xpqz

Box Boxing Disp Display -- only need three more for one for each day of the week.

RGS

2:25 PM

In the Windows interpreter you can change the input history buffer size with "Options" ⇨ "Configure..." ⇨ "Session" ⇨ "History buffer size"

what about RIDE? The menu isn't that big and I looked for it thoroughly many times. Is it some env variable that has to be set in the START/CONNECT menu?

Morten Kromberg

3:44 PM

Calling Mac users: We need to decide where to build Dyalog version 18.1. We previously announced we would move to Catalina. Can anyone think of a reason not to go straight to Big Sur?

ab5tract

@Morten I guess it makes sense for those seeking the latest and greatest to already be on the latest and greatest

Razetime

@MortenKromberg I use El Capitan :(

Morten Kromberg

Mac remains mostly a non-commercial platform for us, and all the commercial customers I am aware of want to go to Big Sur soon. We can generally only be motivated to support old platforms for a fistful of $, I'm afraid.

RGS

I am sure Razetime can pay the big bucks so that El Capitan remains a supported OS version

Razetime

lol

xpqz

3:53 PM

@MortenKromberg always build on the latest version

Razetime

I guess I'll install linux on this thing

ab5tract

@Razetime always a good option for extending the life of macs

i take it yours has aged out of os upgrades?

Razetime

It's a 2013 model

I'm not in a position to install catalina

Morten Kromberg

Version 17.1 should still run on El Capitan, right?

Razetime

I'm running 18 right now

AndyS

3:58 PM

We currently build on High Sierra and QA on Yosemite.

What we don't ever want to do is to change the O/S version (and ideally the compiler) for any released version of Dyalog that is still on support. So we'll leave the machines that are used for 18.0 and earlier alone, and start afresh on new hardware for 18.1.

xpqz

4:14 PM

@AndyS will you also build for Apple-M1/ARM?

AndyS

@xpqz well, let's just say that one of the benefits of moving to Big Sur will be that we'd have the same level of O/S for both Intel and ARM based macs ! Nothing is decided yet, and it is a project that will have to be scheduled in, but I think that we will in due course.

xpqz

@AndyS I've made it my goal to make APL-on-Mac take over the world.

An uphill battle, perhaps.

Nicer error messaging in 18.1, btw: Link Warning: ⎕SE.Link.Create: .NET or .NetCore not available - Link cannot watch directory

AndyS

@xpqz A laudable ambition indeed !

Dyalog APL

<olabaz> Hi, what is the ideal use case for APL, is it best suited for numerics? If so, what kind of numerics?

<Wezl> olabaz: numbers in arrays

<olabaz> Wezl: How does it handle linear algebra operations?

<Wezl> olabaz: I'm not sure but I think there's a 1-character builtin for that

rak1507

4:31 PM

Jan 15 at 23:05, by ngn

apl is essentially linear algebra turned into a programming language

Dyalog APL

<Wezl> olabaz: that's chat.stackexchange.com/transcript/message/56761727#56761727

<olabaz> Interesting. I'm currently using Julia for a lot of my numerical work. How do they compare in terms of performance? Would I gain any advantage by using APL?

rak1507

Not sure how it would compare with performance but depending on what you're doing it would probably be fairly simple to translate

xpqz

olabaz: depending on what your numerical work is, Dyalog also wraps lapack and fftw (github.com/Dyalog/Math)

dzaima

If APL has the necessary builtins to do what you want to do (and you properly utilize array operations), APL will provide nearly the pest performance possible. Not sure how well does Julia utilize vectorization.

xpqz

Julia is good at vectorisation, too.

Dyalog APL

4:37 PM

<olabaz> Are there any benchmarks available comparing APL to different languages?

xpqz

Might have to ask @MortenKromberg or @Adám

dzaima

4:59 PM

in a couple of simple tests, Julia and Dyalog have pretty similar performance. Have to go now though

rak1507

what's the golfiest way to check if f⍣≡ converges? assuming it will in a short amount of time

ngn

5:20 PM

@MortenKromberg what do you perceive as "cheap shots"? are you ethically and responsibly gonna fix ⍳,N, for instance?

user41805

@rak1507 i don't get how this isn't the halting problem, do you mean to find if it converges in X seconds?

rak1507

how about if it converges rather than cycles

I found a solution using dfns.traj but it doesn't seem ideal

user41805

5:36 PM

@rak1507 tio.run/##SyzI0U2pTMzJT/… do you mean like this?

Kamila Szewczyk

is pmat supposed to yield permutations in interleaving parity?

ngn

5:50 PM

what's the easiest way to compute parity? {≠/∊⍵<,\⍵}?

Kamila Szewczyk

edit: i somewhat solved this problem

ngn

"somewhat" :)

Kamila Szewczyk

I was working on implementing the monadic inner product function Iverson mentions a few times in his papers

and I came up with this:

f←{i←⍵ ⋄ o←⍳n←⊃⍴⍵ ⋄ ⍺⍺/(⍵⍵{⍺⍺/{i[⊃⍵;2⊃⍵]}¨o,¨⍵})¨1+↓⍉↑{1=⍵:,⊂,0 ⋄ (⊃,/)¨(⍳⍵)⌽¨⊂(⊂(!⍵-1)⍴⍵-1),⍨∇ ⍵-1}n}

so basically, if you did -f× 3 3⍴⍳9, it works as a matrix determinant

+f× 3 3⍴⍳9 is matrix permanent

this works using leibniz formula for determinants, which can easily be generalized to all rings

I considered using the Mahajan-Vinay algorithm, but it only works over arbitrary commutative rings, so it'd probably exist just as a specific, optimized case

user41805

@ngn is ⍵ not a simple list?

ngn

@user41805 yes, a permutation

user41805

5:55 PM

then why do you have ,\ ?

Kamila Szewczyk

ideas on making it a bit simpler?

ngn

@user41805 because i want to compare each item against the previous ones

@KamilaSzewczyk link(s)?

user41805

oops i misread the scan as a reduce

Kamila Szewczyk

jsoftware.com/papers/satn42.htm

i found another one, but i don't remember where was it

i remember that it mentioned that Dyalog doesn't support monadic inner product ("for now")

ngn

offtopic: i've always wondered why iverson decided to make + and × operands for matrix multiplication / inner product (+.×) but not for encode / decode (⊤ and ⊥). seems a bit inconsistent.

rak1507

6:10 PM

other things f.g are useful and other decodes/encodes seem less useful, so that might be it

ngn

@rak1507 i'm sure many people thought the same about f.g before they saw its uses

rak1507

even in jsoftware.com/papers/tot.htm iverson mentions other uses of inner product

Morten Kromberg

I consider statements along the lines of "they will be resting on their laurels forever, and fighting for preserving the status quo, because that's profitable" as a "cheap shot", when you look at the list of enhancements that Dyalog is producing, and our efforts to encourage the growth of a new array language community.

Competition from other array languages probably make us MORE profitable, since it increases the credibility of the array language idea and we are currently only scratching the surface of the potential market. It is a big problem for us that so many of our APL vendor competi…

@ngn regarding ⍳,N: If we were to change the result, we would make Dyalog APL incompatible with the ISO standard, APL2, GNU APL, APLX, APL+Win, and NARS2000, to name a few. If we managed to convince all our users to suffer the pain of us having broken just about every single application in the world, would you then accuse us of having engineered another cunning act of "vendor lock in"?

rak1507

6:34 PM

related CMC: convert an array with 1 element to a scalar but don't change anything else

Kamila Szewczyk

{1≡⍴⍵:⊃⍵⋄⍵}?

rak1507

fails for something like 1 1 1 1 ⍴ 1

(also for ,1)

Kamila Szewczyk

uh, hm

{1≡⊃⍴⍵:⊃⍵⋄⍵} maybe?

rak1507

fails on 1 2 ⍴ 1

dzaima

@KamilaSzewczyk fails on 1 2⍴2

Wezl

6:45 PM

I was thinking {~≡⍵:⊃⍵⋄⍵}1

but that can't be better

dzaima

@rak1507 just to be sure, 1 1 1⍴⊂⊂⊂1 2 should give ⊂⊂⊂1 2?

rak1507

yep

ngn

@MortenKromberg "resting on laurels" was for refusing to fix known problems in the core language, despite being in a perfect position to do so - small community, control over customers. of course you do try to extend apl in all directions, i wasn't denying that.

Wezl

@rak1507 something like chat.stackexchange.com/transcript/message/56757424#56757424 ?

rak1507

oh

ngn

6:50 PM

@MortenKromberg i wouldn't blame you for not following iso or apl2. i've always thought apl standards don't matter much and it's ridiculous for a "standard" to be paywalled. nobody has any illusions they can just throw away dyalog and adopt gnu apl, for instance.

dzaima

@ngn but "fixing" ⍳,N is still adding another hurdle if you want to move to a different vendor, which is precisely the definition of vendor lock-in, no?

rak1507

@Wezl the proposed solution ⍎⍕ doesn't work for things like 1 1 1 ⍴1 etc

Wezl

@rak1507 wouldn't they both return scalar 1?

dzaima

@dzaima (i have a dfn as 16)

rak1507

@Wezl ⋄ ⍎⍕1 1 1 ⍴ 1

it errors

Wezl

6:53 PM

hm, but why?

dzaima

@Wezl ⍕ gives a rank 3 array

@dzaima (13 as a train)

ngn

@dzaima you think it's realistic to snap your fingers and switch apl impls overnight? that's not going to happen anyway. screw the standards. "living" standards matter more - what people actually use.

rak1507

@dzaima my dfn is 14 I think

dzaima

@ngn then what's the point of discussing vendor lock-in at all?

rak1507

can't easily be converted to a train

dzaima

6:56 PM

@ngn (and all the "living" standards have broken ⍳,Ns too)

ngn

@dzaima well, whose point of view are you taking?

from the point of view of the community, it would be nice to break the lock-in. from the point of view of dyalog, let's not touch it.

Marshall

≍⁼⍟=⎊⊢ works in dzaima/BQN.

Morten Kromberg

If we changed the definition of ⍳,N, the result would be that our customers stopped upgrading to new versions of Dyalog APL, and made plans to get out of APL, since the major vendor of APL had obviously lost it's mind. Or pay us money to put in yet another switch to allow them to stay with the old definition, and drive another silly divide down the middle of the APL community.

dzaima

@ngn from every point of view, I don't see any reason for ⍳,N being at all breaking vendor lock-in

ngn

@MortenKromberg well, exactly

@dzaima fixing ⍳,N is the goal. breaking lock-in could be a means to achieve that goal.

rak1507

7:01 PM

why should ⍳,N be fixed in dyalog specifically?

dzaima

@ngn oh, so fixing ⍳,N is just a tangential thing you've decided to mention for no reason. cool

ngn

@dzaima i use it as a simple example of the general problem

@rak1507 because they are the current monopolist on the apl market

rak1507

what would fixing it achieve though

ngn

this is similar to the situation with internet explorer before firefox took off, if anyone is old enough to remember that

@rak1507 um.. correctness? :)

dzaima

@ngn which is completely unrelated to vendor lock-in

rak1507

7:05 PM

I feel like with things like that they're not enough of a bug to warrant fixing, especially considering how much it would break.

ngn

@dzaima what is your suggestion for fixing it?

4 mins ago, by ngn

@dzaima i use it as a simple example of the general problem

dzaima

@ngn probably to not fix it, and just make a new language that has it fixed

rak1507

@ngn The general problem being that they're not willing to break stuff?

Morten Kromberg

The real problem here is "original sin", the fact that original APL didn't have a "tally" primitive, and so ⍳ was defined as accepting a 1-element vector so you could write ⍳⍴VECTOR. Every single APL vendor has decided to stick with the original definition even though some APL vendors extended ⍳ to work on a vector right argument.

ngn

@dzaima and who would buy that language?

dzaima

7:07 PM

@dzaima only new users would benefit from it being fixed, everyone else suffers from it being fixed. So a new language is the perfect solution

Marshall

@ngn It's fairly different, in that there's not actually a market for array languages (could NumPy make money? Maybe, or maybe everyone would just switch to a fork). There's a market for broadly APL2-compatible languages, which is what Dyalog occupies.

ngn

@dzaima "suffers from it being fixed" - i think the opposite

Marshall

There isn't even a market for SHARP-compatible APLs. Those are dead.

dzaima

@ngn broken code is not fun. The amount of improvement is much less than the cost of fixing old code

ngn

@dzaima if you're relying on wrong behaviour (with ⍳ or anything else), that's a symptom of something more deeply wrong with your code, so you'd better have a look eariler rather than later when it explodes

dzaima

7:11 PM

@ngn or the code being around from before this wasn't the only option

Morten Kromberg

@Marshall As an ex-Sharpie, I still think of J as the super-enhanced SHARP APL.

Or rather, super-rationalised

Marshall

@MortenKromberg Agreed, but it's not really able to support the three co-owners of Jsoftware, so I think my point stands.

dzaima

@dzaima giving a single-item vector argument to ⍳ when you know it works, and is guaranteed to work, imo, isn't at all "relying on wrong behavior" or "deeply wrong"

sure, now you probably shouldn't explicitly rely on it, but it's not exactly anything more prone to breaking than other code

Morten Kromberg

@Marshall Sure

ngn

@dzaima if you're working with multiple dimensions you must add a special case for 1, or else there will be a nasty surprise. i call that wrong.

it has happened to me in golfing. but admittely, that's not a real-world(TM) application, so no billions were harmed :)

dzaima

7:17 PM

@ngn correction: if you're dealing with unknown rank arrays. (personally I wouldn't ever write rank-agnostic code, but for those that do, using ⍳ is unlikely, and the special behavior breaking things would probably be even more unlikely)

Morten Kromberg

@ngn Whatever you think, our choice is purely dictated by our understanding of what would be best for users. Rank-agnostic code is extremely rare in real applications, and you can write a trivial cover-function if you need a rank-agnostic ⍳. If it was more widely used we might consider a new primitive.

dzaima

@dzaima (and given that Dyalog has a max array rank of 15, rank-agnostic code can't really do much)

rak1507

@MortenKromberg What about a 1 element array to scalar primitive? Stick it in front of basically everything and it'll work the way it does just now, allowing you to change the behaviour of primitives with 1 element vectors

dzaima

@rak1507 there is one that fits the wanted use-cases here: ⊃ :)

rak1507

⊃1 2 3 is not 1 2 3

Morten Kromberg

7:25 PM

95% of the conversion could be done by replacing ⍳⍴ with ⍳≢ in applications. But the remaining 5% could be hard to find.

dzaima

@rak1507 oh, you mean changing the language to fix ⍳,N and singleton extension, and spamming that builtin literally everywhere?

rak1507

Yeah, only a few things would need it and like mortern says a s/⍳⍴/⍳≢ would probably mop up most of it but for the rest, use that and then Dyalog can feel free to change stuff

dzaima

@dzaima (which also wouldn't work. Now you'd get scalars where you previously had vectors or higher-rank arrays too)

rak1507

@dzaima no, my suggestion was to change arrays with only 1 element to scalars, but keep everything else as it is

so not ⊃

dzaima

@rak1507 I mean as in 1 ≡ (,1) +⍥SingletonToScalar ,5 which is definitely bad

Morten Kromberg

7:28 PM

@rak1507 But what about arrays that were actually supposed to be 1-element vectors?

rak1507

@MortenKromberg No need to put that new primitive in there then

dzaima

@rak1507 well, how do you find precisely the places, and only the places, where you need it, without doing the same amount of work as just fixing the code?

rak1507

that is a good point

I wonder if there's a way of tracking what and where primitives are used throughout a code base and then automatically refactoring it

That relies on all code being run though :(

dzaima

@rak1507 yep

rak1507

Unless it could be a permanent automatic thing

Morten Kromberg

7:31 PM

If you do start a new array language, my recommendation is that you do not repeat this mistake. But IMHO we are not in a position to fix it, other than adding a new primitive. Alas, monadic ⍸ is already (very nicely) taken.

dzaima

@rak1507 which would be functionally equivalent to (but a slower version of) having a separate dyalog build that keeps the singleton stuff :)

rak1507

@dzaima Not necessarily, potentially you could leave the modified interpreter running for a bit, and then hopefully within not too long most of the code would have been automatically fixed, then the remaining 1% can be manually fixed

Morten Kromberg

@ngn Until then, I think you need to write ⊃∘.,/⍳¨ in Dyalog APL ...

dzaima

@MortenKromberg or just ,¨⍳ (which is also faster)

ngn

@MortenKromberg i sincerely hope you start thinking about managing change before it's too late, so you don't end up repeating history like M$ and the web

dzaima

7:39 PM

@dzaima (oh, it's slower for rank 2 arrays, but faster for rank 3 (or something similar, whatever))

Morten Kromberg

@ngn I appreciate your concern

@dzaima Ah yes, nice

I really don't want more builds of Dyalog APL, we already have
∘.,/(17.0 17.1 18.0 18.1)('Classic' 'Unicode')(32 64)('AIX' 'Intel Linux' 'ARM Linux' 'Windows' 'MacOS')('Debug' 'Optimised')('Runtime' 'Development')
And soon MacOS will split in two

(and some v12.1 builds for special friends, on various POWER chips)

ngn

@MortenKromberg kill classic. or ask for a ridiculous amount of £ growing each year, without the possibility for upgrades :)

Morten Kromberg

We are well compensated for continuing to support Classic

And quite a few features, like the scripting support, are not supported on Classic

arcfide

8:50 PM

@DyalogAPL Benchmarking APL against other languages can be a challenge.

@olabaz , ^^^

Dyalog APL

<olabaz> How so?

arcfide

It's difficult to get to something that people consider a "fair benchmark."

Dyalog APL

<olabaz> ah but that's an issue with benchmarking in general

arcfide

When it comes to microbenchmarks, people usually demand that the two languages be benchmarked for "similar algorithms and code similarity." That is, comparing the same code written in different languages.

When it comes to "realistic" benchmarks, people often want to see their application or a realistic equivalent to it written in the way that they expect it to be written, in two different languages, and then compared for performance.

Unfortunately, both of these are very hard to achieve in APL vs. other mainstream langauges.

ngn

apl vs python may be unfair because the code is either array-oriented enough or not, but apl vs j vs k should be doable

arcfide

8:55 PM

@ngn To be clear, I'm lumping all iverson-style array programming languages under the APL umbrella for this discussion.

ngn

.. vs python+numpy doable too

Dyalog APL

<olabaz> I wasn't able to find any recent benchmarks

arcfide

And in particular, the assumption that most of these implementations for practical systems right now are interpreted.

In the microbenchmarking case, often the whole point of APL is the way in which you craft the solution to a problem leveraging the primitive vocabulary of APL. The general goal is to avoid too much explicit "algorithms" code such as you often see in benchmarks. The result of this for APL is that the microbenchmark isn't really a microbenchmark.

Dyalog APL

<olabaz> I'd be curious in seeing something like: benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/…

<olabaz> which has various algorithms

arcfide

So, for example, there are lots of string mangling or numeric mangling operations for microbenchmarks, but in APL, you'd usually dispatch much of these using some idiomatic combination of primitives, which would then leverage various types of tuned algorithms under the hood. These wouldn't match up against the prescribed algorithm used in the microbenchmark, oftentimes, and thus, the benchmark is considered "unfair."

Marshall

8:56 PM

I did spend a short time trying to compare Dyalog and NumPy and quit because I couldn't get NumPy's own benchmarks running.

arcfide

Right, when it comes to comparing Julia against APL, and especially Dyalog APL, the typical Julia benchmarks aren't much use.

We explored this a few years ago.

Marshall

I'm inclined to say the problem is not any fairness concerns but that no one's spent any sort of effort on this.

dzaima

@ngn but we're comparing Julia here. It has vectorizing operations, and for those it's equivalent. But doesn't hate non-array-y stuff unlike APL, probably leading to performance increase there

Dyalog APL

<olabaz> I see a lot of claim about APLJK performance but I wasn't able to find any evidence. So I'm wondering where the claims are coming from

arcfide

@Marshall I did explore this before.

Marshall

8:58 PM

@arcfide "Explore" doesn't actually suggest much effort?

arcfide

I spent some time specifically looking into the Julia benchmarks and trying to see if there was a way to "write the equivalent benchmark in APL" and I came to the conclusion that any benchmark written in APL would either unfairly biase itself towards Julia or Dyalog.

@olabaz The simple answer is that Dyalog's interpreter is slow, and so if your benchmark comes down to whether or not Julia's compiler will produce faster code than the interpreter's ability to execute equivalent code trees, Julia should win every time.

But if you write your code in a more idiomatic APL style, where you don't stress the interpreter at all, then Dyalog APL is likely to perform quite well.

The problem, though, is that then you probably aren't testing the same things that the Julia benchmarks were intended to test.

Marshall

@arcfide So show both kinds of benchmarks, and if you can, describe when one language or the other would be faster.

ngn

@arcfide with or without 400⌶?

arcfide

@Marshall For some of those I think we could do an alright job, and in those cases, it is a matter of people simply not putting the code together into a unified benchmark.

Although there are some that exist, such as those put out by the Futhark folks and the like.

Dyalog APL

<olabaz> But how big is that difference?

arcfide

9:02 PM

That is problem specific.

dzaima

@arcfide but Julia also appears to take advantage of vector operations (well at least add/multiply/sum is what i checked), so there's not much advantage Dyalog could have either

arcfide

One example, though, of the difficulty would be a benchmark like "binary trees."

Marshall

@arcfide When I think of all the difficulties of running benchmarks, "but how will I assemble them?" doesn't exactly loom large as one of the big struggles.

arcfide

@dzaima Dyalog APL doesn't just win in performance in some cases because of vectorization, but because of data management and a number of other things.

@Marshall It's a lot of grunt work for very little "reward."

@Marshall A group of us at IU were building just such a benchmarking suit, which would have included APL and Julia in it. We got a bit of the ways in and kind of had to abandon ship.

s/suit/suite

Dyalog APL

<olabaz> It's a bit worrying that no benchmarks exist

ngn

9:04 PM

@arcfide chat.stackexchange.com/transcript/message/57181822#57181822

Marshall

Yes, producing good benchmarks is much more difficult than one might expect. Maybe it's reasonable to judge it not worth the effort but that doesn't change the fact that very little effort has been applied.

arcfide

Well, take the binary trees benchmark for instance. From my own personal perspective, the general right answer to that benchmark is, "But why?" That is, binary trees as a generic concept aren't something that you find very often as explicit structures in a wide array of APL programs, but they are something that you find in a lot of programs written in other languages.

@Marshall I disagree that very little effort has been applied.

rak1507

Sounds like a load of excuses to me

Marshall

@arcfide Give an estimate then.

arcfide

As an example, when writing my compiler, there are a lot of things that you might thing would be required to build that compiler, including a few tree like structures for doing certain look ups and indexing and the like. But those can be off-loaded to the underlying primitive implementations of Dyalog APL's primitives, which are able to leverage things like hashtables under the hood.

But if you tell someone that you can benchmark a hash table implementation in Dyalog APL through the use of the Index Of primitive, most of the people I've talked to have explicitly disqualified that.

But if you then implement a hash table explicitly in Dyalog APL without using any of the primitives, I doubt very much that it will be faster than most of the other implementations, or even remotely competitive.

So then, people go, "Well, if I need to use a hash table, APL's the wrong tool." Except that often that's just not true.

Dyalog APL

9:08 PM

<olabaz> arcfide but what's the issue with just including both?

arcfide

@olabaz How do you represent that in the benchmarking in a way that people consider it legitimate?

Dyalog APL

<olabaz> people benchmark loops in python using bare python and numpy for example

arcfide

@Marshall We worked on it for about 6 months and got almost nowhere.

@Marshall Another group had multiple years into the work with little to show for it.

I've known other groups working just as hard and struggling just as much, even with non-APL languages.

Marshall

@arcfide Exclusively??????????????????????

olabaz: There aren't cross-language benchmarks but there are a fair number of performance tests used at Dyalog. I was the most performance-focused implementer at Dyalog when I worked there and I was generally either comparing my work against maximum memory throughput for simple things like arithmetic, or known state of the art for other things like hashing and sorting. I don't think in these cases comparisons against other languages would have been very useful.

arcfide

@Marshall Some were tasked exclusively on the project (maybe 1 or 2 people) but the rest were probably doing between a third and half of their time on it.

@olabaz I totally agree that getting good benchmarks would be really nice.

Often with some of them, like the language shootout benchmarks and other ones like it, the rules preclude approaching the problem in what I would call an "APL style" which makes it pretty demotivating to want to work within those frameworks.

Marshall

9:12 PM

@arcfide Is there any public material on what this group did?

arcfide

And in that case it's purely a "you're unfairly representing APL" kind of thing.

@Marshall It might have been brought down, but I can try to dig some of the stuff up if you email me about it. It was probably about 3 - 5 years ago, as we actually have a couple of IU alumns who were involved with Julia and a number of vector algorithms things.

Dyalog APL

<olabaz> I mean APL was developed in the 1960s and no one has been able to produce any kinds of benchmarks since then?

arcfide

@olabaz As another example, this time in the "real application" space instead of benchmarks, I was working with a potential client who had heard me comment about APL and key-value datastores.

We got to talking and we started building up some benchmarks.

The first hurdle was that we had to agree that there wouldn't actually be a separate "key value store" application to do the key-value store database benchmark. That was itself difficult to get through.

Second hurdle was that when we first did the initial benchmark test, the APL code was essential constant time fast for a very small constant, so the benchmark was almost useless.

Finally we came up with a benchmark that actually demonstrated some work that could push both of the systems and wasn't trivial in APL, and at that point Dyalog APL still outperformed the other system (Haskell + specialized key-value store).

But that was a private benchmarking project for a specific interested party.

But that was really a matter of taking a completely different approach to the problem in its entirety, and there was very little that would have been recognizable as the "same" in the code between the Haskell/KVstore code and the APL code.

So, it is a "fair" benchmark in that we were trying to plot out whether APL would be usable for solving these particular set of problems, but it wouldn't at all be considered a fair benchmark for "show me your key-value store library benchmarks" .

There are a lot of benchmarks around, and you can see talks at Dyalog.tv about business reporting on their benchmark efforts and some of the comparisons that they made, but in terms of the classic "language shootout" type benchmarks that are language against language types of micro-benchmarks or "small application" benchmarks, there are a few floating around, but very little that I would consider complete enough to warrant representing Dyalog APL or other languages in a broad and fair light.

Dyalog APL

<olabaz> The only benchmark I was able to find is from the 90s: chilton.com/~jimw/bnchmrks.html

<olabaz> APL is about 5x slower than C

arcfide

@olabaz @Marshall gave a great talk about his benchmarking of APL against C for a number of algorithms as well as the implementation of search in APL.

Dyalog APL

9:23 PM

<olabaz> where are those results?

arcfide

dyalog.tv

dyalog.tv/Dyalog18/?v=paxIkKBzqBU

Marshall

@arcfide Yeah, I guess I'd like to see a summary of the work done if there's anything available. I'm not sure I'm really understanding what you wrote correctly. You're saying a team including one or two full-time researchers and several part-time started a project to compare performance of several vector languages, and after six months stopped with the conclusion it was just too hard?

arcfide

@Marshall Do you have your C v. APL talk somewhere? I'm having trouble finding it.

@olabaz I also include a large number of tree-related benchmarks in my thesis: scholarworks.iu.edu/dspace/handle/2022/24749

These are benchmarks against Chez Scheme and Racket Scheme.

Chez Scheme being very good in this arena of tree manipulation.

Marshall

olabaz: See my publications page at mlochbaum.github.io/publications.html . The four performance-related talks (Moving Bits Faster, Interpretive Advantage, Sub-nanosecond Searches, and Implementing Reduction) all feature benchmarks of the features in question prominently.

ngn

a fair benchmark can only be performed by a third party without financial interest in either outcome. that's expensive.

Marshall

9:27 PM

I gave the C versus APL example first in Moving Bits but Interpretive Advantage has an updated version at the beginning.

arcfide

@Marshall The conclusion was basically that the amount of time and effort and social and political problems to overcome vastly outweighed the gains. There simply wasn't a path forward that would lead to anything usable by any party any time soon and it would have required pretty expensive funding and a lot of social discourse and problem solving to get off the ground.

Marshall

It's only for one particular problem, and it's more of an existence proof (APL code can be faster than any reasonable implementation in portable C) than a general comparison.

arcfide

It's definitely doable, and I think it's still worth doing, but it's much more espenxive and intensive than a lot of people think.

ngn

"apl faster than c" - bollocks

arcfide

We essentially ran into every problem in the book.

Dyalog APL

9:29 PM

<olabaz> Thanks I'll take a look

arcfide

@olabaz Now, as a Julia user, if you're interested in building up some benchmarks, I would love to support you in that effort, and if you want me to look over some APL code for a specific problem and try to help tune it a bit, I'd be happy to try to give you my thoughts.

ngn

"well-written apl faster than poorly-written c" - possible

Marshall

@arcfide What was the motivation for the research? If it's not just to produce informative benchmarks then the conclusion isn't that benchmarks are too hard, it's that no one wants to do them.

arcfide

@Marshall We were a language design group with multiple projects occurring in multiple languages and we needed a way to build a cross-language benchmarking of the various overlapping projects that were going on across many different languages (Rust, Julia, Haskell, APL, Scheme, &c.) all of which were centered around performance and parallelism.

We were trying to get a unified benchmark suite or even just technique so that the papers weren't so ad hoc and full of benchmarks that were hard to replicate and compare across papers and teams.

Marshall

olabaz: The Outer Product slides have the most significant comparison of C versus Dyalog, testing outer products across many operations and sizes. It's the last section at mlochbaum.github.io/OuterProduct/commentary.html .

arcfide

9:32 PM

This is a pretty well known issue in the PL community, but it might be solved by now (I haven't heard of a solution).

And if it makes any difference, a relatively significant portion of my own work on my own thesis for a tiny selection of operations and only two languages was spent on the benchmarking question, and it was a challenge just getting that through a group of four other people who were rooting for me.

The benchmarking isn't hard from a technical standpoint, it's hard from a social and political standpoint, and it's very time consuming.

Marshall

@arcfide So maybe the problem was more in getting developers of other languages to adopt the system rather than just building it? Or generalizing to arbitrary languages including ones that didn't exist yet? Making a framework sounds like a bigger task than just comparing a specific set of languages.

arcfide

@Marshall Well, we were just doing it internally first for ourselves, but we had a struggle even amongst ourselves who were all motivated to get it done. And we had a specific subset of languages that we wanted to compare first.

dzaima

@ngn I doubt not making sure the compiler properly produces vectorized operations can be called poorly-written C. But writing APL well really just requires you not writing it horribly (i.e. no ¨es) and you're pretty much good

user

@dzaima Not completely related, but does ¨ perform poorly? I thought it'd be optimized for certain cases, at least

arcfide

@ngn "Language faster than Langauge" is almost always bullocks for most mainstream commercially viable languages, because there's always a theoretical path to the metal. But what that phrase usually means is "what people do with language X willingly and naturally often ends up faster than what people willingly do with language Y." But that's a lot of words to say.

Marshall

9:38 PM

@user It's optimized for specific cases, which I don't believe are documented.

arcfide

I think the HCI question of "how people will end up using a language" and how that performance falls out is a more interesting and valuable question for actual application design and performance.

dzaima

@user not if you're ¨ing a complicated dfn at least

user

@dzaima Ah, ok. I'll keep that in mind

rak1507

@user in general the more ¨s the worse your code is, I've found even slightly contrived ways of doing things using arrays are generally much faster

arcfide

@olabaz I've got to take off now, and I'm sorry that there isn't a really great, "look here" answer right now, but I do hope that this will improve in the future.

Dyalog APL

9:39 PM

<olabaz> alright,

<olabaz> thanks for the references

user

@rak1507 I guess that kinda makes sense

arcfide

Maybe the APL Orchard could all get together and start collaborating on a big giant set of APL implementations for the benchmarks games or language shootouts or other things? That would be fun, if nothing else.

@Marshall One of the big problems we encountered with our benchmarking efforts was deployment and building. A lot of the existing benchmark code that we tried to run just failed and couldn't be replicated at all for any result. It also didn't help that vectorization at the time was notoriously finicky in some of the languages, which made it even harder to get right.

And we also encountered huge variances in results depending on the machine configuration.

Anyways, see y'all later!

Transcript for