When I see ¨ i think 80% of times (it depend next operation) there is one unnecessary array duplication... it is right what I think or internal the program all it is seen as a normal loop? Possible I remember wrong... {(⍺⍺¨⍵)/⍵} as a filter for me is well and right.

⎕←{0=3|⍵} {(⍺⍺¨⍵)/⍵} ⍳20

@RosLuP
3 6 9 12 15 18

The problem is reduce the different way to do something, in that count even function and operator; and use only the ones resolve a range + big of problems

{(⍺⍺¨⍵)/⍵} possible better defined as a normal function with a normal loop without the each

I not speak for codegolf code

@RosLuP a tradfn with a :For loop?

I find {f(⍵)} functions and ¨ operator can simulate many loops; possible the one not in that set are the ones I don't know where the loop stop...

@ngn yes using Apl goto instruction for the loop --> ; but in that case would be equivalent to {(⍺⍺¨⍵)/⍵} because I see in both cases one binary array create

@RosLuP so, you think that would be faster? could you post the code you have in mind?

Wrong it is better the normal loop because it is possible to write it without create intermediate array of binary for calculation in pseudo code if aa is the filter and w is the array {j=1;r=@;k=len(w); goto B; A: if aa(w[j]) then r=r,w[j];++j;B: if j<=k then goto A; return r} no intermediate binary array only the array result,

@RosLuP right, i see what you mean by "better" - it uses less memory

@ngn yes one can imagine array of million of elements; duplication should be something one not want... I know possible memory is not a problem for future generation of pc...

@RosLuP I don't think there'll often be a case when doubling ram would cost more/be harder than getting 20-50 extra CPUs :p

and if you're really dealing with hundreds of gigabytes of data (a million is tiny) splitting that up into chunks shouldn't be that bad of a trade

Still, @arcfide, I think it'd absolutely be worth drawing those points out in a blog post - how removing the each and being less abstracted changes performance

:47196849 Sorry, pressed enter before I was finished - took it as autocomplete on the @

In particular: the trade-off (or false trade-off?) between generalised iterators (filter etc) and the inline, 'whole array at once' solutions. This is pretty much what I was getting at with stackoverflow.com/questions/51916990/…; also, there was a good exchange about it 8 (!) years ago here: reddit.com/r/programming/comments/ce7cx/comment/c0s93d8/…

@arcfide So are there better ways to "write an expression to give an array" than "apply a function to an array?"

@RosLuP the thing is that in APL for evaluating things on a big array to be fast, it has to be all primitives executed with no ¨ anywhere, so everything could be optimized as vector operations. An implementation of F would have to do ¨ one way or another.

Executing 0=3∣ a million times on single items is way, way slower than executing it once, on an array of a million items. That's the way APL works.

0=3∣ has to be executed ' a million of time' to all single argument of array for return the right result, but possible CPU has some instruction for doing all in a time or use multi CPU on the same array parallelize it...

@RosLuP That's exactly what happens if your code doesn't have ¨, but if it does, the interpreter has no choice but to evaluate the dfn many times as it might contain a ⎕← which is required to be executed many times

"to all single argument " above is wrong better 'to all single element'

  {(0=3|⍵)/⍵}⍳10000     → 5.5E¯5 |      0%
  {({0=3|⍵}¨⍵)/⍵}⍳10000 → 1.1E¯3 |  +1992% ⎕⎕
  {0=3|⍵}F⍳10000        → 2.0E¯2 | +35871% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕

your F is 360x slower than {(0=3|⍵)/⍵}.

gotos are incredibly horrible & slow always, so I don't know how you expected your function to be fast

It seems to me 20time slower 2x10x 1e-3=2e-2

@RosLuP that's comparing with {({0=3|⍵}¨⍵)/⍵}⍳10000, which is the un-APL-y version

and 20x slower than already horrible code is hard to achieve

But the right piece of code to compare against F is {0=3|⍵} {(⍺⍺¨⍵)/⍵} ⍳10000

@RosLuP but that is horrible code

the whole point of APL is to do everything on arrays as wholes, not on the individual items.

We have to compare {(⍺⍺¨⍵)/⍵} with F not other...

@RosLuP and your r←r,... which is already O(n), making the whole function O(n^2) isn't better than / at all

Than write a function or funtor that does what does F (filter) that has general Boolean function and one array as arguments and compare speed ...

@RosLuP but that is a horrible abstraction function to have in APL. It shouldn't ever ever be used.

I just don't understand you wanting that function if it's gonna be slower always than writing proper APL

@RosLuP ok. Changed that ⍳10000 to ⍳1000000 and now the performance difference between {0=3|⍵}{(⍺⍺¨⍵)/⍵}⍳1000000 and {0=3|⍵}F⍳1000000 is 245x.

F took a whole 27 seconds while {(⍺⍺¨⍵)/⍵}⍳1000000 took 0.11, and {(0=3|⍵)/⍵}⍳1000000 - 0.0048

@Adám is this a bug? The funcion seems to work fine with odd numbers but not with even numbers ⍨. Also, for some reason the error looks weird

@J.Sallé {⎕←⍵ ⋄ ⍵=r←⍵*⌊/⍎¨⍕⍵:r ⋄ ∇r} 2 explains the error

@dzaima ah, I see

Although I think the ⍎SYNTAX ERROR pointing to a dot thing is kind of a bug?

It's just reporting that ⍎ hit a syntax error. Which was .. You get the same message, minus the ⍎ when you just enter a .

Oh, that explains it. Looked weird to me since I'd never seen that before

@RosLuP Here's another core "mental shift". In addition to what I wrote the other day, one is best served to stop "thinking with loops" and stop any notion that APL is "simulating" anything. Both of these enforce a mental notion of abstractive indirection that is antithetical to high-performance, direct, readable APL code.

This is much more pervasive than it might at first appear.

In the short term, the use of an explicit looping construct, or the use of Each, enforces element at a time execution of complex interpreted functions over small amounts of data with significant structural overheads in order to develop the correct solution.

In the case of the current and foreseeable interpreter technology, this is obviously a problem across the board.

It will have a tendency to blow your cache.

It will require significant amounts of CPU to interpret the code instead of working over the data.

It will almost certainly ruin the CPU's ability to do good prediction.

It removes any ability to leverage vector SIMD instructions on the data.

It results in heavy amounts of pointer chasing.

And so forth. And that's just at the interpreter level.

Now, the cases where this might be useful is when you are working on chunks of large amounts of data that need to be processed, but that data is too large to fit into memory. In that case, if MMAPing files isn't going to work for you, then chunking the data up and loading in slices at a time can be useful, and there the overheads of Each are overwhelmed by the amount of data you are working with.

However, that really only comes into play with data in the 32GB+ range of working memory requirements.

Anytime where you begin to think in terms of low-level, inner code iteration using complex (non-primitive) user-defined or derived functions on small sized arrays, you're destroying your code on multiple leves.

Not the least of which is a reduction in clarity and directness of solution expression.

The direct solution often performs significantly better, but is also more readable.

It's very often the case that the simpler, more direct solution in APL is among the highest performing solutions.

There are exceptions, but those are easy to understand with basic algorithms theory.

But it's not just interpreters and current systems optimizations.

One might say, "Well, a sufficiently smart compiler will handle that for us."

And you'd be right.

Look at the mountain of research on polyhedral loop optimization and functional typed function fusion literature that is out there.

But we might never get a sufficiently smart compiler. (I guess that depends on you…)

They are able to take extremely complex and indirected code and turn it into something fast.

Well, that's the solution then, right?

Not really.

The value of a compiler that allows you to get performance from very poorly written, less readable, less clear code is questionable at best.

But it gets worse.

These algorithms are insanely expensive, break down, and often can't handle programs of any meaningful size.

If you wrote APL like that, it would be very easy to result in quad+ nested loops on a single line, and then have a series of 20+ lines of that code in a single function.

That quickly begins to tax such systems beyond our current patience for compiler intelligence.

APL is a tool of thought, and if you can't think fast enough, no amount of tech is going to be worthwhile.

And if it takes 40 minutes to get an answer or 5 hours to compiler the answer to improve the performance to 1 minute or one could simply rewrite the code to be clearer, shorter, more direct, easier to read, and faster at the same time without the need for a compiler, which one are you going to take?

And the distance gets even wider from there.

Because not only is that easier to read code easier for the human to read, it's also vastly easier to compile efficiently.

There are issues with the current interpreter technology related to excessive intermediate arrays (the interpreter actually does a better job than you might think at this), and those can be solved, but the good news is that not only can those things be solved, but if you write canonically good APL code, the issues can be solved using very dumb/simple algorithms for doing this fusion, and across a wider array of possible code forms, than is possible with the vastly more complex solutions.

This is a uniquely valuable quality of APL to the compiler writer.

In the same way that lexical scoping helped functional programming languages handle closure creation in a sane way that was performant, the directness of expression in APL permits most widely desired fusion operations to be performed with a high degree of success using dirt simple algorithms that don't require any explosion in the asymptotics of your compiler.

Btw, @arcfide I really really appreciate your putting so much effort into enriching this room.

I can't help myself. LOL

It's like an OCD or something.

Anyways, the moment that you introduce a goto, a loop, an Each, or some other form of such statements into the high-performance internal parts of your code (the "inner loop" sections of code written in other languages), you destroy that symbiosis and make it significantly harder to get good performance.

And what's worse, is you don't gain anything in terms of readability, ease of manipulation, efficiency of coding, or the like.

It's basically just 95% bad news across the board no matter what platform for APL you have.

The 5% good is that it allows someone who doesn't understand the canonical APL form to write something down, even if it works more like their traditional FP languages or Python types code, since that may be all that they know. And from there, one can begin to see patterns that allow for the manipulation of that code into something more canonical.

So, it can be a good sketching tool, and a good learning tool, and it's helpful at the out edges of code to orchestrate chunks of data munching, but there are very clear lines about where that is good and bad.

When I give workshops, I rant on this a bit each time, to ensure that the functional programmers in the crowd understand that they are almost always wrong at the beginning when they start trying to use Each, Power Operator, or the like to solve their problems.

And I also tend to tell them that they are also probably wrong the moment their function nesting depth goes above 1.

And by that I mean the {}'s.

@arcfide Do tacit functions count?

Tacit functions don't really count, but yes, I would say that if a beginner has managed to start doing tacit programming and the nesting level begins to exceed 2 then that newb should take a step back and re-assess.

Particularly on the sorts of problems they are working with as beginners.

As a showcase for this, I just committed some sketchwork of a new version of my compiler:

github.com/Co-dfns/Co-dfns/blob/master/cmp/e.cd

This is a full compiler front-end, and you can go back and see previous versions of it that do some of the optimization passes, too.

It's doing a lot of work.

Or at least a lot of work when you think about a traditional compiler front-end.

But ask yourself how many Eaches are in the code, and where they are.

You'll notice that there are very few, and the ones that are there, follow a very particular, peculiar pattern.

You'll also notice that there is only a single function in there that uses a second internally defined function (loc).

Thus, the working nesting level for the code there is almost always 0 or 1.

In terms of lexical scopes created by dfns.

Most ¨ are in the phrase shape⍴⍨¨≢i

Furthermore, you'll see very little to no traditional function abstractions.

It's almost all direct APL.

And there certainly aren't any helper functions in the traditional sense.

@Adám Exactly.

Indeed, the Each's in these cases are literally superfluous syntactic candy, and have no real meaningful impact on the code itself other than to cluster a few bulk operations together.

The only other Each pattern is the Idiom ≢¨ applied to a vector of vectors.

Btw, I think you can get a speed-up with ⊢∘⊂⌸ → {⊂⍵}⌸.

@Adám I can. :-) But in this case I don't want to use that because of the style issue. The performance of the APL code there isn't an issue, so the Idiom boost isn't much desired yet.

When I actually benchmark I do change that.

And as a little teaser, after I figure out this blasted complexity that is Name resolution, three of those passes will probably be disappearing into something a lot shorter for each.

@Quintec I don't think I answered your question directly. In short, rather than apply your own user-defined function to an array using either each or writing the function so as to handle array at a time thinking, just inline the expression directly where it is used rather than creating a function abstraction to contain it. That is usually the right answer.

Yes, I understand this. My question was about your line of running saying that APL is about applying a boolean array rather than a boolean function. But how can you create a Boolean array without applying some Boolean function to an existing array? @arcfide

@Quintec In most cases you need an expression that is a composition of primitives that will result in a Boolean array. This is, in the strict sense, the application of a series of Boolean functions to an array. However, I was just trying to highlight the difference between a composition of primitive Boolean functions composed together to form a direct logical expression as the left argument to Replicate, and the use of a separate, user-defined Boolean function being applied.

A Boolean function which is lifted to the higher-ranked array domains is more reasonable than a Boolean function that works only on scalars, but the main point I was making was about avoiding, if possible, the indirection of having a separate Boolean user-defined function rather than just having that function inlined.

In particular, if F is a function with body X, then (X[A → ⍵])/A is usually preferable to (F A)/A.

Where [] in this case is the traditional substitution concept in PL Theory.

And (F A)/A is much preferable to (F¨A)/A.

And both are significantly better than F filter A where filter ← {(⍺⍺¨⍵)/⍵}.

When I was talking about using a Boolean array, I wasn't talking about applying it, but using it as the left argument to Replicate, if I recall correctly. Sorry for any confusion.