The APL Orchard

dzaima

12:00

@MartinJaniczek multiline dfns seem to work fine, but you still have to not ever access any variables defined outside of it

Martin Janiczek

That's expected, that's why I have the left side argument

are name lists allowed on left side of function name in tradfn header?

r←(lines_file allfields wave namespace) Process (length padding readstart)

It tells me it can't fix the file but doesn't tell me why

dzaima

@MartinJaniczek it appears not

Martin Janiczek

r←x Process (length padding readstart)
lines_file allfields wave namespace←x

indeed helps

⍝ but then
x←{leftargs {
  bla←123
  1
} ⍵} IÏ 1 2 3 4 5 ⍝ gives FUTURE ERROR: 6: VALUE ERROR, where ¨ would work

dzaima

@MartinJaniczek leftargs isn't defined in the operand of IÏ, is it?

Martin Janiczek

it's a variable from outside scope

dzaima

12:10

@MartinJaniczek isolate doesn't handle scope at all. e.g. x←1 ⋄ {x} IÏ 1 2 doesn't work

Martin Janiczek

So do I have to create isolates myself and is1.data ←5 6 7 8 or something?

dzaima

@MartinJaniczek or pass leftargs through an argument to IÏ

(e.g. {leftargs id←⍵ ⋄ … } IÏ (⊂leftargs),¨1 2 3 4 5)

Martin Janiczek

That .... seeeems to work! {scope w←⍵ ⋄ ⍴scope} IÏ (⊂⊂leftargs),¨1 2 3 4 5

It works with the full-fledged dfn \o/

Adám

Sounds like limitations in the models. Obviously, the idea here is to have ∥¨ just like we have &¨

dzaima

@Adám as far as i understand, without making the entire Dyalog interpreter thread-safe, that'd require something like copying the entire workspace to each thread though

Adám

12:23

@dzaima The interpreter is thread safe, I think. & works fine, and from a computational standpoint, there should be no difference.

dzaima

@Adám The dyalog interpreter definitely cannot handle multiple operations working from different threads on the same object.

Adám

      a⊣⎕TSYNC{a+←1}&¨⍳1000⊣a←0
1000

@dzaima Like ^?

dzaima

@Adám they're not happening at the same time, which is extremely important (i.e. you don't have to worry about cache, refcount changes on an object at the same time from multiple threads, etc)

this is writing "random" data inside immutable arrays. That absolutely doesn't work if another thread could, at the same time, be reading that.

@Adám (and with & → ∥ there, the result could technically be any number from 1 to 1000, assuming a+←1 is equivalent enough to a←a+1)

Adám

@MartinJaniczek So, it is working? Otherwise, I have another idea for you to try.

Martin Janiczek

@Adám I have just made the chunked parallelized approach to work too.

Adám

12:34

Cool.

Martin Janiczek

Needed to do

process_chunk←{
process←{
...old
}
process¨⍵
}

because process¨¨ by itself didn't work

Adám

OK.

And passing in the arguments?

Martin Janiczek

all on the right side. (⊂leftargs),¨data

then destructure inside the process dfn

I'll try to run it on something big now

Adám

Another approach I just tried was to package all the constants into a namespace, and then pass that scalar namespace as left argument.

Martin Janiczek

Does that prevent copying of the args, or does that happen anyways?

Adám

12:40

It'll have to copy the namespace to the other processes, but it might make your syntax nicer instead of having to build pairs on the right.

Martin Janiczek

@Adám Right, makes sense.

On 3682 x 4172 items: serial 12s, chunked serial 12s, parallel errors out (all processes in use), chunked parallel 18s. So the threshold is apparently somewhere higher. Will try 47622 x 8739 items now (that took around 20min serial last time I checked). Don't have anything in between :)

dzaima

47622 x 8739 indicates what?

Adám

array size.

Martin Janiczek

47622 rows to process, 8739 is just "size" of one row

so I'm parallelizing over the 47622.

dzaima

ah

Martin Janiczek

12:52

Serial: 4m 40s, must have remembered wrong. Chunked parallel still running (started it a minute ago), but in htop I only see one core pinned to 100%, which is suspicious

Ah, two cores now. Weird

Three cores and then "Interpreter exited with code 137" or something like that :D

dzaima

@MartinJaniczek might be that they were waiting on files and so not using much cpu

(how big are leftargs and data?)

Martin Janiczek

I'll try NGET instead of MAP, but other than that I dunno. If going down the chunking rabbit hole, I might as well read multiple lines instead of just one ... move more and more stuff from the "inner" process fn to the process_chunk one...

data is a 47622 x 3 matrix of integers (line length, line start offset, line padding to 8byte boundary)

leftargs: string, 8379 (the other dimension) strings, string, string

dzaima

so the copying of data to each thread couldn't be much of a problem

Martin Janiczek

can I get anything interesting from a core dump that APL wrote to my CWD?

dzaima

you can )load it as a workspace i think and look around

Martin Janiczek

12:59

` is not a ws.`

$ file 7059
7059: Dyalog APL aplcore version 18.4

dzaima

ah, )copy instead of )load

Martin Janiczek

That did it, thanks!

dzaima

though i don't think you'll get much out of it

Martin Janiczek

it doesn't seem to even remember variables

alright back to tweaking the process fn

dzaima

@MartinJaniczek might be the isolate process, which wouldn't have your variables, other than f

Martin Janiczek

13:03

Can't seem to get anything out of it. Also, NGET is a no-go - it only reads a whole file.

I saw the memory consumption rise quite a lot in htop, might be just that it got OOM-killed

Martin Janiczek

13:14

I've tried MAXWS=14G and now it's kinda hovering around 3-4 cores and eats all of that memory. Didn't crash so far.

Makes me wonder whether Dyalog APL somehow reacts to malloc errors and goes to do some memory-reducing CPU work for a while instead of whatever just failed, or what :)

dzaima

how big is that file, and what kind of things are you doing on it? (just indexing some small locations, or applying things to the whole file?)

Martin Janiczek

I'm reading a line at a time (each line is a JSON object). The whole file is 4.3 GB, there are 47622 lines, and the lenghts vary around 95k characters

so for each line I do

dzaima

@MartinJaniczek Dyalog does have garbage collection, but unless you're using namespaces, it should never really do anything. More likely would be unmapping mapped parts of the file or something

Martin Janiczek

padding↓0⌷80 1 (length+padding) ⎕MAP lines_file 'R' readstart

I suppose I could attach strace to the process to see the mapping and unmapping, hmm

welp, I'm getting FUTURE ERROR when processing the results of the llEach call

it took way too long anyways (more than the serial function)

dzaima

is 0⌷80 1 (length+padding) ⎕MAP … somehow different from 80 (length+padding) ⎕MAP …?

Martin Janiczek

13:28

probably not (must have been a leftover from figuring stuff out)

Oh there is a difference in the axis or something

0⌷ leaves shape <N>,without it it's 1 <N>

dzaima

@MartinJaniczek I also removed the 1 from 80 1 (…)

Martin Janiczek

Ah I didn't notice that. Will try

dzaima

(not that this will somehow fix the performance issue, just a random note)

Martin Janiczek

That looks like it's doing the equivalent work, thanks

Martin Janiczek

15:05

This got posted to r/adventofcode, sounds like a nice challenge csokavar.hu/projects/casette

Adám

Sounds fairly easy in APL.

rak1507

yes, although I'm not old enough to know what ' According to the old Datasette manual the casette stores a sequence of 1000, 2000 and 3000Hz audio samples to encode the digits 0, 1 and 2 respectively. ' means

Adám

Commodore Datasette

The Commodore 1530 (C2N) Datasette, later also Datassette (a portmanteau of data and cassette), is Commodore's dedicated magnetic tape data storage device. Using compact cassettes as the storage medium, it provides inexpensive storage to Commodore's 8-bit home/personal computers, notably the PET, VIC-20, and C64. A physically similar model, Commodore 1531, was made for the Commodore 16 and Plus/4 series computers. == Features == Typical compact cassette interfaces of the late 1970s use a small controller in the computer to convert digital data to and from analog tones. The interface is t...

rak1507

Ah, but how do you tell what Hz an audio sample is?

Martin Janiczek

@rak1507 The numbers plot a sine wave + each row = 4ms

rak1507

15:11

Oh right cool

Martin Janiczek

I do not know how to count number of peaks in an array (the peak is not always the same number), but perhaps some histogram thingy could work

Adám

@MartinJaniczek Just look at the sign of the pairwise difference, where adjacent signs differ, there's a peek.

Martin Janiczek

<mind_blown.gif>

Adám

(With slight adjustments if there are plateaux.)

Martin Janiczek

as luck would have it, there are

rak1507

15:18

dzaima.github.io/paste/…

(spoilers)

Martin Janiczek

Post that to reddit.com/r/adventofcode/comments/kmzuc8/… and blow their minds :)

rak1507

Needs golfing first ;)

Adám

@rak1507 ⍎'-'⎕R'¯'⊢⍵}¨ → ⍎¨'-'⎕R'¯'¨

@rak1507 {⍵⊂⍨(≢⍵)⍴8↑1} → (⊢⊂⍨≢⍴8↑1⍨)

@rak1507 {3⊥1↓¯2↓⍵}¨ → ¯2(3⊥1↓↓)¨

Remove the space.

rak1507

dzaima.github.io/paste/… golfed a bit

In dzaima ⍬8⍴whatever would work

¯2↓1↓ can be 5↑1⊖

Adám

@rak1507 What happened to the negative sign transform?

rak1507

15:28

Unnecessary

Martin Janiczek

0.5 vs .5?

Adám

Ah, you're looking for peeks, right.

rak1507

Oh yeah

Adám

⊃¨? Oh.

rak1507

To get the difference between the first two values

Adám

15:31

This is nice (=no parentheses) except for that missing extension to ⍴.

rak1507

Further progress ⎕UCS 3⊥5↑1⊖⍉(8÷⍨≢d)8⍴d←⌊.5-⍨40÷⍨|-⌿2↑⍉↑⍎¨⊃⎕NGET'thing.txt'1

Transcript for