@MartinJaniczek i don't think it's that important to the room, multiline messages work just fine (ctrl+k to monospace exists), my paste is mostly just for syntax coloring
So, question. Where did I go wrong in my // "compress second col by first col" idiom? is.gd/mjcompress ... I'd expect there to be only 3 elements (the non-empty ones) in the final thing
(btw the CSS for the "view with syntax coloring" mode could be improved - I can't scroll in there)
Thanks a lot everybody. I really appreciate all the help. I've managed to get the tree data into the shape I wanted - huge accomplishment for me :) is.gd/mjjson
(My backend colleagues are, over the holidays, playing with and trying to replace their legacy data crunching solution in Go+ElasticSearch with D and Rust and what not. I started learning APL at the same time - so I'm curious whether I'll manage to get near / improve on their times / memory+disk consumption for the same task)
I didn't yet try to get a 5GB JSON file into APL but I heard there are some mmap niceties, so I'll probably have to read up on those.
Who was it that recommended I use array lookup syntax for hashmaps? I have no idea why I didn't implement it earlier. I'm writing a Matrix chatbot in Kap now, and finding a specific value in a JSON parse result is so clean: res["flows"][0]["type"]
@EliasMårtenson That should be res[⊂"flows"][0][⊂"type"] at a minimum - without the map extension, a[⍬] (and so equivalently a[""]) always results in an empty array, which would clash with getting the value corresponding to the empty key
I see, so you catenate the value on the right, because if we have IO=0 then index generator produces values 0..n-1 and without catenation it would break for ⍵=2 or ⍵=3
@EliasMårtenson "dyalog extended" has nothing to do with dyalog (the company). it's adam's source-to-source golfing transpiler running on top of dyalog (the software).
@EliasMårtenson there's a complete definition as ⌷⍨∘⊂⍨ (but that doesn't work in your impl). Regardless, there's a pretty trivial alternative - {x←⍵ ⋄ {(⊂⍵)⊃x}¨⍺}
(also fwiw your ⌷ is "wrong" - 1⌷"ab" "cd" is "cd" whereas it should be ⊂"cd" (as ⌷does more than just get a single element))
OK, I have an explicit test case for scalar arguments to 2-arg invocations of EACH, but not for one-arg. And that's the one that fails... I've added the appropriate tests now.
@EliasMårtenson (⊂,1) ≡ ,¨1 should be true, but is not
(similarly for the dyadic case, (⊂1 1) ≡ 1,¨1 should be 1)
(and fwiw, TIOs dzaima/APL also has both of those bugs. they've been fixed since, but TIO is kind of frozen in the past. so you're not alone in messing it up :p)
@dzaima It used to. However, I wanted to make it explicit since it was otherwise easy to accidentally write code that caused infinite loops when resolving deferred values.
@dzaima I've fixed this one
@dzaima And now I've fixed this one too.
Thank you for all your help, without it there would have been much more bugs. :-)
Hi folks, I'm trying to get around "big data" problem. I have (for now) 5GB JSON array that I need to preprocess before using it in queries. (In the full problem there are easily around 30x5 of these 5GB files, each of which is a "wave + datapoint" combination, so I'm definitely looking for somehow compressing and preprocessing the data as much as possible.)
I've tried the naive approach of making MAXWS=12G and just trying to
raw←80 ¯1 ⎕MAP 'path.json' json←⎕JSON raw
with hope of perhaps ⎕MPUTing it later in a more efficient data representation, but that JSON parsing step already gives m…
@MartinJaniczek it should work (as long as you're careful to not keep all the loaded objects in RAM at the same time), if you can work out a good way to read the file line-by-line
it's mostly a bitset inside each of the objects, so I'm hopeful that holding a vector of list keys and then some corresponding bool matrix / list of bool vectors would lend savings
Right now I'm solving how to (outside APL) split the 5GB JSON array into "object-per-line" file. Then awk '{ print length }' file.txt works pretty well it seems, and that could give me a way to feed offsets to ⎕MAP since I can't do 80 ¯1 ¯1⎕MAP to read a vector of lines
@MartinJaniczek i replicated that to 52 lines for ~5MB. do you have an apl program that parses that to a tree? i'd like to compare performance against my k.
I have a question about syntax: If you have an Agh train and another function f, and you want (Agh)f (apply f, then apply Agh), is there a simpler syntax than using parentheses around Agh or doing Ag(hf)? E.g.(6∧somefunction)(0,∘⌽∘⍉⍣16⊢)
I have {({6::0 ⋄ ⍺.data.core.q1_2015⍎⍵})¨bitfields}¨jsondata, trying to return 0 for a single element computation if the hydrant expresson gives VALUE ERROR. I'm not getting VALUE ERROR anymore but I'm also only getting 0s out (there should have been some 1s, ie. results of the hydrant expression). Am I applying the error guard in a wrong way?
I'm measuring this is.gd/mjbatch from the "outside" (time dyalog -script parse_batch.dyalog) for a 5MB (52 lines) file too and getting 3.6s runtime. (Is there a way to measure a single run of a multi-line / diamond-containing program? I was thinking about ⎕PROFILE but it probably skews the results)
yeah the time and -script approach has no problems with multiline files... I was thinking I could get rid of some initialization costs by measuring inside a session though
Anyways now I'm thinking about trying to do the same thing with the 'M' variant. Here we go again, depth vectors :D
is there a way to destructure this in one go? n←fst[;1] v←fst[;2] t←fst[;3]
I have an issue where I have n objects with sparse fields, ie. one might have a,b,c, another b,d, another b,c,d,e and so on (each field value being a bool). I've found a superset of all the fields names, but now when trying to make a bool vector for each object (either the 0/1 value if found, or 0 if not present at all) I realized this approach will do a bunch of searches. Is there a way to go about it so that I can somehow reuse the bool vectors in the objects? Some kind of grading?
:56487904 The ⎕JSON⍠'M' approach gives me a list of field names and a corresponding list of values. Those are obviously sorted wrt each other, but not wrt the union of fields across all objects
I wonder whether I can somehow think about this as a "set membership" problem. Each object would be reduced to the fields that were 1 (the vector of values could be discarded after that)... and then ... something something outer product with the union of all fields?
the non-matrix approach did this with {6::0 ⋄ ⍺⍎⍵} but I think I should somehow use the fact that I already have semi-prepared columns in the matrix, instead of searching for each field from the union from scratch
Ha! ↑{bitfields ∊ ⍵}¨ {((⍵[;2] ≡¨ 1) ∧ ⍵[;1]∊bitfields) / ⍵[;1]}¨ jsondata No idea if it's correct (there are 1s inside though, so there's hope) but finally something at least has the right shape I'd expect :)
