« first day (1505 days earlier)      last day (1158 days later) » 
00:00 - 16:0016:00 - 20:00

12:37 AM
hmm is there a way to somehow use ⍨ to get rid of the parens in {(expr)@⊢⍵}
 
That's... take a boolean array and replace ones with the values in the expr?
 
oh
lmao, oops
oh, no, I can't do it with ×, yes it is that
 
1:08 AM
@ is somewhat weird in that it is non-trivial (to say the least) to interchange array operands for function operands or vice versa (for both sides).
 
did you answer your latest problem in APL? I have a 62 but I reckon it can be a lot lower
 
Didn't seriously try yet, but I can imagine some combination of 2</, 2>/, , ⍤1
Also note that you can take a boolean matrix or its transpose
 
ya I'm taking it as a 5x32 boolean matrix
 
The input is already guaranteed to be a Gray code. I guess you don't need the last condition
 
oh
oops, I totally missed that, thanks
was surprised there wasn't something for this sort of thing in dfns, although obviously a non dfns solution is more interesting
 
1:34 AM
And you can save 3 by eliminating the variable a: {(∊⍸¨↓⍉b)≡(∨⌿b←2>/⍵)/¯1↓⊣/⍒⍤1⍉(∊+\¨¨⊆⍨¨↓⍵)@⊢⍵}
 
oh yea, thanks
ffs, 13 byte jelly answer, wonder if it can be translated
oh, oops, I don't check if it is cyclic
crap
@Bubbler can you think of any cyclic test cases where my code fails?
ok, found one, 8 3 ⍴ ⍎¨'000001011010110100101111'
I think this would be a good one to add
 
2:03 AM
@Bubbler what does this mean?
@Bubbler and this?
 
@rak1507 And the input is guaranteed to be cyclic too
 
oh is it
 
@nathanrogers That's about this primitive operator
 
I know what @ is
I wonder what you mean about it being non-trivial to replace the operands with functions
 
2:08 AM
For the right operand, an array operand is the indices to modify, but a function operand should return a boolean mask instead
 
Yes, so what makes that non-trivial?
 
(which is quite nice because replacing all 1s in a boolean array becomes @⊢)
 
do you simply mean coercing an index look up to a boolean expression?
or index generation to a bitmask?
 
I think just that it's inconsistent, it either deals with a mask or with indices, so it can be annoying switching (correct me if I'm wrong bubbler)
 
An array operand can use three different indexing modes, but the boolean mask returned by a function can specify only the individual top-level items (no cells, no deeply nested stuff)
 
2:12 AM
i mean, a bitmask can be coerced to indices with ⍸ so I don't see the problem
@Bubbler I see what you mean
but can you provide deeply nested indices?
 
The doc says that @ supports deep indexing. Didn't actually try it out
 
neat, I never knew you could deep index
wonder if it works with squad
 
⋄ 9@(⊂1 3 2)⊢((1 2(3 4)5)(6 7 8))
 
or just pick
that's neat
 
idk if it works with squad but you can do 1 3 2⌷↑⍣≡a instead of 1 3 2⊃a but obviously you'd be insane to actually do that
 
2:17 AM
 
PRetty sure I've used that before in a bounty
 
@nathanrogers I don't think it works with squad, but it definitely works with bracket indexing
 
yay, because who doesn't need 3 different ways to index something
 
@rak1507 anything else is invalid
 
And another fun golf: (∊+\¨¨⊆⍨¨↓⍵)@⊢⍵ → ⊥⍨¨,\⍵
@rak1507 ^
 
2:30 AM
that is genius
I knew there was a better way to do that
damn
I feel bad putting that into my solution because that's so good
 
It's from my own answer, but it took some time to realize it works on matrices too
 
ah I knew I should have searched for that
damn, I was pleased with that ⊆⍨ as well! :(
time to update my answer for the billionth time
 
Wait, I guess it's possible to remove those s by taking the input transposed
 
I don't think it'd save any because you'd have to still figure out how to get ⊥⍨ to work for example
 
⊥⍨¨ still works since it is independent of the outer structure
You just change \ to
 
2:36 AM
true, but ¯1↓⊣/⍒⍤1 wouldn't
mostly just the ⍒⍤1 bit
 
You just remove the before that
 
I think {(∊⍸¨↓b)≡(∨/b←2>⌿⍵)/¯1↓⊣/⍒⍤1⊥⍨¨,⍀⍵} works
 
oh yea
it probably does
no, something breaks
wait, it does work
35 chars!!!
 
 
3 hours later…
5:44 AM
@rak1507 @Bubbler {⍵⊣@⊢⍨expr}
@Bubbler Yes, imo @ has unfortunate design. I wasn't quick enough to react when it was described. It should always have taken indices, even from a function right-operand, as it is trivial to convert from mask to indices (), but not vice versa. The elegant @⊢ would just be @⍸ too.
 
6:03 AM
@Adám I don't think it saves any bytes, though that's an interesting way to write it down
 
@Bubbler He asked about using to remove parens, not about saving bytes.
 
Oh.
 
 
1 hour later…
7:21 AM
@rak1507 isn't that just \?
 
 
2 hours later…
9:21 AM
CMQ: If was added as {(⍺⍺ ⍺)⍵⍵ ⍵} wouldn't it also make sense to "overload" by allowing up to one array operand, similarly to , but as opposed to , it would ignore the argument on the curried side? Thus 10(+⍛1)20 would be 11 and 10(1⍛+)20 would be 21. I think this would make many trains neater.
 
@Adám Why not extend the domain of existing ?
Currently the result of f∘x or x∘f is strictly monadic, so it shouldn't interfere with existing functionality (unless some existing code depends on the erroring behavior)
 
@Bubbler Oh right.
Well, I currently use presence of f∘A or A∘f to indicate that a train is monadic. That hit would go away.
 
@Adám Again a reason to have a monad/dyad built-in.
(A nice combo would be introducing a built-in that always errors. The sole use of J's [: is for that combo (to make the valence explicit), except for the capped fork thingy.)
 
9:44 AM
Anyone care to pitch me some dfns that I can practice my train making on? Thanks, trainspotter @ Bristol.
 
@Bubbler Currently (and maybe even with an extended ) ⍬∘⍬ always errors.
@xpqz Easy or hard?
 
Moderate to hardish, perhaps?
Not co-dfns level.
 
Starts digging into 100-byte-long code golf answers of mine
 
@xpqz Maybe search for no-assignment dfns on APLcart?
 
9:50 AM
@Bubbler Ah, so the dfn one, {+/{∧/∊2≤|(⊢-¯3↓¨,\)+\0 1,×\0j1*⍵}¨∪{(⊃∘⍋⊃⊢)(⊢,⌽¨)⍵(-⍵)}¨2-,⍳3⍴⍨0⌈⍵-2}
 
Yeah.
 
(⊃∘⍋⊃⊢)(⊢,⌽¨)⍵(-⍵) is going to hurt.
 
It's one of the longest assignment-free dfn I could find out of my answers.
 
@Bubbler hm, if that's your definition of moderate, I dread to think what you'd consider hard.
 
It's already partially trainified, so it might not be super great for "train making" per se
 
9:54 AM
Actually, you can assign in a tacit function, but the assigned value is a function which you can then reuse later (to the left).
 
I found this in aplcart: {⍵⌷⍨⊂⌽⍒+\⍺} -- although I can't really see what it's supposed to do, trying it out.
 
@Bubbler Not hard to define, though: _MD_←{×⎕NC'⍺':⍺ ⍵⍵ ⍵ ⋄ ⍺⍺ ⍵}
@xpqz Avoid attempting to translate functions that use an operator with explicit mention of an argument in the operand (e.g. ⍣⍵). Such cannot be translated directly to tacit, though other approaches may exist.)
 
Ok. Out of curiosity, that dfn (Reversal (⊖) of subvectors of Y indicated by Av) -- how is it supposed to be used? It only seems to pick the items at ⍺?
 
@xpqz This:
      1 0 0 0 1 0 0{⍵⌷⍨⊂⌽⍒+\⍺}'ABCDXYZ'
DCBAZYX
      1 0 0 0 1 0 0{∊⌽¨⍺⊂⍵}'ABCDXYZ'
DCBAZYX
Such techniques were common before nested arrays, and they will still give you optimal performance because your array stays flat.
 
This can be made tacit too, if you analyze it carefully: {a b c←⍵-1 0⍺⋄⍎dab⍕c↓a↓b~⍨⍳⍺} (dab is a monadic function from dfns; source)
 
10:09 AM
@Adám hmmmm.... might steal as example for webinar
 
@Adám Might be worth cmpx-ing
We know that keeping flat does not always win
 
@Bubbler But here:
      mask←1,1↓1=?1e4⍴10 ⋄ data←⎕A[?1e4⍴26] ⋄ nested←mask⊂data
      Flat←{⍵⌷⍨⊂⌽⍒+\⍺} ⋄ Nest←{∊⌽¨⍺⊂⍵}
      ]runtime -c "mask Flat data" "mask Nest data" "⌽¨nested"

  mask Flat data → 3.6E¯5 |    0% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕
  mask Nest data → 1.4E¯4 | +273% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕
* ⌽¨nested       → 7.2E¯5 |  +96% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕
 
So "reverse each" is the bottleneck I guess?
 
@Adám Aha, I see. I read 'subvectors' to mean that the right arg was nested. In aplcart's notation, is there a way to indicate that a vector is boolean? It wasn't obvious to me that left side should be. I think that one would benefit from a TIO link.
 
@Bubbler Yes. It has to chase a thousand pointers.
@xpqz The left argument is given as Av which the legend (just below the query field) explains as Boolean vector.
 
10:24 AM
ah, yes, it would help if I could read :./
 
10:36 AM
@RikedyP Interesting. I didn't think of +\boolean as a "boolean scan", but sure, that is a useful interpretation.
 
@Adám Yes I will be explaining at one point that literature about "boolean functions" defines them as returning boolean results for boolean input, but I will stretch the definition for the sake of showing useful constructs
 
RGS
For people used to using RIDE: I thought RIDE was supposed to open multiple tabs when you have the execution interrupted in several different places?
 
@adam aplcart.info/?q=position%20of%20comments# this doesn't seem to do what it says but I will fix and PR
 
RGS
e.g. right now ≢⎕SI is 3 and I thought I would get 3 tabs I could cycle through..?
 
@RikedyP Will you include pair-wise Boolean reductions?
 
10:40 AM
@Adám Only some, but yeah
 
RGS
(Same thing happens when I enable floating windows and make sure that "single floating window" is disabled.)
 
Hopefully finish writing this morning, @rgs @adam you maybe do a run through this afternoon or tomorrow?
 
RGS
@RikedyP sure; what are you writing? :P
 
@RGS Yeah I'm not seeing that behaviour (can't remember if it's supposed to do that or not, but sounds useful)
@RGS webinar for next week
Also quite cunningly wrote it as a Jupyter notebook which I'll publish just before
 
RGS
@RikedyP clever.
 
10:42 AM
@RikedyP Sure, want to decide a time? I have a meeting at 2 pm (also, we should probably take this to internal chat).
 
@RGS if you had 3 tabs open at the same time, what would happen to the rest of them when you go forward in a higher one? Which tab would be represented in the REPL?
 
@RGS no, but the earlier stack frames can be accessed by closing the debugger window, and subsequent stack frames by stepping in the debugger.
 
RGS
@dzaima I think I understand your point, but the windows editor allows cycling through the various interrupted functions.
@xpqz Alright, so you are saying that if my stack was interrupted at f → g → h, I can reach f by pressing Esc on h and g and then use the debugger to step in g and h again if I need to?
 
@RGS yeah, i don't see any way to do that in RIDE
 
@RGS yes (if esc closes the debug window).
 
RGS
10:47 AM
@dzaima ah ok, so it is a RIDE "limitation"?
@xpqz For me, it does.
 
@RGS oh, maybe view→show debug
 
I've not (yet) discovered a way to jump directly to an arbitrary stack frame. Although not really needed it so far.
 
@dzaima no, doesn't seem like clicking the items changes the scope
 
RGS
@dzaima Yeah, I think that is just a dynamic visualisation of )si and whatever command tells you the threads that are running.
 
@xpqz You mean up N levels?
 
RGS
10:49 AM
But I hadn't noticed it before, so thanks for that.
 
@Adám yes. Or down, depending on how your stacks build in your mental model.
 
RGS
@xpqz Obviously diagonally, from lower left to upper right.
 
hahha :)
 
Ha. In my defence -- I've not looked for it, as not needed it, but also -- even reading that now I'd never relate it to using the debugger.
Using, say, the Python debugger in VS Code, you get a nice little window showing the stack, and you can click on any stack frame and jump directly to that state. I assume that's what the Dyalog-win-IDE does, from what @RGS suggested.
 
11:05 AM
Is there a way of making an array of n new namespaces, is this the best way, or can I somehow avoid the eaching? {⎕NS⍬}¨⍳n
 
@xpqz ⎕NS¨5⍴⊂⍬ might be marginally better but otherwise i don't think you should be able to avoid an ¨
 
Ok, that looks like a nicer pattern.
 
@xpqz Actually, this may not work. I think it'll cut individual suspensions, not move up the stack. You may need to execute →0 for each step up. That certainly works.
 
11:22 AM
More from my deepdive into namespaces and json: this, whilst achingly elegant, is also too inefficient for any practical use unless I'm being stupid (not unheard of): docs.(≢nodes.⎕NL¯9) -- docs here an array of 207 namespaces, each containing a namespace nodes of 3, 6 or 12 namespaces. The above takes me nearly 15 seconds to run.
 
seems fast to me; maybe the gc is acting up?
 
11:41 AM
@dzaima (ref; but shouldn't be an issue as docs appears to be a variable)
 
@dzaima the docs namespaces are rather big, and after that call has completed, the interpreter is unresponsive for a while.
 
how slow is just a.nodes?
 
Instantaneous.
 
huh. docs.nodes.(≢⎕NL¯9)? ≢¨docs.nodes.⎕NL¯9?
 
First one 'hangs' -- still waiting.
 
11:50 AM
no not instantaneous. that's all that matters
 
Done. Maybe 15-20 secs (I'm in the debugger, so can't ]cmpx)
Sameish for the second one.
 
@xpqz (really?)
 
+/a.nodes.(+/≢¨⎕nl-⍳10) might be a useful metric
 
Second one still running, now close to 90s
Done.
 
Yikes. You should report this. Unacceptable imo.
 
11:53 AM
+/docs.nodes.(≢∊⎕nl-⍳10) ---> 4104
 
huh, so the size of names can't be the problem. what about the speed of just docs.nodes.⎕nl?
 
...still waiting
 
oh wow
and you're sure ≢docs.nodes is just 207?
 
So I wonder if this is an 18.1 pre-release issue, or if my ws is memory starved
 
@xpqz Would it be possible for you to send (an anonymised version of) your data for us to experiment with?
@xpqz Can you try dzaima's example? It is instant for me in 18.1 too.
 
11:57 AM
(still waiting)
 
Now I wonder if it is a mac thing.
 
Hmm. Dyalog's eating 100% CPU.
 
On dzaima's example?
 
what about the boring choice of {≢⍵.nodes.⎕NL¯9}¨docs?
 
No, still on docs.nodes.⎕nl
 
12:00 PM
@xpqz OK, but that has to format a gigantic nested array of names.
 
May have to rude-kill this.
 
Hard interrupt doesn't cut it?
 
i wonder if the runtime of these expressions just goes up unconditionally the more it's called, or if we just happened to ask them in increasing difficulty
 
@dzaima Good point. It may well be that the GC is failing and the workspace is getting full of lost refs.
 
Ok, kill -9 it. Will try again from afresh.
@Adám it's actually real data I probably shouldn't share, but I can look at cleaning it somehow maybe.
 
12:03 PM
@xpqz Well, try first with dzaima's sample.
 
oh yeah, you could test if my replication creates the same issue
 
@xpqz If the source is json, just replacing all letters with random letters and all digits with random digits should do.
 
@Adám no, hard interrupt rarely interrupts in RIDE, as it only interrupts the interpreter, thus not stopping huge formatting things as probably was the case here.
Ok, one moment.
 
@xpqz printing the 207 #.[Namespace]│.│⎕NLs is instantaneous for me on RIDE/linux, so that shouldn't be a problem
 
@dzaima which example did you want me to run?
 
12:07 PM
@xpqz this
 
Yep, that's quick:
      ]runtime a.(≢nodes.⎕NL¯9)

* Benchmarking "a.(≢nodes.⎕NL¯9)"
             (ms)
 CPU (avg):     1
 Elapsed:       2
Is there a way of measuring the byte size of a ns?
⎕SIZE dont' seem to do it
 
@xpqz compare ⎕wa before and after creating it i guess
 
@xpqz Aha, then there's something with your specific data. If the source is json, just replacing all letters with random letters and all digits with random digits should do it.
 
@Adám the source is json. Do you have a oneliner to do such a thing?
The data for sure is convoluted, big and gnarly.
 
@xpqz '\w'⎕R{⎕A[?26]}'\d'⎕R{⎕D[?10]}
 
12:11 PM
@Adám well, that'll replace the nodes we want here. Not hard to fix though
 
Ah yes.
 
In json terms only want to change the "values" not "keys"
 
@xpqz Does it matter, other than for nodes?
'nodes' '\w'⎕R{⍵.PatternNum:⎕A[?26]⋄⍵.Match}'\d'⎕R{⎕D[?10]}
 
Ok, let me see what that results in
Interesting. So first I do this:
docs←(hc.GetJSON 'POST' url params hd).Data.rows.doc
that's the fetch, and that's quick (half a second?)
Then I tried turning it back to json:
Not completing.
j←1⎕JSON docs
That -- hangs
 
Interesting. I'd report that too.
@xpqz How did that go?
 
12:19 PM
Still running
 
how big is the docs string?
 
Not a string.
 
Yes that ^^
@xpqz What is it then?
 
Array of namespaces from GetJSON
 
@xpqz oh, i misunderstood, it's originally an object, not a string
 
12:20 PM
Yeah
I think that's killed my interpreter.
 
@dzaima No, it is a string from the internet, but HttpCommand.GetJSON converts it to namespaces.
 
Interesting -- converting FROM json (by GetJSON) is quick. Converting TO json dies.
Is there a round trip issue here.
 
Can you fetch the raw JSON and play with that?
 
I'd expect ⎕JSON to let me know if it was.
 
@xpqz i'm guessing there's just an issue with namespaces in general
 
12:22 PM
@Adám Sure. Let me fetch the raw data with curl to a file.
 
@xpqz how much ⎕wa do you have after that?
 
@xpqz Or use HttpCommand :-)
 
Stefans-MacBook-Pro:~ stefan$ ls -las data.json
4224 -rw-r--r--  1 stefan  staff  1929163 10 Mar 12:23 data.json
 
2MB seems reasonable to be handleable
 
Yeah, agreed.
 
12:24 PM
I needed to strong interrupt the 1⎕json command.
 
can you try running the data through an anonymiser and then uploading it?
 
Let me read the file and try Adam's regexer.
      ⎕wa
2144196488
Not sure what that is.
 
workspace available
 
@xpqz 2GB free memory. Plenty
 
@Adám your anonymiser makes illegal json says jq:
Stefans-MacBook-Pro:~ stefan$ cat anon.json | jq .
parse error: Invalid numeric literal at line 1, column 18
 
12:35 PM
@xpqz ⎕D[?10]⎕D[1+?9] probably
 
Yeah.
Try 'nodes' '[^Ee\W]'⎕R{⍵.PatternNum:⎕A[?26]⋄⍵.Match}'[1-9]'⎕R{⎕D[?10]}
A more fun edition would be to replace vowels and consonants separately which should leave words mostly readable.
 
wouldn't that be useless for anonymising though
 
I don't think so:
      '[aeiou]' '\w'⎕R{⍵.PatternNum:(⎕C⎕A~'AEIOU')[?21]⋄'aeiou'[?5]}'wouldn''t that be useless for anonymising though'
viehgj'y vwaj po axosobt riz usujcpiralx wyiusp
 
fair enough
 
@Adám doesn't look any more readable except for those matching the vowels and consonants to a dictionary
 
12:45 PM
True.
 
Interesting. Dyalog's ⎕JSON dislikes the data.
Parses fine in jq
 
@xpqz Even more interesting to get it now.
 
 
@xpqz Oh, remove the 1 from ⎕nget
 
:)
Ok, that worked
And was super quick
 
12:49 PM
Or alternatively, you can use 0⎕JSON∊f
 
Your anonymizer still leaves things like: {"aaa":Y} -- unquoted letters here and there.
 
@xpqz That's strange. I wonder how that can happen.
 
It turns some digits to letters.
 
Yes, because of \w
 
Here's the original:
{
  "total_rows": 207,
  "offset": 0,
  "rows": [
    {
      "id": "bm-cc-au-syd-01",
      "key": "bm-cc-au-syd-01",
      "value": 1,
after:
Stefans-MacBook-Pro:~ stefan$ head -c 100 anon2.json
{"OWPLIKFBVK":RUN,"HDOReX":Y,"OVUK":[
{"MA":"NU-CS-OA-EKS-PI","GeG":"PS-ZQ-CN-KYJ-EX","IOKLe":X,"
 
12:56 PM
@xpqz Try 'nodes' '[^a-df-z]'⎕R{⍵.PatternNum:⎕A[?26]⋄⍵.Match}⍠1'[1-9]'⎕R{⎕D[?10]}
Sorry f not e. we skip e for 1.2e3
 
@Adám at that point probably better to just replace "([^"]|\.)*"
 
ff←'nodes' '[^a-df-z]'⎕R{⍵.PatternNum:⎕A[?26]⋄⍵.Match}⍠1'[1-9]'⎕R{⎕D[?10]}f
DOMAIN ERROR: Invalid regular expression format
      ff←'nodes' '[^a-df-z]'⎕R{⍵.PatternNum:⎕A[?26] ⋄ ⍵.Match}⍠1 '[1-9]'⎕R{⎕D[?
      10]}f
 
Arrg. It is wrong anyway.
'nodes' '[a-df-z]'⎕R{⍵.PatternNum:⎕A[?26]⋄⍵.Match}⍠1('[1-9]'⎕R{⎕D[?10]})
 
Nope: {"OWPLIKFBVK":RUN,"HDOReX":Y,"OVUK":[
 
What am I doing wrong here?
 
1:03 PM
@xpqz are you sure you're running that on the initial data?
 
@xpqz On this input, I get:
{
  "POGTN_BZJH": 205,
  "RUIDeC": 0,
  "YSNF": [
    {
      "EB": "SE-WT-TG-UWK-03",
      "AeQ": "FN-BV-AK-BCS-00",
      "JSZKe": 9,
 
my version: '"([^"]|\.)*"'⎕R{⍵.Match≡'"nodes"':⍵⋄'"','"',⍨⎕a[?26⍴⍨¯2+≢⍵.Match]}⍠1('[0-9]'⎕R{⎕D[1+?9]})
 
Running it all again:
Stefans-MacBook-Pro:~ stefan$ cat anon4.json
parse error: Invalid numeric literal at line 2, column 157
Stefans-MacBook-Pro:~ stefan$ head -c 200 anon4.json
{"WIJBO_PBRN":904,"NEMIeW":0,"BUAX":[
{"IY":"TQ-MB-UJ-BSE-00","KeQ":"IC-XN-XA-TUD-08","OVIDe":0,"LWT":{"_WS":"YX-ZJ-BE-IZI-02","_BeK":"908125-90DPN87O7METI0U94e08AAeEX0PB3U73","ZEIeQLCKeY":RFSUe,"GKOeStefans-MacBook-Pro:~ stefan$
 
RFSUe
 
jq still moans, although I can't see the error.
 
1:08 PM
> "ZEIeQLCKeY":RFSUe
 
preserving keys: ':\s*"([^"]|\.)*"'⎕R{':"','"',⍨⎕a[?26⍴⍨¯2+≢⍵.Match]}'[0-9]'⎕R{⎕D[1+?9]}
 
@dzaima works!
 
Then you were lucky.
@dzaima Can that not change the valid 123 into the invalid 012?
 
@Adám 1+?9
and i highly doubt zeroes not existing in the data will change much
 
Ok, that's potentially worthy of APL Cart.
 
1:12 PM
@dzaima oh, that'll not anonymize strings in arrays.. (and booleans too)
 
@dzaima Oh, I missed that.
 
I need to have a quick break from here -- I'll be back shortly.
 
Heh.
 
honestly I'd probably just use some external tool rather than doing it in APL
 
@dzaima yes, strings in arrays look untouched, although not too many instances
 
1:17 PM
OK, I think I know how to do it now.
 
'[0-9]'⎕R{⎕D[1+?9]}'"([^"]|\.)*".'⎕R{':'=t←⊃⌽m←⍵.Match:m⋄t,⍨'"','"',⍨⎕a[?26⍴⍨¯3+≢m]}
 
1:27 PM
@xpqz Try:
 Anon←{
     Subst←{
         p←⍵.PatternNum
         0=p:⍵.Match
         1=p:⍕?10*≢⍵.Match
         2=p:⎕A[?26]
         3=p:⎕C ⎕A[?26]
     }
     pats←'"[^"]*":' '\d+' '[A-Z]' '[a-z]'
     pats ⎕R Subst⊢⍵
 }
 
2:21 PM
@dzaima's version works. @Adám I get an error:
DOMAIN ERROR: The selected random number generator does not produce a sufficient
 range of values
Anon[4] 1=p:⍕?10*≢⍵.Match
 
Oh, big numbers there :-)
 
But it's 100% anonymised now. Let me see if I can carve down the issue to the smallest possible reproduction.
Yes, problem persists, and is reproducible. Do you want the data then?
 
Yes please.
 
Let me know when you are done with the link and I'll remove it.
Here are the offending lines:
j←(⎕JSON data).rows.doc
(clusters←⎕NS¨(≢j)⍴⊂⍬).(name tenancy location nodes)←j.(_id tenancy location (≢nodes.⎕NL¯9))
assuming you read the file's contents into data
 
@xpqz Got it.
 
2:34 PM
Ok, will leave it up for a bit if anyone else fancies a go.
 
just ≢doc.nodes is slow for me
 
Not just me then, and not a mac-only or 18.1-pre issue?
 
Hmm, that's weird slow
 
3:00 PM
anything that accesses the contents just is slow. ¯\_(ツ)_/¯
even after new←{⊢⍵}¨doc (which does take ~forever), ]runtime ≢⊃new is 100ms
 
3:12 PM
it is the refc=0 issue!
 
@xpqz tar -xf anon6.json.gz dumps the content into the terminal. How do I unpack it into a file?
 
or something like it at least
@Adám tar … > file.json
 
@dzaima nope, that creates an empty file and dumps the content to the terminal. Maybe I need to redirect a specific stream?
 
separating (⎕json readFile).rows.doc into obj←⎕json readFile ⋄ doc←obj.rows.doc fixes all performance issues
 
It's not a tarball.
gunzip anon6.json.gz
 
3:14 PM
@xpqz So what?
 
@Adám just gunzip anon6.json.gz will create the json file
 
@xpqz that worked.
@dzaima And it also seemed to delete the archive. Odd.
 
@Adám it does that
@dzaima ayy my first guess (or something close enough) was correct (even if i had no clue where it came from)
 
3:30 PM
@dzaima why is this?
 
@xpqz the base JSON object has a reference count of 0 or something (i'm totally just guessing here) and Dyalog wants to GC it away so badly, but can't as a child of it is referenced somewhere
 
We need a
:DoNotGarageCollect
    code
:EndDoNotGarageCollect
 
@Adám or just correctly count references or something
 
@dzaima Well, it is correct.
 
@Adám I'm pretty sure a JSON object which requires its parent to be alive should keep a reference to its parent
 
3:35 PM
@dzaima Ah, good point. Children have refs to their parents. I've got to ask someone about that.
 
@dzaima (no clue why it needs to keep its parent around, but i didn't decide that)
 
@dzaima They are not dictionaries. They need to know, just in case you do ##
 
@Adám oh, that'd mean that namespaces could never be GCed by refcounting as they'd never be able to decrement the other
 
@dzaima so when doing
(clusters←⎕NS¨(≢j)⍴⊂⍬).(name tenancy location nodes)←j.(_id tenancy location (≢nodes.⎕NL¯9))
Trying to move things out, I can't seem to get it to be faster.
Or is it too late already then?
 
@xpqz you need to just add _← or something for the result of ⎕JSON
 
3:46 PM
This is the problem? (⎕JSON ff).rows.doc?
Ah
 
@xpqz yeah. (_←⎕JSON ff).rows.doc
then you need to make sure that _ outlives all other JSON objects you may read from that
(and as a hack, if you want to keep some around for longer, ⎕JSON⍣2 them :D)
 
Hmm, now j←(_←⎕JSON ff).rows.doc hangs instead
 
aw man
 
Unless my ws is already screwed up by this.
 
@xpqz you do need to clear all older objects that don't do it correctly. Otherwise, there shouldn't be permanent damange
 
3:49 PM
Let me try a )clear
 
yeah, that's the easy way out
@dzaima so it's either this issue every now and then, or all namespaces can only ever be unallocated by a full GC. :/
(alternatively, don't GC on like every function call, though that's only a temporary solution and would probably just start quietly slowing things down instead of doing so loudly)
 
@Adám are there any ]link hooks? If I wanted to tell link to execute arbitrary code on initial link, is there a means to do that?
 
@nathanrogers Yes.
 
@dzaima this is now instant, yay:
f←⊃⎕NGET'/Users/stefan/anon6.json'
j←⎕JSON f
docs←j.rows.doc
data←(clusters←⎕NS¨(≢docs)⍴⊂⍬).(name tenancy location nodes)←docs.(_id tenancy location (≢nodes.⎕NL¯9))
Amazing.
 
@nathanrogers See docs.
 
3:57 PM
@Adám what about on initial linking?
 
CMP: What APL glyphs should unicode have included? E.g. "Unicode should have 'left-right tack' for left and right arguments" or "Unicode should have 'intersection del diaresis' because penguins".
 
00:00 - 16:0016:00 - 20:00

« first day (1505 days earlier)      last day (1158 days later) »