I have a question for people fluent with array programming languages, so I have a very strightforward problem that I need to fix. In ecommerce platform (something like Amazon) we got some duplicate product offers, it is illegal acording to platform rules.
I can export all offers to CSV file, and than I have columns, productID, EAN, SKU, catID, UnitsSold. I want to remove duplicated offers, so first I sort them by units sold so I won't remove best selling offers. I create data structure in Golang (struct) then I do double loop so I extract unique elements etc
Then I sort it by accounts and export 4 files with id of offers to remove
it took me around 150 lines of Golang, and I already did 50% of this with 4 lines of J
So my question is, is this very strightforward problem something that you can sit down and just write from your head if you are fluent with APL ?
Obviously there are some thinking involved etc. But this it is 1) remove quotes for all elements 2) sort them by column units sold 3) split them by account into 4 tables 4) find uniques from each table on columns EAN and Category ID 5) extract only offer IDs column 6) save thoose to 4 seperate files
For me it took more time to write thoose 4 lines in J and solve 50% than to write 150 lines of Golang and solve it completely
but I am extremely noob and everything still takes a long time for me. I am maybe 5% as productive in J as in Golang right now
Still, I have never experienced processing very large datasets, so I would start with small example that can fitting into the screen, and start experiment it with the REPL
Then make the solution scalable to the suitable problem size
so, Dyalog has system function for processing CSV files, so the 1) can be solved fairly easy, 2) is just a grade 3) should be easy to be done even interactively 4) unique mask 5) select, which is a primitive 6) do some I/O, and that should be done
in golang I also first edited .csv file with text editor to a very small problem that fits on screen, that used tons of print statements to debug and see each step
@alwalo Unlike APL, the rank concept in J could be considered more confusing. And the tacit programming style promoted in J isn't easy if you are not familiar with the lexical rules, binding strength, trains stuff.
@Adám I think I found one, it is related to calling ∇ with the error-guard, I'm working to minimize it.
ExecuteEach←{
BBA←'BBA' ⍝ Test
{0::0 ⋄ r←⍎⍵ ⋄ ~(⊃⎕NC'r')∊3 4}⍵:1 ⍝ test ⍵ to be an functional expression
⊂∇'{×⍵}1' ⍝ Enclose is needed to prevent tail recursion.
}
run with something like ExecuteEach '[', and the local variables will leak :P
everything is inside quotes, I would first remove them
duplicates are for a given account_name, (sku is the same and category is the same) or (ean is the same and category is the same) or (sku is the same ean is the same and category is the same)
and output of this program should be 4 files, one for -Kylo-, one for electricmax, one for progadzety, one for kylopremium. They should only contain string of auction_id that need to be removed for each account
for example progadzety.txt, content:
5906749154750, 5860900000006, 5901130068364, 090159318453 ... and so on
so one of the first step would be to split data into 4 seperate tables on "account_name" column
