@halirutan wow, way to go!

@Kuba I've been hooked by the immediate distributability of web apps (currently I'm working through re-writing something that I found so it's React-based). I keep trying to think of ways to use this with Mathematica or something but nothing really comes to mind.

I was considering trying to come up with some "standard Mathematica-esque web components" library so that I can attach a set of hooks in my Markdown->XML processor that'd call into this, but what would that look like?

I guess I could write a static form of notebooks that gives very light-weight cloud-notebook-esque support with stuff like CellOpeners or cleaner Copy as Code support.

Or rather than going straight down to Markdown I could finally take my Notebook->Markdown infrastructure and make it so that the bindings can be customized to work with an arbitrary set of functions (i.e. let someone define a grammar based on CellStyle and I'll let the code just use these registered definitions).

That way I could directly map Cell[..., "Input"] to like <NotebookCell style="Input"/ options="..."> that my React code would then be able to process.

This'd be in some ways a partial rewrite of the current CloudNotebook stuff but way less sophisticated (and hence likely faster and good enough for 90% of use cases, the same way Markdown is).

I suppose I'd want to document all that though...and I really don't want to have to write documentation.

Eh I'll just get a prototype up and running this weekend to see what people make of it. The same kind of walk-down-the-cell-tree-and-export-recursively structure I'd be doing for this can obviously be reused in all sorts of places so at a minimum that part of the infrastructure would be useful if it existed (in better form than the buggy old Transmogrify package) and if I were to write docs for it.

Unless you want to convert it back, then you need to be more careful about properties.

@b3m2a1 I think this is basically exactly what the cloud notebooks currently do

What is the "best" way to normalize a Dataset column by its sum, or by some other normalization function?

Example:

dataset = Dataset[{
    <|"a" -> 1, "b" -> "x", "c" -> {1}|>,
    <|"a" -> 2, "b" -> "y", "c" -> {2, 3}|>,
    <|"a" -> 3, "b" -> "z", "c" -> {3}|>,
    <|"a" -> 4, "b" -> "x", "c" -> {4, 5}|>,
    <|"a" -> 5, "b" -> "y", "c" -> {5, 6, 7}|>,
    <|"a" -> 6, "b" -> "z", "c" -> {}|>}];

This is one way:

c = N@dataset[Norm, "a"];

dataset[All, MapAt[#/c &, "a"]]

Is there a one-liner that is actually readable?

What is a good way to compute a new value from the existing columns, and add it as a new column?

Best effort so far: dataset[All, Append[#, "d" -> Total[#c]/#a] &]

Suppose that I have already computed something like dataset2 = dataset[All, Total[#c]/#a &]. What is a good way to add this pre-computed dataset to the original dataset as a new column?

Funny that there's no yaml support in WL at all

Someone wrote a package, I think

I think I remember a question

@Szabolcs Would just be a small wrapper around the JSON import, I'd hope

thanks, I'll go and search

Isn't YAML theoretically more flexible (capable of representing more things) than JSON? I am not sure.

What do you think about this question?

There's an example dataset in the question.

Ah, mathematica.stackexchange.com/questions/43297/…

OP is basically asking, why does GroupBy[ds, "Name"]["Heikkinen, Miss. Laina", "Age"] not work?

GroupBy[ds, "Name"]["Heikkinen, Miss. Laina", First, "Age"] does work. It turns out that there is an extra level.

But notice that GroupBy[ds, "Name"][All, First] and GroupBy[ds, "Name"] render exactly in the same way.

Would you consider this a bug? It is certainly confusing.

@Szabolcs I think it's confusing, but not a bug. There's definitely a comment in the docs about the different levels of the association somewhere... My usual method is to Transpose after a GroupBy so that the levels are what I expect (I added a comment)

mmm "The special descending operator GroupBy[spec] will introduce a new association at the level at which it appears and can be inserted or removed from an existing query without affecting subsequent operators."

I suppose they should add something to the display so you know what level is what

@CarlLange Yes exactly.

@Szabolcs I will use my newfound power as the most frequent commenter on the SW livestreams to make this happen 😅

Do you watch most livestreams?

I simply do not have time.

Was there anything graph related discussed recently?

@Szabolcs Yes, most of them, although I've been very busy in the last few weeks so I haven't caught up on a lot of them.

@CarlLange If you use Dataset regularly (I don't) can you comment on my questions above?

Here's one more:

And I find that if you don't catch them live, they're less fun to watch 🤷‍♀️

@Szabolcs Yeah, I checked this out, I don't know that there's a more readable method than the one you have

I have a dataset with column names but no row names. I want to add an identity to each row so I can refer to it unambiguously even after filtering the dataset (e.g. with Select). What's the simple way to add an index to each row? E.g. a new "ID" column which has consecutive integers.

Seems pretty readable to me, actually :)

I also like the ds[All, <|#, "total"->Total[#col]|>&] trick

Module[{c = 0}, ds[All, <|#, "ID" -> (c += 1)|> &]] I suppose, I don't know that there's a nice autoincrement

Join[
 Dataset@Transpose[<|"ID" -> Range@Length[dataset]|>,
   AllowedHeads -> All],
 dataset,
 2
 ]

Do you find yourself doing a lot of Transpose with AllowedHeads -> All?

@Szabolcs No, don't think I've ever done it actually

I keep doing that when working with associations

Dataset doesn't require it

Maybe I should use Dataset

Join[
 Transpose@Dataset[<|"ID" -> Range@Length[dataset]|>],
 dataset,
 2
 ]

@Szabolcs Pros and cons, I think Dataset is nice if you're only using the "critical path" that they clearly expect you to use

But then downside is that when you want to do something more involved you end up writing Normal a lot

@CarlLange Yes exactly.

the "id" case above is a nice use-case for generators

@Szabolcs I have to say I use it entirely because I prefer the display method and it is handy when I want to do easy stuff quickly

There were some really weird things like f/*g working fine but g@*f not working at all. Now both seem to work. Are you aware of any cases when they work differently?

+1 for the display method

I have to re-learn the ascending/descending stuff

Nope, I actually don't have a huge knowledge of those operators, I use /* pretty often with dataset but I don't have a real idea of what it's doing. It's funny, it's one of those things where I've learned what it does a dozen times but just can't get it to stick in my head

Same with currying

Another thing that bothered me was that the missing data handling (which seems to be only possible through Dataset) was not that great, not as good as in other systems with "dataframe" objects.

There's a question today exactly about that: mathematica.stackexchange.com/q/207927/12

Yes! This is still annoying

They had one or two streams about how to handle missing data but I don't think any of it's landed yet

I think the issue is taking the Mean column by column or taking the Mean of a list of {a,b} values. It does not handle the latter case.

MissingDataMethod

Would be nice if it had more wide usage in the codebase

Do you have the prerelease?

@Szabolcs Yes, although I don't use it as much as I should

Busy few weeks and I don't use MMA at work

Do you have access to Redmine? You can comment on #292 if you have anything to add (this is about formatting of the dataset with GroupBy)

@Szabolcs Ah, I think so, but I haven't actually used it yet. Thanks for the reminder, I'll try and get to it this evening

Actually meant to say that I'm in Munich for the next few days if anybody happens to be in Munich and would like a coffee

(Standing invitation in case anyone's ever in Dublin, but that seems less likely than Munich 😅)

It'd be a 6 hrs train ride, so no, but let me know if you come to Dresden :)

Next time maybe :)

@Szabolcs dataset[All, {"a" -> ( #/dataset[Norm, "a"] &)}] maybe? That's what I have used in the past.

@chuy That recomputes the norm as many times as the number of rows. Of course, if it doesn't take any noticeable time to run, that is perfectly fine for interactive use.

@Szabolcs I have used: dataset[All, <|#, "d" -> (Total[#c]/#a)|> &].

Thanks for the feedback.

@Szabolcs Yes, maybe something like dataset[All, {"a" -> ( #/Once[dataset[Norm, "a"]] &)}] then?

Very creative use of Once!

Maybe a bit dangerous as dataset may change.

very true!

probably best to keep that in a Module

I am sure this was discussed somewhere ... what exactly accounts for the difference between ds1 and ds2 below?

ds1 = Transpose@Dataset[<|"a" -> Range[5]|>]
ds2 = Dataset@Normal@Dataset[ds]

That should have been ds2 = Dataset@Normal[ds1]

The performance is still not quite there ...

In[54]:= SeedRandom[123];
arr = RandomReal[1, 10000000];

In[56]:= ds = Dataset@Association["val" -> arr]; // AbsoluteTiming
Out[56]= {0.034449, Null}

In[57]:= tds = Transpose[ds]; // AbsoluteTiming
Out[57]= {12.929, Null}

In[58]:= Total[tds] // AbsoluteTiming
Out[58]= {2.63925, <|"val" -> 4.99829*10^6|>}

In[59]:= Total[arr] // AbsoluteTiming
Out[59]= {0.004574, 4.99829*10^6}

In[60]:= tds2 = Dataset@Normal[tds]; // AbsoluteTiming
Out[60]= {5.25286, Null}

In[61]:= Total[tds2] // AbsoluteTiming

Are Dataset columns of real or integer type still not stored as packed arrays?

I know that Dataset has some knowledge of the types it contains, but do we already have any control over this?

Or is it all behind-the-scenes, supposed to be transparent to the user stuff?

@chuy @Szabolcs also dataset[All, {"a" -> Curry[Divide][dataset[Norm, "a"]]}]

@Szabolcs The datasets have different types -- try evaluating Dataset`GetType /@ {ds1, ds2}. Dataset type is sensitive to the history of evaluations that created the dataset. See for example (146177).

@Kuba lol originally I was gonna ask what I should do with this newfound knowledge but then I came up with a thing. That was more or less my current plan and then do much more of the options processing and stuff on the React side.

@CarlLange I've been thinking about ways to make this process something that doesn't require any extra processing, too, so you can just say LightweightCloudDeploy[currentNb, fileName, "Location"->"GitHub"] or something and I'll handle all of the necessary work to make it happen. Trying to decide how I should handle things like images in that setup, though.

If I cook images into the file it'll be possible to distribute this "notebook" anywhere

But obviously this could make them bloated

@b3m2a1 better talk to @CarlLange @C.E. and others with more experience with web dev than me :) It is reasonable to export it to an idiomatic html which then can be adapted by any framework that supports custom tags/components.

Otoh maybe different frameworks expect different structure, idk, as I said, ask them ;)

@Szabolcs When are you coming to Leipzig? I owe you a beer for all your help with paclet documentation

@Szabolcs I've never been able to get them packed, if you figure something out please let me know. I can get a CSV imported and converted to a packed array that takes up 25MB, but the same file as a Dataset is 550MB.

@ThunderBiggi ???

Not sure what you mean. In general, no you can't return a ragged array.

But you can Map the CompiledFunction over a List of matrices

@ThunderBiggi No, "officially" you can't return ragged arrays from a compiled function. You also can't use "symbolic" matrices as input since only basic datatypes are supported. But please ask a question on main with a small example so that we have the whole picture. There are many different ways to work with compiled functions and there might be a solution to your problem.

@ThunderBiggi Have you seen PadRight?

@ThunderBiggi that doesn't return a ragged array?

It is 2 by 3 by 3 - that is not regualr is it? I thought that regualr arrays have to be square

@ThunderBiggi "Ragged" means something different.

{{1}, {1,2}, {1,2,3}}

This is ragged. You cannot give the dimensions of this matrix like you can in your case.

(Btw, the N in your code is useless. You are already operating on Real numbers)

@ThunderBiggi No, you don't even need square arrays as input. What you define in {b, _Real, 2} is the "Rank of the tensor". So 1 is a vector, 2 is a matrix, 3 is a 3d matrix, etc..

However, even this here

{{0, 3, 8, 4, 5}, {5, 0, 3, 1, 2}}