@kirma I've wanted to pursue that for a long time... and we may end up using some of the same symbols that GrammarRules uses... but that set of stuff is really about natural language, not artificial languages. things like binary formats require some different things, too, like length-prefixed lists, which i don't quite know how we should represent symbolically.

@kirma but i do have a prototype of such a binary grammar that i used to pull apart .mx files. we may have something experimental in 10.1.1 to play with...

@MichaelHale yes we fixed that. turns out that DumpSave captures attributes of all symbols it dumps, not just the ones in the context you asked to dump. there is now a hidden option to turn off that behavior, which is what we do when creating .mx files for things like the predictive interface.

@TaliesinBeynon, nice to see you here. Something changed in DumpSave?

@TaliesinBeynon do you have news about efficient tabular data in Dataset?

@Murta yeah it's very much going to happen

@Murta only a question of whether for 10.1.1 or 10.2. and there will be some very powerful mechanisms there that users could harness for their own uses.

@Murta basic idea would be new symbols called ColumnArray (or TransposedArray), MissingArray, SubArray, LookupArray. they would 'fake' the ol' list-of-associations paradigm, with efficient, compiled support for Map, Select, and so on.

@Murta you wouldn't see them unless you did InputForm of a Dataset, but you could of course construct them yourself for your own purposes. some examples: MissingArray[arr, indices, missings] 'masks' the inner array at certain positions with certain Missing[...] values. so if you do Total[...], it just ignores those. point being that the inner array can remain packed, even if missings are present.

@Murta or SubArray[arr, indices] would say 'consider only Part[arr, indices]', so that when you do a Select, no memory is copied (which is particularly expensive for column-oriented tables), instead a list of indices that passed the criterion is allocated and stored (packed, of course). again, if you then write Total, it would just skip iterate over the original array to calculate the total, which is massively more efficient.

@Murta ColumnArray[{col1, col2}] or ColumnArray[<|key1 -> col1, key2 -> col2, ..|>] is of course the way to represent a row-oriented array, but using column-oriented storage. it means we can pack many common types of columns as RawArrays, and again things like Map and Select would be 'patched' to do the efficient thing in those cases (which is quite tricky).

when you couple all this with @LeonidShifrin 's streaming framework, we can get what i believe will be quite stellar performance and memory usage... i hope beyond what things like Python and R currently offer.

@hftf Taliesin confirms it's fixed. @TaliesinBeynon The Dataset improvements sound very cool.

@TaliesinBeynon @kirma I just realized custom grammars can only be evaluated on the cloud. That's unexpected. For length-prefixed lists I would want to do something like:

GrammarApply[
 GrammarRules@{FixedOrder[
     a : (b : DigitCharacter :>
         GrammarRules@{c : Repeated[DigitCharacter, {b}] :> Total@c}) ..,
      "END"] :> a}, "2343222END"]

And get {7, 6}

@MichaelHale yes, that could work. though i think it is a bit ugly... for binary formats it is so common i think we should support something more direct

@TaliesinBeynon Sure, but I think people that are used to parser-generators wouldn't mind defining recursive grammars directly.

i suspect that the way grammars are defined and implemented in the cloud will not support the use case you described.

not my group

I'd guess for performance reasons for Wolfram Alpha that the typical flexibility of the rest of the language wasn't a top priority for the grammars. I was just giving an example of how I imagine you would extend it. You could of course break out the sub-grammars to separate definitions for readability.

sequenceSum[length_] :=
 GrammarRules@{a : Repeated[DigitCharacter, {length}] :> Total@a}

GrammarApply[
 GrammarRules@{FixedOrder[a : (b : DigitCharacter :> sequenceSum@b) ..,
      "END"] :> a}, "2343222END"]

@MichaelHale yes that's it. to be efficient, the grammars have to be compiled. but that means that run-time construction of grammars is not possible, or if it is, not efficient

@TaliesinBeynon great news!.. I can't wait to try it! I believe life would much better for handler SQL Tabular data in Mathematica.

@halirutan We are planning on overhauling the JSON import/export framework for 10.1.1. At the very least associations will be fully supported as the form corresponding to JSON objects, subsuming lists-of-rules, which are much harder to work with. There will also hopefully be ways of easily exporting and importing rich objects like dates, times, entities, quantities, and so on.

@TaliesinBeynon notation like dataset[[All,1]]=something will happens?

@Murta Dataset's notes section was revamped, to hopefully be much clearer. it has a diagram, for example, to show some of the mappings between the data and the displayed form. it also has a more gradual introduction, starting with simple part specficiations, and then showing how the more general query operators stem from them, giving a table of common types of query. there are 'sections' to make it easier to navigate. hopefully we can create an actual tutorial about Dataset eventually

@Murta but for now it is much better, I think.

@Murta as for general updates for Dataset, there are a bunch of type inference fixes, and speed should be dramatically faster on many part operations, queries, and on display of the Dataset itself, too. between 2 and 100 times faster, depending on what you are doing. as you say, for now, memory usage is the same.

@Murta no, that's not documented to work, is it? i thought we removed that

@TaliesinBeynon yes, it was removed. But there are plans to have it? Here is an exemple of operation that I would like to do with dataset, that is not possible yet:

dataset[[All, 1]] = Normalize[dataset[[All, 1]], Total]

or better:

dataset[[All, "myCol"]] = Normalize[dataset[[All, "myCol"]], Total]

Another example, when reading data from SQL Server into Mathematica, I always do something (that with dataset would be like):

dataset[[All, "dateCol"]] = #[[1, ;; 3]] & /@ dataset[[All, "dateCol"]]

so I can change date column data from SQLDateTime[{2015,3,30,0,0,0}] to {2015,3,30}

@Murta I understand the need. You can also write dataset = dataset[All, {"dateCol" -> (#[[1, 1;;3]]&)}]

@Murta actually, [All, {"dateCol" -> Query[1, 1 ;; 3]}] should also work

@Murta that set mutation has its own peculiarities -- as soon as LHS and the RHS are of different shapes, it can become very confusing to understand the semantics. and even then, it is going to be the same performance in the end to creating a new dataset and setting it, because our language is immutable and the associations have to be rewritten, etc.

@TaliesinBeynon, interesting, it solves the second case. Some idea for the normalization case?

@Murta total = dataset[Total, 1]; dataset = dataset[All, 1 -> (#/total&)];

@Murta not pretty. as bad as doing Normal, then Normalize, then Dataset again, probably.

@Murta I would argue that Normalize should support a level-spec, and that level-specs should generalize to columns via (All, 1) type mechanism... but that's of course a radical proposal that most people in WRI don't like

@Murta but then that would be dataset = Normalize[dataset, Total, {All, 1}], which I think is plenty understandable.

@Murta sorry that should be dataset[All, {1 -> (#/total&)}];

@TaliesinBeynon, I get. With the new Tabular memory efficient structure I can put Dataset in production. Tks for yours explanations! Time to sleep (late here in Brazil)

@TaliesinBeynon PS: I like the Normalize 3th argument ideia. It's elegant.

@Murta sure, good night

Hi @TaliesinBeynon, thanks very much for the information.

@TaliesinBeynon same here, much appreciated.

Is there any neat way to create a window which size will fit the contents and be at least as wide as it's title?

CreatePalette[Button[1], WindowTitle -> "My new palette",
 WindowSize -> All]

?

@Searke @SjoerdC.deVries The checksums are available in the user portal when you download Mathematica.

Quiet around here - must be that everyone (else) is playing with 10.1.

@bobth then as someone who isn't, I must speak out

@Rojo no hay alla?

They really seem to have their company in high esteem with that Wolfram Student Ambassador program

@RolfMertig Jaja, por ahora no

@Rojo Glad to hear I'm not alone.

"Don't panic"

I have 10.1 open, but I'm really just sitting in my cubicle listening to folk songs. As usual, I'm trying to figure out the fastest way to turn my computer into a better version of myself that doesn't get tired or bored so it can continuously produce things for me.

@MichaelHale I'm trying to figure out how to get my computer to figure out how to do what you said for myself

while eating a "milanesa napolitana"

Looks good. I feel like the data we're curating is all so detailed and domain specific that it's hard to use it to get the computer to do something automatically creative.

I've been thinking about my thought process if I go to make a painting of a person vs write a story about a person.

Like if I'm making a painting, I spend more time specifying details about viewpoint, the pose of the person, what their face looks like, etc. If I'm writing a story though, I will spend more time specifying details about their back story, birth date and area, occupation, interests, friends, etc.

I like having easy access to the eccentricity of the moon's orbit, but I want more computable data at a higher level of abstraction.

still no 10.1 for me :(

@xslittlegrass There wasn't advertisement of any sort for me (Home Edition user), but upgrading was available right away. Just that I needed to know that I want to buy an upgrade.

@Pickett same thing here

@Pickett and me :(

@xslittlegrass @Kuba Ahhh the injustice! But I'm still on 10.0.0 so I guess I can't claim to be an early adopter anyway :)

mine 10.1 just came :)