Wolfram Mathematica

ciao

04:03

@SjoerdC.deVries Not a word in English - but used to mean something that performs (i.e. efficient)

Szabolcs

09:26

@SjoerdC.deVries This is what I mean:

arr = RandomReal[1, {1000000, 2}];
ds = Dataset[<|"a" -> #1, "b" -> #2|> & @@@ arr];

The second one is much nicer and has named headers. But the first one is much more efficient in every sense.

In[56]:= ByteCount[ds]

Out[56]= 432000840

In[57]:= ByteCount[arr]

Out[57]= 16000152

In[58]:= ds[Total, "a"] // AbsoluteTiming

Out[58]= {0.200223, 499469.}

In[59]:= arr[[All, 1]] // Total // AbsoluteTiming

Out[59]= {0.013218, 499469.}

In[60]:= ds[All, "a"]; // AbsoluteTiming

Out[60]= {0.099544, Null}

I am very skeptical about storing these values row-wise, each one being an association. Why not store them column-wise, each being a packed array?

Transpose[ds] is like that, but Dataset seems to be designed to work with named columns first, not with named rows ... BTW ByteCount[Transpose[ds]] is down to 16 MB from 432 MB. It's just as small as arr.

Szabolcs

09:54

Working with large data is not handled well in Mathematica. First, you can't import it because the importer is either slow or unreliable (and manual checking is not feasible). And there's no fast importer which allows specifying data types per column, information which could in principle improve both performance and error checking.

Second, when you do import, the data doesn't fit in memory because see above: 432 MB is 27 times more than the 16 MB really necessary.

Third, the framework for out of core procession is still not there. MATLAB now has it. And the idioms it uses are much more at home in Mathematica than in MATLAB. Why didn't Mathematica have it first then?

MATLAB, R, Python and Mathematica are all suitable for this sort of work. What else is out there? And out of these four, which one do you judge to be more or less suitable? While I admit that my experience with the other three is quite limited, and thus I tend to believe whatever marketing demo I see, it appears that Mathematica is at the last place when large data is concerned.

I want Mathematica to be at the first place!

Watch this and repeat it with Mma: go.continuum.io/datashader You can't.

Kuba

@Szabolcs do it, release your rage. Does not matter. As pointed out many times on chat or comments around mma.se, priorities imposed on developers are quite often different from what users expect. Even long term users.

And since goals are not transparent it is not surprising that people tend to say that the main goal is to add something new that we can Tweet nicely.

Szabolcs

10:24

@Kuba OK, you're right. Maybe it's a CSV importer that I should release. Will happen eventually.

What I want or other users want may be ignored. But pointing out that MATLAB, one of the primary competitors, is pulling ahead, should hit home ... MATLAB also has a good Python interface now BTW.

3

Kuba

@Szabolcs good point I hope it will hit :)

C. E.

11:01

I've compiled a C program to a Unix executable and now I want to read the result into Mathematica. For this I'd like to use Get or ReadList, but how do I specify the program path?

Doing f.e. << "!ls" works, but << "!/dir/to/my/program" doesn't work.

Rolf Mertig

11:40

@C.E. RunProcess["/bin/ls", "StandardOutput"]

@Szabolcs In 2014 I saw this live and I thought it will be working in Mathematica any time soon. The question is what soon is. E.g., what does Coming soon at the bottom mean, really?

Jacccy

Hi everybody, I want to export a video which was created via MANIPULATE. My problem is the speed of the switching graphs in the .avi file. When I have 3 graphs, I want that each graph is shown for 4 seconds and than the next graph shall be shown. If each graph was shown once the video shall end. I have the tool AUTOSEQUENCING and "FRAMERATE". But I cannot handle the right settings. Has someone an idea?

Testdata = {{{1, 10}, {2, 20}, {3, 30}}, {{10, 100}, {20, 200}, {30,
300}}, {{100, 1000}, {200, 2000}, {300, 3000}}};

C. E.

@RolfMertig Thanks, that does it.

Szabolcs

12:25

He uses ass for all his associations! When v10 came out I was accused of being inappropriate when doing that same :P That's why I have to use asc now.

@Jacccy Perhaps try producing a list of frame, then export got GIF. GIF definitely supports such low framerates (1/4 fps). I am not sure about real movie formats. Check the settings on the GIF export format documentation page.

Jacccy

12:53

@Szabolcs thank you for the tip ;)

Kuba

14:45

std = 1;
monitor = CreateDialog @ ProgressIndicator[Dynamic[std], {1, 200} ];



Do[
      std = studentNr;
  Print[std];
  Pause[.01];
  ,
      {studentNr, 200}
  ];

NotebookClose @ monitor;

Notebook content is blank unless I put Pause[1] before Do loop. Is there better appraoch?

Unless I comment out Print statement, ProgressIndicator won't be updated properly. Can this be fixed without FinishDynamic?

Kuba

15:00

@Karsten7. any ideas? :)

hmm, additionally it seems to be a regression to how it worked in 10...

Karsten 7.

15:23

Unless I misunderstood your problem, it seems to work fine for me using version 11.0.0 under Windows 10.

@Kuba

JasonB

15:39

So now that I have this gold tag badge, if I try to mark a plotting question for closure, it just happens automatically without waiting for 4 other votes? I don't really like that, I prefer to have consensus

JasonB

15:56

Guess I'll have to be sure from now on when I mark a duplicate, that's probably why the wiz points out possible duplicates in comments rather than votes

Kuba

17:00

@Karsten7. thanks for check, have to prepare better example.

Karsten 7.

17:40

@Kuba Maybe changing Dynamic[std] to Dynamic[Setting@std] helps in your application. That's my current workaround for stubborn Dynamics.

C. E.

Why is it so hard to get this question closed as out of scope? It's a question about the Flickr API (i.e. has nothing to do with Mathematica) by a user who has asked almost 300 questions and should know better.

Let me be very clear: I want that question closed on its own merits, not because of who posted it. But sometimes users perplex me.

Sjoerd C. de Vries

19:52

@Szabolcs I thought it might be some fancy smancy CS term that I should be familiar with, but wasn't. I agree with the Dataset issues you mentioned. I could live with the slowness, but the memory issues are major to me. I only use it for smaller sets with a few thousands of rows. If you can come up with them (I'm still wrestling with the syntax and concepts of Dataset operations), queries can be phrased rather elegantly sometimes.

For example, given the Dataset generated by the code below, can you come up with a query that yields a subset of the dataset, such that for every duplicate combination of "A" and "B" I pick only the row for which "C" is the largest?

ds = Dataset[Flatten[Table[Association["A" -> a, "B" -> b, "C" -> RandomInteger[100]], {a, 3}, {b, 2}, {c, 3}]]]

desired output:

Michael Hale

ds[GroupBy[{#A, #B} &] /* Values, TakeLargestBy[#C &, 1]]

Sjoerd C. de Vries

Also fully agree with the CSV issue, too slow, too much time and memory consuming (though XLS is worse).

@MichaelHale Yep, that's almost the one. But try to index the result ["C"]

errr, [All,"C"], I mean

Michael Hale

ds[GroupBy[{#A, #B} &] /* Values, First@*SortBy[-#C &]][All, "C"]

But I agree that it can be frustrating both MaximalBy and TakeLargestBy always return a list.

Rolf Mertig

Is anybody aware of a digression or bug in LogPlot in V11? I have an involved function being plotted, and in 10.4.1 all is fine and in 11 it is not ...

Szabolcs

20:11

@RolfMertig Yes, let me find it for you ...

Rolf Mertig

@Szabolcs I use Evaluate ... I am trying to narrow it down. It is eating RAM in GB ...

Szabolcs

@Rolf Here: mathematica.stackexchange.com/questions/123943/… mathematica.stackexchange.com/questions/125127/…

@RolfMertig Then it's not that

Rolf Mertig

looks like 11.0.0 is really a beta version ...

Szabolcs

During the beta I did notice some irregularities, such as sampling the function twice, in different way, completely unnecessarily. But there weren't any bad effects that I'm aware of.

Sjoerd C. de Vries

@MichaelHale Interestingly, in v11 the versions with and without First are displayed the same (but the latter cannot be indexed). In v10 they are displayed differently so that the issue is more clear.

@Szabolcs There are various CSV parsers on GitHub and Sourceforge. Perhaps they could be brought to good use?

Of course, the issue is that the convert data to some programming language's internal data representation which not necessarily will resemble Mathematica's

JasonB

20:35

Can anyone tell me what the difference between defining some initialization variable as

init := Module[{}, <do some stuff>]

versus

init[] := Module[{}, <do the same stuff>]

?

Szabolcs

20:47

@SjoerdC.deVries It would be a good opportunity for me to learn boost::spirit ...

Rolf Mertig

21:27

@Szabolcs Thanx,. Though I cannot narrow it down (quickly). My real-world case is too involved, so I just use Plot instead of LogPlot. These bugs are really annoying. WRI needs to take much more care about code quality.

Transcript for