« first day (3126 days earlier)      last day (1656 days later) » 

CA Trevillian
CA Trevillian
00:14
Is there possibly some way to better use Table? I can’t seem to find a solution for this, but I’ll wait to see the final time of Importing all of the files, hoping it will be quicker than how long it took to export them (~24 hours!), before I make a question on it
Carl Lange
Carl Lange
What's in the files? Like, what's the data type?
Reading 200k files will probably be fairly slow no matter how you slice it, depending on OS, filesystem type, media the files are on, and layout of the files on the FS
Huh, if you catenate multiple .mx files, WL will still read the output but only return the first mx file's data. Makes sense, I guess it stops reading after (End of Wolfram Language binary dump file)
 
7 hours later…
CA Trevillian
CA Trevillian
07:22
@CarlLange they are all .mx files with variable length (the way I am reading them the largest files are in the middle of the table/list) but no larger than 200ish kilobytes. Each mx file ends up being composed of some n complex 3 x 3 matrices
@CarlLange can you explain what you mean by If you catenate them? I understand what that means via say lists or something, but do you mean that reading in multiple mx files won’t work? I thought cutting them up like that (when exporting the data) would be best for crash proofing
They’re on an m3 ssd drive connected via usb c
 
5 hours later…
CA Trevillian
CA Trevillian
12:37
Still importing, whew...I'll have to change up how I do this in the future haha!
 
2 hours later…
kirkus
kirkus
14:15
@CATrevillian you could try Get instead of Import, I did some testing here on a ~200k file with a random matrices and Get (e.g. << "file1.mx") was ~2.5x faster than Import.
did you need all the files separated individually for a reason?
I just did a test DumpSave of ~200,000 files that took under 3 minutes, maybe you could post some sample code? I suppose one thing that might be slowing things down if you are on Windows could antivirus software, maybe try excluding that directory if that's the case.
CA Trevillian
CA Trevillian
14:46
@kirkus I don’t need them all separated individually—apart from them being runs of different parameters, I was trying to do something that would be “crash proof” by exporting in pieces.
I had a similar run of these that I included an incorrect parameter for (resulting in fails in the center), and I only made the Table of final results, which took maybe ~6800 seconds, and the export of which took only 44 seconds!
I’ll come up with some dummy code & post a MWE once I’ve given this at least 24 hours to run. I figured since reading in one of them was so “instantaneous”, that a multitude of them would not take so much longer! I’ll try Get & to see if excluding the directory lends better results! (I am on Windows)
Meta
Meta
15:16
0
Q: Can anyone suggest a way that I can formaly acknowlge users in a piece of accademic work?

Q.P.Over the years I have learnt a great deal from MM SE, in fact I think it has one of the best communities I have experienced out of the ones I have used/lurked on. Several regular and experienced users, but one in particular, have educated me and not just in Mathematica but in statistics also. I'd...

discussion
 
2 hours later…
Meta
Meta
16:53
0
Q: Soliciting opinions to specify an open source WL-like language

alancalvittiThis question from 2013 asks whether m.se is appropriate forum to discuss languages similar to WL. One mentioned, mathics, doesn't seem to be around anymnore, or at least page is unresponsive. I have a slightly different question. Is m.se appropriate to solicit user opinions for what features one...

discussion feature-request

« first day (3126 days earlier)      last day (1656 days later) »