Is there possibly some way to better use Table? I can’t seem to find a solution for this, but I’ll wait to see the final time of Importing all of the files, hoping it will be quicker than how long it took to export them (~24 hours!), before I make a question on it

What's in the files? Like, what's the data type?

Reading 200k files will probably be fairly slow no matter how you slice it, depending on OS, filesystem type, media the files are on, and layout of the files on the FS

Huh, if you catenate multiple .mx files, WL will still read the output but only return the first mx file's data. Makes sense, I guess it stops reading after (End of Wolfram Language binary dump file)

@CarlLange they are all .mx files with variable length (the way I am reading them the largest files are in the middle of the table/list) but no larger than 200ish kilobytes. Each mx file ends up being composed of some n complex 3 x 3 matrices

@CarlLange can you explain what you mean by If you catenate them? I understand what that means via say lists or something, but do you mean that reading in multiple mx files won’t work? I thought cutting them up like that (when exporting the data) would be best for crash proofing

They’re on an m3 ssd drive connected via usb c

Still importing, whew...I'll have to change up how I do this in the future haha!

@CATrevillian you could try Get instead of Import, I did some testing here on a ~200k file with a random matrices and Get (e.g. << "file1.mx") was ~2.5x faster than Import.

did you need all the files separated individually for a reason?

I just did a test DumpSave of ~200,000 files that took under 3 minutes, maybe you could post some sample code? I suppose one thing that might be slowing things down if you are on Windows could antivirus software, maybe try excluding that directory if that's the case.

@kirkus I don’t need them all separated individually—apart from them being runs of different parameters, I was trying to do something that would be “crash proof” by exporting in pieces.

I had a similar run of these that I included an incorrect parameter for (resulting in fails in the center), and I only made the Table of final results, which took maybe ~6800 seconds, and the export of which took only 44 seconds!

I’ll come up with some dummy code & post a MWE once I’ve given this at least 24 hours to run. I figured since reading in one of them was so “instantaneous”, that a multitude of them would not take so much longer! I’ll try Get & to see if excluding the directory lends better results! (I am on Windows)