Streaming package (Wolfram Language)

Leonid Shifrin

09:59

Hi everyone!

Feel free to ask here all Streaming-related questions that may require discussion that don't fit the SE Q/A fotmat.

Szabolcs

10:15

@Leonid I actually never used it and I have forgotten about it. It would be a good solution for what I tried to ask here, right? community.wolfram.com/groups/-/m/t/1781821

What I am looking for: have a "generator" for combinatorial objects of which there may be too many to generate at the same time.

Problems like this one: mathematica.stackexchange.com/a/1285/12

My impression was that the Streaming stuff is focused on cases when one might want to store data on disc, and when processing individual items takes long enough that the overhead from the Streaming framework is negligible.

The combinatorial applications I had in mind would not need to use the disc, and would need to process a very large number of items where processing each individual item is very quick (thus the overhead might be a big deal)

There is actually a use of the Streaming framework exactly in that thread. mathematica.stackexchange.com/a/140679/12

Leonid Shifrin

11:55

@Szabolcs Yes, exactly. Streaming can be used for exactly the type of problems you mentioned in your community post. In fact, that post of yours initiated a vivid internal discussion within the company, where Streaming was also a part

@Szabolcs Streaming has a mode which does not involve the disk - can be done entirely in memory.

I guess what I need to do soon is to provide a comprehensive set of examples to compensate for a lack of documentation

mikado

19:21

@LeonidShifrin thanks for offering to look at this problem.

Leonid Shifrin

We could look at it now for a little while, if you feel like it

mikado

I'm using 12.0.0 for Linux x86 (64-bit) (April 7, 2019)

Happy to try anything you suggest

Leonid Shifrin

I see. I can later check on Linux. For now, give me a second, I will tell you what to try first

The first thing to try is open the file:

FindFile["StreamingPatch`"] // SystemOpen

This has to open Init.m in that package

mikado

It's opened a file "init.m"

Leonid Shifrin

ok, now you should see there the Scan function call

mikado

19:26

Got it

Leonid Shifrin

Add Echo @ to Flatten @ StringCases[...]

So you get Echo @ Flatten @ StringCases[...]

mikado

Done

Leonid Shifrin

Then save the file and run <<StreamingPatch`

What do you see?

mikado

Done in a new kernel. See "{}"

Leonid Shifrin

Ok, that is a problem

There should be a bunch of symbols

What do you get for Names["Streaming`*"]?

mikado

19:29

{"CachedObject", "StreamingDataset", "StreamingEvaluate", \
"StreamingMap"}

Leonid Shifrin

This is correct

Ok, give me one sec

This might be a bug in the patch indeed

Try this:

Replace the line

Names["Streaming`DataStructures`Interfaces`*"]

with the code

Block[{$Context ="Streaming`DataStructures`Interfaces`"}, Names["Streaming`DataStructures`Interfaces`*"]]

And try the same thing on the fresh kernel

I mean, the result of the Echo @ Flatten[...]

I have fixed the backticks

And let me know whether now Echo is showing a different thing (not an empty list)

mikado

Think I've made the correct change. Still get "{}"

Leonid Shifrin

Ok.

Now, try just Names["Streaming`DataStructures`Interfaces`*"] - what does this give?

mikado

{}

Leonid Shifrin

hmmm. Did you run Get["StreamingPatch`"] before this?

mikado

19:39

No. I'll do that again.

Leonid Shifrin

Yeah, try first Get["StreamingPatch`"], then that

mikado

Is Get and << the same?

Leonid Shifrin

Yes, they are the same

But for <<, you don't need quotes

around the context name, I mean

mikado

OK I did "<<" and "Get" and have same thing both ways

Leonid Shifrin

Interesting. Let me think a bit

mikado

19:45

Leonid Shifrin

I see. Try the following:

On a fresh kernel

First, Needs["Streaming`"]

Then

Block[{$Path = Prepend[$Path, DirectoryName @ DirectoryName @ FindFile["StreamingPatch`"]]},
    Get["Interfaces`"]
]

And finally after that

mikado

Get::noopen: Cannot open Interfaces`.

Leonid Shifrin

Names["Streaming`DataStructures`Interfaces`*"]

@mikado Ok, this is the root of the problem apparently

Give me another moment

So this:

Block[{$Path =
   Prepend[$Path,
    DirectoryName @ DirectoryName @ FindFile["StreamingPatch`"]]},
     FindFile["Interfaces`"]
 ]

Will probably return $Failed for you?

mikado

$Failed as predicted

Leonid Shifrin

Yeah. It does not fail for me on Mac

Let me think a bit more

Can you do SystemOpen on this: DirectoryName @ DirectoryName @ FindFile["StreamingPatch`"]], and upload a screenshot of the resulting folder structure?

mikado

19:56

$ ls -l .Mathematica/Applications/StreamingPatch
total 12
drwxr-xr-x 3 mike mike 4096 Sep 8 20:18 Interfaces
drwxr-xr-x 2 mike mike 4096 Sep 8 20:18 Kernel
-rwxr-xr-x 1 mike mike 957 Sep 8 20:18 StreamingPatchBootstrap.m

That OK?

Leonid Shifrin

Yep, looks fine

mikado

$ ls -l .Mathematica/Applications/StreamingPatch/Interfaces
total 28
-rwxr-xr-x 1 mike mike 4714 Sep 8 20:18 Accumulator.m
-rwxr-xr-x 1 mike mike 3855 Sep 8 20:18 Iterator.m
drwxr-xr-x 2 mike mike 4096 Sep 8 20:18 Kernel
-rwxr-xr-x 1 mike mike 3386 Sep 8 20:18 Set.m
-rwxr-xr-x 1 mike mike 2676 Sep 8 20:18 Storage.m
-rwxr-xr-x 1 mike mike 1670 Sep 8 20:18 Writer.m

Leonid Shifrin

And inside Kernel in Interfaces, there is Init.m, right?

mikado

$ ls -l .Mathematica/Applications/StreamingPatch/Interfaces/Kernel/
total 4
-rwxr-xr-x 1 mike mike 33 Sep 8 20:18 Init.m

Leonid Shifrin

This is really strange. Get should be able to identify this as a package and load it (FindFile["Interfaces`"] in that Block should have resolved to this Init.m), yet this did not happen

Can you copy that Interfaces folder to some test folder you have in your user space, which is not hidden or read-only etc, and then try the same from there:

Block[{$Path =
   Prepend[$Path, the-folder-you-copied-Interfaces-to]},
     FindFile["Interfaces`"]
 ]

mikado

20:06

Ok - I'm checking I've got this right, as it failed at first attempt

Leonid Shifrin

So, still $Failed?

It might be the root of the original build issue, that actually required the patch in the first place

Since the build machine is probably Linux

mikado

I have

$ ls -l ~/MathematicaTest/
total 4
drwxr-xr-x 3 mike mike 4096 Sep 8 20:18 Interfaces

Block[{$Path = Prepend[$Path, "~/MathematicaTest/"]},
FindFile["Interfaces`"]]

Fails

Leonid Shifrin

Can you try using full / complete path instead of "~", just to be absolutely sure?

mikado

Still the same

Leonid Shifrin

The next thing to try is to rename the folder Interfaces in MathematicaTest, to say "MyInterfaces", and then also in FindFile, to FindFile["MyInterfaces`"] - can you try this?

I suspect that "Interfaces" might be some magic token

mikado

20:13

Stile Failed

Leonid Shifrin

hmm. I think for now I am somewhat out of ideas, and also I can't continue with this today for much longer. But we have narrowed down the problem very considerably. The next step will be for me to try reproducing this on Linux, and playing with this.

What flavor of Linux you have?

mikado

Linux Mint (17.3 I think)

Leonid Shifrin

Ok, thanks. I have CentOS, but hopefully the difference should not matter. I will try to reproduce this issue in the next couple of days, and come back here.

Basically, the problem is very stupid: we just need to find a way to load that subpackage

That is, Interfaces

mikado

Thank you very much for looking at it

Leonid Shifrin

No problem. Also, I suspect that this issue is the reason why the Streaming build in the layout is corrupt in the first place, with Interfaces missing from it - which is why that patch is needed

Keep an eye on this chat, I will ping you whenever I get something

Transcript for

Streaming package (Wolfram Language)