@C.E. Thanks.. I will play with your ideas and see what I get

Is there a way to get youtube.com/watch?v=AgkfIQ4IGaM (yosinski.com/deepvis) in mathematica for example LeNet5 for handwritten digits

@Kuba as I use this thing I keep running into features I need to implement :| Happily it's getting to be fewer and fewer, but as I was working on my Git integration I realized I needed a way to emit messages in a clean easy to use way so that I'm not rewriting MessageDialog code all over the place (I'm gonna do this with a nine-patch formatted Panel inside an AttachedCell in the bottom right corner of the notebook).

I will also at some point need to write some stylesheets for things like Input and MessageDialog too, though, so that the important features can just be handled in the at the StyleDefinitions level.

@b3m2a1 don't commit too much before you make sure it works on Win at all :)

@Szabolcs I agree with most of your statements. I think a consistent, centrally designed, unified framework that can uniformly interpret input and evaluate calculation is a very badly scaling design. The more subfields you try to integrate the harder it is to make them compatible with each and every other aspect of the language, and ultimately there will be dead-ends and abdondoned parts where making it fully compatible is just too much work and too little benefit.

And, as a user, I've also realized that robustly extending Mathematica to my needs would mean that I actually develop at WRI. Otherwise, everything I code would be something half-baked, half-integrated superficial package.

@b3m2a1 It really depends on your data. In general, only OK, but it's a lot faster than using Interpreter on every field. There's a lot of room for improvement :)

@psimeson Are you looking for something like this? community.wolfram.com/groups/-/m/t/1369571

@CarlLange Are you using vectorized calls to Interpreter? You can get multiple order of magnitude improvements from that.

@Kuba True :) right now I'm doing a refactoring to make sure that even if the thing doesn't work cross-platform I have components that others can still use (i.e. tabs, attached popup messages, toolbar interfaces, file browser menus, etc.)

The Windows FE might not be able to keep up with everything, but at least it can do some of this stuff

@Kuba I think you want {2, 3, 1}

The list specifies that dimension 1 becomes dimension 2, dimension 2 becomes dimension 3 and dimension 3 becomes dimension 1

The mapping is really MapIndexed[#2[[1]] -> #&, dims]

@b3m2a1 now it makes sense. Somehow I assumed the second argument should be dimensions' permutation of an output array.

Thanks.

@Kuba you made the mistake of assuming it would have been implemented in the more logical way

Always dangerous to assume logic on the part of Mathematica design

@b3m2a1 I often interprete things the other way around so I was hesitant to make that statement. But I am glad that makes sense for you too.

Yeah I assumed it'd be like ArrayReshape at first

@b3m2a1 Yeah. Still kinda slow though. Plus some parts of it use credits. SemanticImport seems to do everything offline, which is cool

I also find that Transpose notation to be kind of difficult, it leads me to trial and error pretty often

@Kuba You can use InversePermutation[] as a helper function if you want to be able to enter the reordering for Transpose[] in that fashion.

Compare Dimensions[Transpose[Array[C, {5, 4, 3}], {3, 1, 2}]] and Dimensions[Transpose[Array[C, {5, 4, 3}], InversePermutation[{3, 1, 2}]]].

(As for Flatten[]...)

@J.M.isslightlypensive good to know. While longer it is way easier to read for me

Thanks

Of course, you can always write something like myTranspose[l_?ArrayQ, levels_List] := Transpose[l, InversePermutation[levels]] for the sake of convenience.

I understand your predicament. I have a similar complaint about the spherical coordinates functionality of Mathematica. My favorite convention is exactly the reverse of Mathematica's preferred convention, so I either write wrapper routines or avoid them altogether.

A cute way to get notifications to output in a notebook:

These can also be set to disappear after a given amount of time by making a Dynamic with an UpdateInterval that deletes itself on second evaluation.

@J.M.isslightlypensive is there a valid reason behind the current design? For me it requires flexing my brain while the inverse seems straightforward. But maybe that is because I am not used to something that is common somwhere.

@b3m2a1 nice, I did it for my recent project but it was uglier as I don't have patience for FE resources, ninc patch etc. I also decided to close them with scheduled tasks as I try to avoid 'dynamic calculates something'.

@b3m2a1 can a directory by a paclet server?

@Kuba I steal all my nine patches from the Mathematica layout

Yup

Use "file://" as the URL

It should work as of 11.2 I think? In 11.1 and below you need a minor patch that is here:

@b3m2a1 thanks I suspected it should work but you never know :)

@Kuba Tensor calculus/algebra is not one of my strong areas, but it is my understanding that specialists prefer that permutation convention (e.g. this preprint).

github.com/b3m2a1/mathematica-BTools/blob/…

@b3m2a1 11.3+ is fine enough but thanks

Part 1 of Git integration down :)

(well the big coup here is that I can provide actual feedback on these operations and so let people know if they failed)

@b3m2a1 any reason for not meaningful messages? :)

@Kuba it's what git commit on the command line returns

I might change it up to say more, but what's nice is that in the case of errors it also makes that clear

@b3m2a1 I mean, why are you always using auto generated commit messages?

I wonder if it would be possible to do a full git porcelain, like magit for emacs, in MMA. Probably thousands of hours of work...

Regarding the my messages from yesterday that were deleted: I wanted to make it clear that the deletion was done at my request because I think that this opinion could have been expressed more politely. Just in case someone suspected censorship ...

@Szabolcs, from my point of view, I think you were polite enough.

Good morning, MSE world. Just a friendly note to let you know that Stephen is scheduled to do a livestream today at 1:00 CDT.

Of course, as you all know, that definitely could change between now and then. I'll keep you posted if it should happen to get cancelled.

@JKlug CONFIRMED 12 Release Livestream at 1300 ;)

(I am just kidding, sorry)

@chuy @Szabolcs yeah it seemed to be a very well-thought-out well articulated opinion

@Kuba Ohh. Just means I don't have to write a DialogInput wrapped to capture a commit message and for the most part all I really want out of a commit message (for my own stuff) is when I committed it.

I'm gonna move to capturing DialogInput eventually to do it, but not yet

@b3m2a1 You are correct. No "glorious bugs" would be worth the wait.

@CarlLange, you have no idea how anxious I am to have v12 finished! Perhaps Stephen will address the release during the livestream. No promised though!

@CarlLange @b3m2a1 I second the suggestion to use listable calls to interpreter, as they are much faster than single calls. However on top of that structured (Number, Real, EmailAddress...) interpreter types are up to 20 times faster on large lists of input.

in v12

@Carlo want to give us a pointer to where this will be in the docs?

Or do you just mean those standard types are a lot faster?

@Carlo 👍Perhaps an example of how to do nice listable calls within a dataset would be good for the docs of Dataset

ah, what am I saying, doesn't dataset[All, Interpreter["Quantity"]] do that?

nope, it sure does not.

where the values are just "$SomeNumber people"

Sorry guys, I'm sure this will come as a shock, but the livestream meeting has been cancelled.

@JKlug CONFIRMED 12 RELEASE CANCELLED

😆

We're going to just skip v12 and go right to v13 if that's okay with you, @CarlLange

@JKlug As long as it releases in the next three weeks that's fine with me

I have some trails that need to get walked after that and won't be able to afford my license!

Ah well, thanks anyways @JKlug, appreciate the attempt :)

this is 12

this is 11.3

@CarlLange Quantity has to talk to alpha, so it has not been improved in v12, however the big performance difference has always been there, because if you do a single call we can batch the calls to the alpha API.

you can use ddd[Interpreter["Quantity"]] (but you're gonna lose the association structure

@CarlLange This one looks more like a style transfer.. I want to visualize what each layer of neural net is learning... basically I want to see what features were important for each classes ... This will help us to see what non-trivial features NN picked up

@ChrisK currently integrating my docs stuff into my little IDE. It's looking like having the main project structure there will make everything cleaner to work with (and I can add more stuff to the toolbars and plugins without it feeling clunky):

Shouldn't be much more work on top of this, but I gotta make sure the workflows are really clean.

@Carlo That's the problem, right? You want to make the most efficient call but you also don't want faff moving things in and out of a dataset structure. Particularly if I'm doing some query on some multidimensional dataset (ds[All,Select[foo==bar], ;; ;; 4, Last, Interpreter["Whatever"]]

How annoying is it to have to move the list of interpretable values out and into a dataset in that way?

@psimeson Ah, you should just be able to do Image /@ NetTake[NetModel["SqueezeNet V1.1 Trained on ImageNet Competition Data"], "fire3"][ExampleData[{"TestImage", "Lena"}]]

might look better on a different network but that's the one I had lying around

@CarlLange Thanks, I will look into it... on a completely different topic... is it a good to assume that (for pictures) the missing data can be replaced by the mean of the neighboring points... I have optical profilometry data that measures the height but some data is just -1, which is nonsense and I am trying to think of a way to get around it

@psimeson try Inpaint

@CarlLange You can try to force batch interpretation with Block, e.g.:

batchInterpret[call_] :=
  Block[
   {
    interpMe,
    interpRes,
    interps
    },
   {interpRes, interps} =
    Reap[
     Block[{Interpreter =
        Function[{k}, Sow[#, k]; Hold@interpMe[k, Hash[#]]]},
      call
      ],
     _,
     # ->
       AssociationThread[Hash /@ #2, Interpreter[#][#2]] &
     ];
   interps = Association@interps;
   interpMe[k_, r_] :=
    interps[k, r];
   ReleaseHold@interpRes
   ];
batchInterpret~SetAttributes~HoldFirst;

Something batchInterpret[ds[All, Select[foo == bar], ;; ;; 4, Last, Interpreter["Whatever"]]] might work

Haven't tried it though so you'll probably need to dekink at the end of the call. The only tricky bit is to force interpMe to reevalutate with the new definition I think.

Or maybe you can drop the Hold and ReleaseHold and it'll just work...

Ah wait wrap a Function around that Sow and stuff

Ah yeah clearly it's buying us a little bit:

waterfalls = CommonName[BlockRandom@RandomEntity["Waterfall", 100]];

batchInterpret[Interpreter["Waterfall"] /@ waterfalls] //
  RepeatedTiming // First

0.035

Interpreter["Waterfall"] /@ waterfalls // RepeatedTiming // First

0.117

batchInterpret[Interpreter["Waterfall"] /@ waterfalls] ==
 Interpreter["Waterfall"] /@ waterfalls

True

Ah yeah but the reevaluation is a bit annoying to make work. @CarlLange try this:

batchInterpret[call_] :=
  Block[
   {
    interpMe,
    interpRes,
    interps
    },
   {interpRes, interps} =
    Reap[
     Block[{Interpreter =
        Function[{k},
         Function[Sow[#, k]; interpMe[k, Hash[#]]]
         ]},
      call
      ],
     _,
     # ->
       AssociationThread[Hash /@ #2, Interpreter[#][#2]] &
     ];
   interps = Association@interps;
   interpMe[k_, r_] :=
    interps[k, r];
   interpRes /. e_interpMe :> RuleCondition[e, True]
   ];
batchInterpret~SetAttributes~HoldFirst;

waterfallsDs =
  BlockRandom@
   Dataset[
    AssociationThread[
     RandomReal[{}, 100],
     With[{re = RandomEntity["Waterfall", 100]},
      CommonName[re]
      ]
     ]
    ];

waterfallsDs[All, Interpreter["Waterfall"]] // RepeatedTiming // First

0.11

batchInterpret[waterfallsDs[All, Interpreter["Waterfall"]]][[1]] //
  RepeatedTiming // First

0.043

batchInterpret[waterfallsDs[All, Interpreter["Waterfall"]]] ==
 waterfallsDs[All, Interpreter["Waterfall"]]

True

You could also try to target things more by using Replace with a levelspec instead of ReplaceAll at the end

That would avoid sweeping through big packed arrays if you have them

e.g. do:

batchInterpret[call_, level_: All] :=
 Block[{ ...},
  ...;
   Replace[interpRes, e_interpMe :> RuleCondition[e, True], level]
  ]

And feed in {5} if you're only operating at level 5