Just kidding around :) Didn't mean to come off like I was criticizing!

NDSolve`ProcessEquations is horribly slow for large systems (see mathematica.stackexchange.com/questions/131411 for example)

Does anyone think it's possible to work around it by constructing our own NDSolve`StateData?

Assuming that the system is already defined as either first-order equations (for ODE solvers) or everything equals zero (for DAE solvers)

Or is that too undocumented to even fathom?

@user21, @MichaelE2, others: any ideas?

I've got a system that takes 25.3 seconds to set up as a DAE and 1.68 seconds to solve until t=10^7!

Or 114.25 seconds to set up as ODEs and 71.2 seconds to solve

😤

@CarlLange wanna race to 10K?

I've got a head start but you're moving a lot faster

@ChrisK no idea. Probably it's just a lot of symbols to process.

@user21 It's putting those symbols in the right order in the undocumented internals of NDSolve`StateData that worries me :)

Haha no worries @CarlLange!! Any languages you’d recommend? I think python is the next “scientific” language I’d pick up, not sure about Julia or not...though there is some amount of special to having WL be my first (and currently only) language ;)

@halirutan going to give it another go, maybe the web verification is what got me and I have to input my license key somehow? I’m willing to pay for it, as it is well deserved, but your website does say something about students/teachers/public science folks having free use :))

@CATrevillian python and C++ are the most common languages these days, as far as I can see it. Python is nice because it's a well written, well thought-out, easy-to-work-with language that can serve as an interface layer on computationally intensive numerical work written in C++. And of course, C/C++ is the language that most large-scale numerical software is written in these days.

An example of where these two work together is here.

That's a modern library to do some of the complicated chemical calculations that @JasonB. or I might use as starting points for our work.

@b3m2a1 and C/C++ carry same/similar syntax save for C++ having additional something-or-others? If I understand correctly, one would want to write something that needs to run fast in C, and if one would like to call it from the WL within mma, this is possible using some of the CCompiler functions? I do know that one is more procedural, whereas python is still something of a functional language? Trying to relate it to my WL knowledge ;)

@CATrevillian python is somewhat more functional, yes. But it's all pretty procedural. The big thing C++ does is allow you to use OOP--which is huge since OOP is hands-down the least mentally taxing programming style.

Python is pure OOP. Everything in python is an object. There is no raw int or anything like that.

You can call C++ code in Mathematica via LibraryLink, but my recollection is there's lots of boiler plate.

@Szabolcs has tools to make that less annoying.

Alternatively you can use MathLink to create the most flexible interfaces to C++ from Mathematica, but it took me ~1 month to handle all of the little tricks and difficulties and work around the bugs in it when I was writing my python/Mathematica bi-directional interface.

It's basically undocumented which makes it a pain to work with, but if you really want Mathematica to play well with your program you use MathLink.

Whenever @halirutan gets to hooking his IntelliJ plugin into a Mathematica NB runtime it'll be through MathLink (but probably working through J/Link as an interop layer).

The big annoyance in C++ is the rigidity of the type system, but this is also what makes it fast. Python allows you to write flexible code with basically no brain-power. Mathematica is an interesting language design-wise, but some poorly-considered decisions make it a bit painful to work with.

Bug or no bug?: ParametricPlot[{x, x^2}, {x, 0, 1, 2, 3}]

@MichaelE2 It'd definitely be more graceful with an error message; were you expecting any output?

Trying to figure out how to do something that seems like it should be simple, though if it's not then I may make a site question out of it:

I first create 3-tuples of numbers from -2 to 2 via Tuples[Range[-3,3],3]

I now want to group them according to the following equivalence: Two tuples are equivalent if either 1) they are permutations of one another, or 2) the second is a permutation of the first if I change exactly two signs.

so {1,3,-2} is equivalent to {1,-2,3} and {1,-3,2} but not to {1,3,2}

Getting just the first part to work is not so difficult: GatherBy[list,Sort] seems to suffice (though it may be rather slow)

But the signs part is not obvious at all

@ChrisK No, just checking how the system handles input errors.

not well in that case

I ask, because I remember your amazing journey to the center of the InterpolatingFunction that helped me one time

do you have any thoughts on my question about the feasibility of making a new type of NDSolve`ProcessingEquations?

otherwise I might have to resort to fortran!

@MichaelE2 ^^^\

@b3m2a1 ah ok, that explains a ton, thank you! So would this evaluate to true then? "Everything in Python is an object"==="Everything in Wolfram Language is a function"?

no no - everything in WL is an expression

@JasonB. D'; aaaahhh---oops, I'm not good at words!!!

I see those as equivalent because my words aren't entirely well.

@CATrevillian The two are similar, except objects are many times more useful and powerful. The (general) stateless-ness of an Expression is often helpful, but I'd much rather have something that gives me better encapsulation and expressive power, even if it requires some level of mutability. Of course, WL expressions themselves are implemented as C++ objects (either that or as C structs...). That means that their inherent mutability sometimes leaks out.

E.g. with System`Private`SetValid

And this mutability is very powerful when it bubbles up

@ChrisK I've never figured out where the solution data is kept while NDSolve is integrating. It seems to be in a private local variable attached somehow to the state object. Also, when you play with it or with interpolating functions, the kernel tends to unceremoniously crash, because deep down the code assumes you've given it bug-free input. So if you want a new ProcessingEquations to work with NDSolve, it sounds difficult, esp. given no docs.

@MichaelE2 I see... time to dust off the old g77 :)

Maybe I'll send WRI a suggestion

I mean, how hard could it be for the first step of ProcessingEquations to be check whether all the equations are first order and the derivatives on the LHS?

and if they are, then skip the Solving for derivatives step

Anyone have a question where a sort-of automated calculation monitor is useful? I just took a few minutes to write up a nice one:

@ChrisK How fast is Solve compared to ProcessEquations? On the linked problem with nMax = 50 it's 40 vs. 2.5 sec. Hmm, NSolve takes only 1 sec.; I thought it might balk since you don't get numeric solutions. In any case you have to construct the NumericalFunction. Perhaps one could (should be able to) give NDSolve a first-order RHS, kind of like FindRoot[#^2 - 2 &, {1.}]...

@b3m2a1 I'd integrate that--I calculate way too many Eigensystems way too often for approximating nonlinear dispersion spectrums, this kind of thing would be useful to monitor those hour+ long calculations!

mMax = 50;
Ffuncs[r_] = Table[F[n, m][r], {n, mMax}, {m, mMax}];
(nsols = NSolve[
      Flatten@{Thread[
         Flatten[D[Ffuncs[r], r]] == Flatten[Ffuncs[r].Ffuncs[r]]]},
      Flatten@Thread[Flatten[D[Ffuncs[r], r]]]
      ] /. 1. -> 1) // {ListQ[#], Dimensions[#]} & // AbsoluteTiming
nf = Experimental`CreateNumericalFunction[
    Join[{r}, Flatten@Ffuncs[r] /. subs],
    Flatten[D[Ffuncs[r], r]] /. First@nsols /.
     rhs_ :> (Hold[#] &@subs /. Rule -> Set /.
        Hold[v_] :> Block[{F}, v; rhs]),

@b3m2a1 None come to mind. I might add, for your consideration, that in my simple ad-hoc monitor, I often use an expression, something fast like Last@expr, and not just variables. And if I don't mind a slight performance hit, maybe ListPlot[data] when the visualization might show me where something is happening.

@MichaelE2 yeah I was envisioning it either being automatic, or another option like "EquationSimplification" -> "FirstOrder" where you're trusted to provide a first-order system and it skips the solve bit

one other crazy possibility: try SystemModelSimulate instead of NDSolve

@CATrevillian Python would be the most useful to learn in my opinion, it will give you a good grounding in how "most" people build software. Probably the most valuable language out there right now.

@ChrisK Maybe to 15k, even the odds a little 😉 particularly since I won't be around for another week or two

Well, I'm off. Time to go become the first man in history to walk all of Ireland's trails. See you all in a few days ✌️

(and do give us a subscribe on YouTube if you're mildly interested, we make videos about each trail and are tantalisingly close to 1000 subscribers - youtube.com/toughsoles !

)

@CarlLange great :D I'm on at least remotely the right trail then! you stay on assuredly the right trail, though!! ;P I'm gonna take a nice gander at that channel of yours's for my diffused mode processing :))