Wolfram Mathematica - 2018-09-21

« first day (2438 days earlier) ← previous day next day → last day (2043 days later) »

1:13 AM

@Szabolcs I'm really into this custom encoder / decoder framework. It's much nicer than what I had been doing. The only question is where I should save these things and how much I should do with that.

Currently I have three folders, two python side and one Mathematica side, which hold the custom encoders/decoders.

6 hours later…

6:57 AM

0

Q: How to enforce priority on custom directory over paclets default repository

As explained in $Path hijacked by PacletManager? loading packages from custom directories may not be so intuitive. Not to mention absolute lack of documentation of interaction between $Path, PacletDirectoryAdd, paclet's version etc. Recent update to linked question explain something that hit me ...

packages paclets development inheritance

@Szabolcs @halirutan @b3m2a1 and others, please help :) ^

7:57 AM

@Kuba Could you please explain again why you need to have the package installed in the default repo?

8:11 AM

I still think that the guarantee to load the latest version is worth something.

Would it be a workable solution to never install in the default repo?

Instead, different versions will be installed in different repos. These repos will be added and removed as needed with PacletDirectoryAdd and PacletDirectoryRemove. Ideally, only one version will be active at a time, so the rule to load the latest version won't even be relevant.

I've been managing my packages in a similar way for a long time, from before I started looking at paclets.

I have a little framework to add/remove packages from the $Path, and I would generally rely on one version being present in the $Path at a time instead of the specific ordering of directories within $Path.

One practical problem was parallelization: $Path had to be set correctly on each subkernel.

So I have additional functions that also modify $Path on subkernels, and do it in such a way that the modification happens automatically when a new kernel is launched.

Other than the parallelization issue, what other problems are there with managing paclet directories and avoiding installing into the default repo?

To people voting to reopen closed questions:

Please write a comment under the question arguing why it should be reopened. This is especially important when the question hasn't been edited much. For example, "You missed this important point about why this question is still important" or "I want to write this-and-this kind of answer".

I'm absolutely willing to contribute my reopen vote, and I'm not hard to convince. Just post a note and explain why.

In fact, in many cases the ideal outcome is that OP fixes the question so that it can be reopened.

8:29 AM

@Szabolcs as mentioned it may happen by accident, sometimes I install stuff to test, sometimes I forgot to clean up.

I understand expecting version to be resolved automatically is desired, but imo it should not be a global property and first path or repo type should be picked and version only within given path / repo type.
If not, fine, but I gave a clear use case so at least an ability to control it is expected.

@Kuba Is there even a public function to see the list of paclet directories and see their order?

There's an Add and a Remove function, but how can I see their ordering?

@Szabolcs I was not digging in internal so much. Stuff I need to do needs to work robustly across versions so I don't play with Private content too much.

Yes, that's my point: there is no public paclet directory list, so "you're not supposed to know or care about the order"

I understand where you're coming from, but I think that if paclet directory ordering determined what gets loaded, that would be just as annoying in some cases and could lead to errors just as easily.

@Szabolcs with respect to documentation, $Path rules. I

I think I would prefer the ability to pin a certain paclet explicitly.

8:34 AM

And with respect to reality, it does not.

I.e. PacletFind["somethign"] might return several results. It would be nice to be able to temporarily mark one of those as the active paclet, so it's the one found by the documentation centre and the one loaded by Needs/Get

You can compare it with FindFile

@Kuba What is PacletDisable and PacletEnable? Have you spelunked these?

@Szabolcs I don't know.

OK, I'll take a look.

Maybe these can deactivate a certain paclet, either temporarily or permanently.

Then you could deactivate the one in the main repo.

8:36 AM

Also, paclet directory add seems to work with Extra Repository (collection), as opposed to User collection (defult)

And I'm surprised that there are different types of collections/repos and nothing comes out of it.

Because Version is superior

@Szabolcs As explained, I don't know up front if there is something newer somehwere, I just need to give priorty for given workspace

And the main package may only be one part of the problem because it affects whatever I have in worspace and expect to load by mypackage

But at some point we need to look for reasonable solutions within the confines of the existing paclet system, even if some of us disagrees with how that system currently works, no? :-)

@Szabolcs the solution is to build packages from dependencies which will be converted to unique contexts etc.

We seem to disagree on some points and agree on some others, but the reality is that Wolfram might not listen to us at all anyway ...

A robust approach to that does not exist and I need to have stuff done earlier :)

My question is not a major one, but not knowing how paclet lookup works may cost you time.

user image

8:42 AM

At least the message is clear.

But the implementation suggests something different ... wait ...

Still no, sorry.

@Szabolcs no worries. I don't really expect a satisfying answer here

The only way to go is what you once asked on main, how to manage dependencies and avoid interference

@Kuba "As explained, I don't know up front if there is something newer somehwere," ...

PacletEnable still seems to be good for this purpose, look -->

user image

> PacletEnable["location"] leaves just this one enabled

Isn't this precisely what you need?

Now to figure out how to enable this functionality ...

Hmm, looks promising

I expected that the the Enable/Disable functionality would be disabled dynamically while the paclet manager is being loaded

8:50 AM

@Szabolcs though PacletEnable // PrintDefinitions does not show anything related to locations

But it seems that this is hard coded ... the code is commented out

user image

It's in Manager.m

I wonder if it's disabled because it's broken or because "we're not supposed to use it"

If you get the opportunity, you could ask TG ...

I will, yes. My guess is because PM isn't finished yet.

9:25 AM

@b3m2a1 What's the status of PJLink now? I see the version number is still 1.0.0. Was the paclet changed? I think it doesn't update unless the version number is incremented.

I just tried to reinstall, and it just hangs.

user image

Wasn't it a bit premature to post on Wolfram Community before it is made a bit more robust?

Is there anyone else in this chatroom who got it working?

10:01 AM

@b3m2a1 Do yo have the location of Mathematica hard coded for Mac? E.g. I do not have /Applications/Mathematica.app. Instead I have /Applications/Mathematica 11.3.app. Can this be set?

More trouble:

user image

What's an easy way to see the compiler output so I can identify the problem?

Also, when compiling a .cpp file, won't using gcc instead of g++ cause trouble?

Actually on the Mac, gcc has been just an alias for clang for a long time. Maybe the alias cc (and c++) is better (as in more universal, and will work on just about any Unix)?

2 hours later…

12:16 PM

Here patt/;test what is done first? patt is matched and if it is matched test is evaluated OR test is evaluated and if it is true patt will be matched?

@vasili111 how could the test be run first: {x_} /; x < 0

@Kuba So that means IF pattern {x_} matches THEN do x<0 ?

@Kuba also can we write {x_} /; x + 1 ?

@vasili111 yes, but unless x = True - 1, it does not make sense :)

12:33 PM

@Kuba Why x = True - 1 ? Does {x_} /; x + 1 means that IF there is an element in list THEN reffer to it as x and THEN make x+1 ?

@vasili111 I'm confused, have you seen ref/Condition?

@Kuba Yes, I read that. But have not understood it well. I cannot understand well and truing to understand it now what exactly /; means and how to use it.

@vasili111 isn't that helping: {6, -7, 3, 2, -1, -2} /. x_ /; x < 0 -> w?

And I said it does not make sense because test is supposed to return True or otherwise is failed.

@Kuba Replace all elements with w that are less than 0?
Maybe patt/;test means IF the patt is true THEN also test same patt with test AND if is true too that patter is matchecd?

Let's call patt in patt /; test a pattern and the whole expression the pattern. So, the pattern matches if a pattern matches and test returns True

12:43 PM

@Kuba Very good explanation. I understand it now. Thank you :)

2 hours later…

2:56 PM

@Szabolcs it’s currently in process. I’m adding a preliminary type decoder system to hook in. The version or location of Mathematica can be passed to start different kernels but for starting a link from Mathematica it shouldn’t matter. That doesn’t reference a binary at all.

@Szabolcs you can run the setup.py file in the PJLinkNativeLibrary directory from the command line. I've just been taking the defaults in the python setup module but I can check to see how that's changed.

3:13 PM

@b3m2a1 will try

Thanks. One issue I can foresee there is that I don't know if I was smart enough about finding the path to the current MathLink library .a file.

Admittedly I'm not sure why it wouldn't complain right out of the gates if that were the case

@b3m2a1 THe thing is, there is no way to do that reliably

There should be a way to specify the location of Mathematica

@Szabolcs yeah there is for starting a kernel but not for this right now. That's definitely something to add.

I'm not so experienced with Python. Can you help me out? Before I run setup.py, I'd need to tell Python in which other directories to look for the required packages. How do you do this?

The idea is to put them on the PYTHONPATH before running. So you'd do something like PYTHONPATH=/path/to/PJLink python3 setup.py build_ext --inplace

3:24 PM

OK

setup.py might actually handle that itself...if it's like the version I used in my original development. It may not be.

Ah nevermind I took that out for this

If you look at the __init__ in the PJLinkNativeLibrary directory, though, you'll see that it manipulates sys.path to make itself available. That's generally what I do instead of manipulating from the command line.

I think it's getting too complicated for me (other issues while running setup.py). I'll wait util you sort it out ...

running build_ext
building 'PJLinkNativeLibrary' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/szhorvat/anaconda/envs/mma/include -arch x86_64 -I/Users/szhorvat/anaconda/envs/mma/include -arch x86_64 -I/Users/szhorvat/anaconda/envs/mma/include/python3.6m -c PJLinkNativeLibrary.cpp -o build/temp.macosx-10.7-x86_64-3.6/PJLinkNativeLibrary.o
PJLinkNativeLibrary.cpp:29:10: fatal error: 'cstdint' file not found
#include <cstdint>

@b3m2a1 See above. That's actually what I would have expected when I saw that C++ source was being compiled in C mode ..

But then why does this not fail for you?

Huh...

I think my default might actually be a C++ compiler

I also don't see any extern "C". I don't know how to extend Python, but I would have expected it to have a C API and not support C++ directly. Is there an issue here?

I don't think so. Many python extensions use C++ I think.

3:33 PM

Or that's because the functions are listed in the PJLinkNativeLibraryMethods array anyway?

Yeah

That's really the key there. And all python extension functions only ever work with pointers to a PyObject

Which is just a struct typedef I think

Basically make a bunch of functions that take and return a PyObject pointer, make that array, add an _init function or something like that that does some boiler plate, and you've got a python extension.

I'm seeing this from setup.py:
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -I/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c PJLinkNativeLibrary.cpp -o build/temp.macosx-10.9-x86_64-3.7/PJLinkNativeLibrary.o

Looks like I can use an environment variable at least for Mac and Windows to have g++ by default

But there's still at least one function in the library that must be present and python will call from the outside based on its name, no?

Which one is that?

PyInit_PJLinkNativeLibrary?

Yeah I think it's the PyInit_* function

and it has PyMODINIT_FUNC, which is a macro that has extern "C" in it

now I understand

Yeah I think that's the only place it really touches. There are also a bunch of naming conventions I think you have to follow, too, in terms of prefixing/suffixing with you module name.

But that might all be python side.

3:44 PM

yes, that's not the issue

the problem is that Python's build system should be instructed to compile this as C++

Yeah I can get that to work with os.environ["CC"] = "g++"

On the other hand I feel like setup or Extension should have an option for this.

@b3m2a1 But that's a hack, and hacks always break eventually ...

You shouldn't accidentally compile C code in C++ mode. It will break.

That's why I'm looking for an option in the actual modules

It is still strange that it does work on your machine

and I can't find a mention of this in the docs: docs.python.org/3/extending/building.html

it seems as if it would detect the file type automatically

The docs say something about the compiler itself being smart enough about handling the .cpp file but that feels wrong to me.

Hmm... it looks like I may have to do this: docs.python.org/3/distutils/configfile.html

Weird though, as I'd expect to be able to pass that directly to setup

3:57 PM

I have a feeling that something else is wrong

even if I take that command line and change gcc to c++, I get "./Python.h:115:10: fatal error: 'context.h' file not found"

setup manages a bunch of paths and things to make the python runtime headers and things available

I might actually be able to drop the included Python.h header in the directory if I'm using setup but not entirely sure.

Yeah I don't even need to included Python.h when I use that

Just checked

yes, actually why do you have that included?

To give Qt Creator some help in knowing which functions are defined

why don't you point Qt Creator to the correct include directories?

Literally just for the syntax highlighting (I originally had everything)

Because that would have required figuring out how to do that :)

This was the quicker but obviously "wrong" way to do it

4:02 PM

Now works even with gcc instead of g++ ...

that was a Python.h apparently not compatible with my python ..

Ah interesting

you should probably remove mathlink.h for the same reasons

Yeah I'll do that

instead, point your IDE to find these in the correct location (where your python and Mma are installed)

When I come back to do more lib edits I'll be sure to.

Meantime I'll need to edit the setup.py mildly to get mathlink.h in there too

4:04 PM

I also see this:

PJLinkNativeLibrary.cpp:549:13: warning: address of stack memory associated with local variable 'view' returned [-Wreturn-stack-address]
    return &view;
            ^~~~

looks like a real bug

view won't exist once the function has returned, so a pointer to view will not be safe to use anymore

That one is a real bug but it's one I need to do a lot more doc reading to figure out what to do about and it hasn't caused me any issues yet. The python buffer interface is poorly-documented and so I'm not sure how to initialize view with malloc.

Like I don't know how much memory it expects me to malloc

4:31 PM

> and it hasn't caused me any issues yet.

^ that's very very dangerous when it comes to C/C++

This is the kind of thing that will work 9 out of 10 times, then it will crash, and you won't know why it crashed

I really do recommend that you add the include paths to Qt Creator

then you can very easily explore the headers with it

it's not just about syntax highlighting, it's about giving the IDE enough information that it can understand the code, so you can get live warnings, you can follow defintions/declarations, etc.

then it's easy to check what a Py_buffer really is

it's just a struct with only trivial types inside

it's not huge so the easiest is to return it as-is

(not return a pointer to it)

If you really want a pointer, then simply allocate it on the stack. Either malloc or new. E.g. Py_buffer view = new Py_buffer, then use view instead of &view. However, if you choose this, you will need to know when and where to deallocate it (delete). Thus I recommend just returning the struct instead

@b3m2a1 ^

Of course I wanted to say "do NOT allocate it on the stack, allocate it on the HEAP"

sorry

Yeah no worries. I'm thinking I'll just return it. I think the original reason for returning the pointer was because I was working off some template code I'd seen for working with these Py_buffer objects.

Then take the address later when I need it

I am trying to understand rules. Here is text from official docs that is modified by me for clarity. Is it right?
lhs->rhs immediate rule or Rule[]: rhs is evaluated only once when the rule is first given.
lhs:>rhs delayed rule or RuleDelayed[]: rhs is evaluated each time when the rule is used

also in both examples, rhs is evaluated before the rule is used?

4:53 PM

With :>, the rhs is not evaluated within the rule

However, if you use it for replacement, then the result of the replacement might no longer be held, then it's evaluated

Example:

In[1]:= Hold[a + a] /. a :> b + b

Out[1]= Hold[(b + b) + (b + b)]

So, "rhs is evaluated each time when the rule is used" is not accurate

:46839778 The docs are often more educational than accurate. I think that is a good thing. Sometimes they give you an easy-to-understand first-approximation initially, but you need to dig deeper to get the full story. It's good to keep that in mind.

But then I didn't look up what they say for RuleDelayed

@Szabolcs What about:
In[14]:= {x, x, x, x} /. x :> RandomInteger[{0, 100}]
Out[14]= {9, 39, 63, 26}

Here RandomInteger is evaluated each time for each replacement.

No, it's not.

In[1]:= On[]

On::trace: On[] --> Null.

In[2]:= {x, x, x, x} /. x :> RandomInteger[{0, 100}]

RuleDelayed::trace:
   x :> RandomInteger[{0, 100}] --> x :> RandomInteger[{0, 100}].

ReplaceAll::trace:
   {x, x, x, x} /. x :> RandomInteger[{0, 100}] -->
    {x, x, x, x} /. x :> RandomInteger[{0, 100}].

ReplaceAll::trace:
   {x, x, x, x} /. x :> RandomInteger[{0, 100}] -->
    {RandomInteger[{0, 100}], RandomInteger[{0, 100}], RandomInteger[{0, 100}],
     RandomInteger[{0, 100}]}.

RandomInteger::trace: RandomInteger[{0, 100}] --> 4.

First the replacement is done. Then the result can be evaluated further, so it is. As is usual, any expression keeps getting evaluated until no more rules apply to it.

Note the intermediate result ` {RandomInteger[{0, 100}], RandomInteger[{0, 100}], RandomInteger[{0, 100}], RandomInteger[{0, 100}]}`

Also, if you want to try the On[]/Off[] thing, in recent versions it's only safe in command line mode

otherwise it triggers a message explosion

You can still use Trace and TracePrint, but I find their output harder to read

5:09 PM

@Szabolcs
So, as I understand:
with ->
1. evaluate rhs.
2. use rule as many time as needed with already evaluated rhs.

with:>
1. use rule as many time as needed.
2. evaluate result of the rule.

Is it correct?

@vasili111 Yes!

But none of this is specific to Rule or RuleDelayed. It is all a natural effect of the usual evaluation rules with or without Hold* attributes

@Szabolcs
Thank you :) Also that On[]/Off[] thing is great! I tryed it in notebook and it freazes for some time but that is not proble. Output of it is really helpful :)

@vasili111 Try it on the command line

@vasili111 Also, given your interest in this topic, the first 5-6 sections of reference.wolfram.com/language/tutorial/… are a very useful read

5:26 PM

@Szabolcs Nice link with definitely read it.
Also is there any good tutorial about On/Off beside ref/On[] ref/Off ?

Sorry, I don't remember where I learned about them ... it was a really long time ago, when they still worked well in notebooks.

Maybe version 5-ish ...

@Szabolcs no problem. Thanks again for help :)

2 hours later…

7:48 PM

Stephen_Wolfram's Twitch video RSS: Live CEOing (146): New Deep Learning Features in #WolfLang - LIVE

posted on September 21, 2018

1 hour later…

9:09 PM

Stephen_Wolfram's Twitch video RSS: Live CEOing (147): Chemistry Functions in #WolfLang - LIVE

posted on September 21, 2018

« first day (2438 days earlier) ← previous day next day → last day (2043 days later) »

Transcript for

Sep '1821

Wolfram Mathematica

Welcome! This is the main Mathematica chat room for mathematic...

join this room
about this room

00:00

06:00

12:00

18:00

all times are UTC

site design / logo © 2024 Stack Exchange Inc; legal

mobile