Leonid Shifrin

@LukasLang Good point. That's an interesting tidbit, which I either forgot or actually did not know about these functions. I did find a reference on M SE on that.

Otherwise, this looks very nice, and FWIW probably close to what I would've tried as well.

Finally, there is this problem that symbols with both UpValues and DownValues are not handled in this approach, and I see why you want this behavior, but this certainly looks like a limitation that needs at least to be mentioned.

@LukasLang It also seems to have a typo, I guess you meant Language`ExtendedFullDefinition[proxy] = ..., not Language`ExtendedFullDefinition[] = ....

@LukasLang That's certainly a nice attempt, and it seems to go a long way towards the desired result (I mean the code you added to your answer). I am a little concerned about redefining Module (even with InheritedBlock), since this is done for all Module-s inside that part of execution stack, out of which only very few might be concerned with this particular closure. In other words, I would try to make things more lexical. This is not so hard to do.

@user293787 I intend to revisit this later when I get more time. I think this is really important question / topic. But this solution certainly is a good starting point.

@LukasLang One particular case that will badly fail within the current approach, is when the closure being defined, returns other closures as a result of its operation, and those other closures use some of the inner local symbols. In that case, removal of local symbols will make those returned closures invalid, whereas they could still be referenced / in use. So the problem is more complex, and a better approach would probably be to wrap something like addGCMarker around every local symbol that is found to have DownValues, SubValues or UpValues. That's where "static" analysis can help.

@LukasLang There might be a different scheme altogether, where one simply renames all relevant Module variables of inner Module-s in some way, during the defs processing stage (when the closure is created from "source" code), that would allow the cleanup to unambiguously remove them all based on some name pattern, that will be unique for a given closure, but not tied to a particular internal Module or their set.

@LukasLang I was thinking of something along the lines of this, modified to the needs of this problem (like not excluding defs inside Module - I have some comment for that part in that code), and also recording the depth of the Module to identify conflicting local vars. That is, for the "static" approach. If that can't be made fully robust, then the "dynamic" approach based on Language`ExtendedFullDefinition is probably the way to go, but I would try to be more selective there, while still accounting for nested Modules etc.

@LukasLang Very nice solution overall. But I think that, for this to be fully robust, the dependency tracker should be more sophisticated, to process possible nested Module- s. Also, I would try to perform the analysis at the point when addGCMarker is applied, not when AddCleanupFunction is called - although for this point I don't have a very strong opinion (in particular because one has more information available at run-time, compared to "definition-time", which might be important and is an argument in favour of your current approach).

@user293787 You will probably need to wrap addGCMarker around any local symbol with DownValues. Can be done also inside the main Module that defines f. That can probably be automated by some meta-programming which finds local dependencies. I will look into this. At the moment this is indeed a bit unwieldy: getMultiplier[n_]:= addGCMarker @ Module[{f, g, gFinal},f[x_] := n * gFinal[x]; g[x_] := x; gFinal = addGCMarker[g]; f] and manual.

@JasonB. I have starred this project of yours long time ago, with a plan to use it in some of my projects. This Q/A seems like a perfect application for it.

I have a version based on your marker idea, but which does not require a SubValues syntax, but instead uses Function: addGCMarker[funcSymbol_Symbol] := Module[{marker}, AddCleanupFunction[marker, Echo @ "Removed!"; Remove @ funcSymbol]; Function[marker;funcSymbol[##]]];. This allows one to define simply: getMultiplier[n_]:= addGCMarker @ Module[{f},f[x_] := n * x; f]. I also have a utility function to download / install the ExpressionCleanup paclet. I could edit this into your answer, if you don't mind. This would be better than updating mine, since mine uses a different idea.

@PierreALBARÈDE Sure, np )

I really have no time for either right now, sorry :) But you sure can ask a new question, there are many others who might have the time to answer.

In other words, by the time you get to Unevaluated[Function[a,999][a]]/.a->0 or similar, the buggy behavior has already happened

Depends on what you mean by equivalent. It is that equivalence that breaks things, so I'd rather say these two things should not be equivalent - but they are in the current semantics of WL

Then what is it you disagree with me about?

By the time you get Unevaluated[Function[a,999][a]]/.a->0 returning Function[0, 999][0], it is already too late

It is about scoping. ReplaceAll is about replacements

It is not about ReplaceAll. It is about RuleDelayed, and other constructs which do the renamings

Honestly, I don't see what more is here to discuss, and don't have much time for it either. My comments seem quite clear, to me at least.

@PierreALBARÈDE "The best way to avoid renaming problems is to avoid names altogether, which is entirely possible" - it may be true from the pragmatic point of view, and I usually suggest this myself. But it is still a workaround to avoid using an essentially broken language feature.

@PierreALBARÈDE You seem to be missing the point of this example in my post. It was not supposed to be "minimal" in the sense you mean. It was supposed to be a realistic (if short) example of use, which leads to confusion (fully justified) for uninitiated, and which illustrates the behavior that many, including myself, consider a serious language bug (which might be unfixable, but bug nonetheless). Specifically, the a parameter inside Module should've been renamed during g[fn] function call, had renaming mechanism worked in that case - and this would've been one way to fix this.

@luyuwuli Sure. Waiting for other answers is always the right thing to do.

@PierreALBARÈDE "Replacement occurs before Function evaluates. This is normal and not related to renaming." - there is nothing normal about it from the language design standpoint, because it breaks lexical scoping for Function with named args, and makes it unsafe to pass it as an argument to other functions. But it appears that you didn't read my post carefully. I did not say that this was related to renaming. I said that this was one of the instances where renaming could've fixed the problem, but was not used by the system, causing this bug.

@luyuwuli Actually, I made a mistake - I meant not using $ in the fashion varname$123. I will correct the text.

@Wjx Thanks. Glad you found it useful.

@luyuwuli Thanks for spotting this, I had a value of a in my session and overlooked that. Corrected.

@luyuwuli This sounds correct, but "StrictLexicalScoping" does more than that, and not only for With. I don't really know the details of how it works, the right person to ask is Daniel. However, in some sense, the whole point here is that we don't really want to know these details.

I agree that whenever a question can be legitimately closed, closing is enough and supersedes downvotes, so to speak. I also think that closing tends to not be taken as personally as downvoting (I have never cast a single downvote ever, because I consider them useless, but I realize that this view is rather extreme).

@PlatoManiac Thanks! I hope so too :)

@IstvánZachar Was glad to help. Good luck.

I hope I am not wrong, but it has to be double-checked.

@IstvánZachar It should always do that. I don't think it is documented, but when you add a single key-value pair to an assoc, it can be considered O(1) operation regardless of the size of an assoc. If you add a new assoc with say M entries to an existing one with N entries, then I'd guess it will be O(M) operation, regardless of N (which can be much larger than M).

@IstvánZachar Yes. Dataset is just a wrapper around lists and assocs, which brings some fancy formatting and a query language which some people find more convenient / concise. It does not fundamentally change the underlying data structures.

Ok, sounds good. If I have something, I will reply back either here or below the original question, or simply post an answer there. Have a good time on your leave :)

I don't have anything ready. I might look into it (this topic is of interest for me as well), but I can't promise when, since time is scarce at the moment, so you will be better off not depending on this. Or, we can do it this way: if I don't come up with something by Monday, don't wait for me and try on your own. You can always post specific questions on the site, and if they are short and self-contained, you have good chances for getting help.

But pretty much any single machine - based security scheme will have vulnerabilities. The scheme with Wolfram Cloud is better because your end users don't need the token at all, they simply send a request to the cloud, and it is the cloud code that has that github token and actually sends the request to github. But the cloud code is not accessible for the users and is not executing on their machines. You should be able to use the free account for the cloud, as long as the number of users is not too large and updates are not too frequent, which seems to be the case for you.

Well I guess I misinterpreted your request. If your audience are your collaborators, then I don't understand your collaboration model - i.e. why not give them full access to that repository? You can set things up so that changes are only possible via pull requests. Or do you want them to only have access to part of your codebase? This all sounds more a workflow / github / permissions issue to me. Also, what does this have to do with running your app in Mathematica and (auto) updating it? Your collaborators can presumably check out proper branch, as long as they have access to the repo.

It's hard for me to assess how well your current scheme fits your needs, but it all boils down to the level of security you need. From what you have described so far, your scheme seems fine. The critical part of course is the manipulations with array of tokens. You may protect your own functions, but built-ins like Part and other list-manipulation functions can be easily overloaded in such a way that they will log their input and output, in addition to their normal actions. And as soon as someone gets an array of tokens, it is just a technical matter to try them all and find the working one.

But even with all that, I would still follow the suggestion to have a separate github private account with just that repo. Github's story for read-only access to repo seems to not be particularly promising, despite the huge demand expressed in that thread. But there seem to be some workarounds and ways out (even with the setup I described above, it is better to not give your cloud API a write permission)

One last thought: you will obviously have to think also about protecting your code from being read. You may find this Q/A helpful, although you probably are already aware of most of the suggestions discussed there.

Let me know if these suggestions make sense to you. If they do, then tomorrow I can post a more coherent suggestion / scheme as an answer (it is way too late here now, so calling it a day). That scheme would allow you to completely decouple users from any sensitive information, they will essentially have to register once, and then only send in their HTTP requests their $MachineID (by send I mean that your app will be sending that info on their behalf, of course, fully automatically, without any involvement on their side).

[1/2] I think the entire scheme using github-only, is flawed. You probably want to set up your own server, which would accept end-user HTTP requests (with proper authentication), and will communicate with github - in this way your github repo will be insulated from any prying eyes. If you don't expect too much traffic, you might even be able to use Wolfram Cloud for that, deploying a custom APIFunction and setting up proper permissions. In this way, your secret token will only unlock execution of that API function in the cloud, ...

[2/2] ... and that APIFunction will be responsible for actually checking out the new version of your package from your github repo, encoding it, and returning the response to the WL client, which would decode it and run / install updates. You can set cloud permissions so that the users will have a permission to execute the cloud object / APIFunction (provided that they have a secret token), but not read that CloudObject or modify it - so the code you put there, that will actually communicate with github (using this time a real github token - which of course you don't share), should be safe.

@Roman "you're feeding a troll" - apparently yes, at least in the context of this Q/A. I was confused by their high reputation.

@user64494 What I gave you is an argument. An (Riemann) integral is defined as a limiting procedure. You have two limits to take then. As long as we understand what the notation of delta function means - namely, a specific order of limits, integral first, and regularization last, there is no problem with the validity of the integral. I have spent a big chunk of my life working with delta functions in physics, first studying, then getting and publishing actual results. What I told you was what mattered in practice in my work. And yes, I am familiar with formal theory of generalized functions.

@user64494 You can always use the regularization and the limiting procedure to make sense of these kinds of integrals, understood as a limiting procedure rather than directly as a standard (Riemann) integral. From the practical viewpoint, what matters is the order of the limits, which is what I mentioned in my comment.

@MarinaNebot The limit you mentioned should be taken already after the delta-function is integrated over t in convolution with some other function. It only makes sense when you think of a delta function as a kernel of an integral operator. If you simply need to represent it somehow, why using the regularization and not the standard abstract delta-function notation (or, DiracDelta[] in WL)? Or, if you want to use the regularized version, simply denote it somehow and use that notation. Either way, you don't need to take the limit just for representation purposes, and it would make no sense.