Thanks for joining me here!

I put up a link for Szabolcs as well, hopefully he would arrive soon.

No problem. I probably can't stay here for very long though.

So, why did you decide to not go the same road as I did with $Packages?

I really don't remember, perhaps I wanted to collect only those files that were not loaded already when Mathematica starts.

Anyway, I might just go ahead with my proposal, as Szabolcs would be able to read the transcipt back, wouldn't he?

So we don't sit here wasting time.

@IstvánZachar I see. Re: proposal - sure, go ahead, he should be able to read it

I was busy developing domain-specific buttons for symbols and packages: a SymbolButton offers all sorts of functionality (as an ActionMenu) for symbols (like clear, remove, attributes, etc. somewhat like MATLAB's workspace), and PackageButton offers specific functions for packages and contexts.

Hi Szabolcs!

just a min

I hope you don't mind that I have already started explaining my proposal.

no no

I was planning to listen to this WRI workshop today and I totally forgot ...

Hi @Szabolcs!

The idea is then that these buttons for symbols and packages can be used all over different GUI-s to explore symbols, contexts, etc. For example, I now have a ContextExplorer to explore symbols of a context, PackageExplorer to explore packages of a directory and DefinitionExplorer that is basically the same as what Spelunk does.

OK, "user limit exceeded". Never mind

Hi @Leonid and @István

@IstvánZachar Do you know about my plans for the pure FE-based IDE? I have a 0.0.1 version which I am currently using, and I planned something like you are describing as a one of the plugins there.

@IstvánZachar Once I am done with the plugin system, your stuff could perhaps be integrated in and written as a plugin, without that many changes w.r.t. what you currently have

No, i didn't know about that, sounds great! It seems like we are doing a lot of work in parallel, which is a pity : )

@IstvánZachar I don't think there is much duplication. I was more focused on the core IDE, and these explorers I only had in mind, and have very very basic versions for them.

@IstvánZachar The whole idea is that once I get the plugin system ready, anyone would be able to write plugins, so this will become a community effort, hopefully.

A better venue for collaboration would really be useful.

The key missing part is learning about all these projects.

Agreed. And I don't want to flood the Spelunking thread, as my work is only partly related to that.

@Szabolcs One of the plugins for this IDE would be a code exchange / collab system.

But don't want to start three different threads either...

@Szabolcs Yes, I agree, we need some dedicated channel for that.

Not sure if W Community would be good ... after all it's a forum where anything can be posted.

@Szabolcs, @IstvánZachar dedicated chat room might now be good enough.

@Szabolcs I think this is one of the worst places for that. We have to be maximally independent of WRI in these efforts

Leonid, do you see a date in near future when your IDE code will go v.1.0?

@IstvánZachar The biggest problem now is that I am busy as hell, and it is not clear when the situation would change. I might, though, get some chunk of free time enough to finish the plugin system, relatively soon.

@IstvánZachar But it's really hard to say, the workload on all of us is completely insane these days.

@Szabolcs I would think that for now, we can bend the SE rules a bit, and make a meta-thread. I do have a better plan for the later, but for the next couple of months this might cut it.

We have meta thread for other stuff that didn't fit elsewhere, e.g. the package list

@LeonidShifrin I completely understand you on that. I've stolen two weeks from my official work to get to this point. I'm only asking about the date because of what to do with my existing code. So what you two suggest is to make a meta thread where we can list the individual pieces as github links?

@IstvánZachar Re: IDE - well, an increased interest in this certainly will work as an additional factor to make me finish this faster. I am using the current version all the time for my work these days, don't know what I'd do without it - so, it is already usable on a very minimal level. But the real thing we need is a well-designed plugin system

@Szabolcs I think that thread is not good for us. We need another one, specifically devoted to some projects currently being developed by our community.

@LeonidShifrin yes, what I meant is that since that thread exists, it's okay to start a new one

@IstvánZachar Yes, and I think it can also work rather nicely in giving the context around these projects. I am sure it will work fine for us, for the time being.

@Szabolcs Oh, I see.

@LeonidShifrin So, would you upload the project in the current state if a new thread were started for it?

@Szabolcs The reason I hesitated to announce that was that I wanted to have some quite time to design the clean core, so I wanted private beta-testing. I didn't want the pressure from the community until the core is done (to the first approximation). I probably can upload it / create a GitHub repo for it, with a warning that it is under construction and things will change a lot. What I don't want is for people to create early forks of this, since it has not yet been developed to the state where

@Szabolcs I feel it appropriate.

Making it public would also put a lot of pressure on you --- lots of comments and user requests

I have to admit I haven't used the one you sent me much

has it changed since then?

Considering the pressure: can a meta-thread be made restricted to users above 2K or something similar?

No.

Well, I don't think so.

I haven

Yeah, that would be too anti-democratic and elitist in a community-driven site : )

I haven't done much package writing recently, so I didn't really use any IDEs that much. I mostly just used notebooks recently.

@Szabolcs Re: pressure - yes, but when the core / plugin system is done, it will be all right. Re: changes - no, it has not. But I am using this heavily every day, for a couple of months now. Perhaps, it is also a question of understanding the workflow - right now, I benefit mostly from tab interface / quick window resizing / ability to re-run a single function definition inside a package easily.

Hi @rm-rf!

@IstvánZachar Getting back to safeGet, it'd be quite useful if you put it in an independent package, e.g. on GitHub.

@IstvánZachar Hi!

@Szabolcs Re: notebooks - but my current IDE (if it can be called that), is currently nothing more than a notebook manager, more convenient (for me), than a plain FE. Did you actually give it a try - I mean, try different buttons, see how tabs work, how you can resize windows, etc? I actually work completely with MIDE these days, rather than the plain FE. It has a number of bugs, but I still find it much more convenient.

Hi @rm-rf!

@LeonidShifrin Yes, a little bit. Have you changed anything since then?

@Szabolcs No. I wish I had the time for this. It won't even take that long. But no luck.

@LeonidShifrin I think bitbucket allows you to have a free private repo and allow specific users to clone the repo. That might be a good way to do a private beta for MIDI. Then you can just tell users to pull and you're all set.

@LeonidShifrin What OS do you use it on?

@Szabolcs Well, I could do that, but the problem is twofold. I really need all of your insight on this. First, I know nothing about github, and once I accidentally deleted my local repo when playing with it. Bad start. Second: now all these functions (package-, context-, symbol- and definition-level functions) are dependent on each other, for example the modified Spelunking code uses specific package and symbol buttons, so the whole thing seems to nicely interlock.

@rm-rf Good idea. Also, Bitbucket has a Wiki, which is also good.

@IstvánZachar The inter-dependency problem you mention certainly sounds like a design problem to me

@IstvánZachar You can create a "Gallery chat room" and only allow certain users to talk (but all can read). That's about the most you can do with restricting access on SE.

@Szabolcs Mac OS X

@IstvánZachar You mean it would be useful to have them in the same package because they share some code for button formatting?

@LeonidShifrin Github does too. For the most part, they're similar, but I love the free (unlimited) private repo hosting on bitbucket and have all my projects there. The only problem is that Github has a bigger social audience, so if you're aiming for a wide reach, then it's better to put the public package on Github (although, for mma, it's terribly small on both sites)

@IstvánZachar, @Szabolcs Guys, that's why I have been thinking about the fine-grained (gist - based) code-sharing system. The granularity of a package is very often too coarse.

@rm-rf Github also has gists with the web API, which is a big plus for what I was thinking of

@rm-rf But yes, private repos are nice

@Szabolcs, @IstvánZachar, @rm-rf We might set up our project thread still, and for some of them which are private beta, we might just list the current status, without download links - this might still be a good start.

@Szabolcs Yes, exactly. This can be factored out a bit more not to be too interdependent. The design-problem is actually in the originak Spelunking.m: when I tried to include some functionality into its interface, it could not handle dynamic buttons at all and had other problems that I had to iron out.

@IstvánZachar I've been using my OO package to considerably simplify the factoring of dynamic code. Perhaps we might talk about it some time soon, if you'd like

@LeonidShifrin Sure, that would be useful, can I download your OO from your site? (or is it private)

@LeonidShifrin Adoption of packages would help if you also placed it on github and provided a nice README. Github allows you to convert the README into a simple and nice website (or you can design your own) and that can serve as a good place to read and download the package.

@IstvánZachar It is here

@rm-rf It's already there (OO), for a long time, but as a gist though.

Sorry for going slightly off topic, I'm also trying to keep an eye on the ongoing presentation

@LeonidShifrin Gists are primarily used for snippets and the layout of the site is designed around that, so most people won't look at it much. Take the OO gist, for instance... the readme is at the bottom of the page! Whereas, a proper repo will show the readme on the homepage of the package and you can also convert it into a nice website. I understand your reasons for using gists, but there is a human & social aspect to it too :)

And they mentioned StructudedArray

I've never seen this before. Has anyone here used it?

@Szabolcs, @IstvánZachar, @rm-rf One thing I take home from today is that I will try to push this IDE stuff a.s.a.p. The reason I believe this is important is that it can unify all our tools as plugins, under a single unifying (and eventually also powerful / convenient) interface

@Szabolcs Nope. No hits on the site either... looks like it would be useful.

@rm-rf Yes, I agree with this. In fact, gists in my plans are intended for a slightly different purpose - to allow M users to share / publish small pieces of code, on a very small granularity. I think both approaches are useful under different circumstances. OO should be a full-fledged repo, that's for sure.

I still have a pending question here, which has to be answered if I were to take home anything :) How should I publish this package? It's clearly more than Spelunking, but it uses a modified version of it. Breaking the package to pieces for posting in different, specific threads seems counterproductive, as - while major functions are independent - interfaces depend on shared functions and gui elements.

@Szabolcs This has been used in tensors functionality, IIRC

@IstvánZachar Only you know this code at the moment, so if you decide to merge it together with the existing Spelunking package, you'd have to maintain it. Which is okay. The only reason I have the Spelunking repo on github is that I put it there first. It was written by Simon Woods, Leonid and you.

@IstvánZachar I can give you commit rights to the repo.

@IstvánZachar I think there are several problems here. If I had my fine-grained system in place, it would have a "compile" option that would take smaller-granularity pieces and combine them together into a single package. But in any case, the final form of the project / package has to be a complete thing, and might need some build step from the pieces you have.

Well, that is the other side of my question: would it be reasonable to update the Spelunking code AND bundle further package files with it?

@IstvánZachar I think yes.

@IstvánZachar Yes, I think so. Whatever you do internally, is your business, but for the user, it is always good to give a complete thing - the user could not care less how it was built, just needs an API from you.

It would be useful to have this set up now. When there's a better infrastructure in the future, it can be transitioned. But it's just a useful tool, so let's make it available.

@Szabolcs @IstvánZachar So then, why don't we set up a project thread (single thread for many projects) on Meta, for the time being?

@IstvánZachar Personally, I feel that it's useful to not go off-topic for a package... I mean, if you start a spelunking repo, then adding a plot-explorer to it would not be meaningful (even if the package is useful). Your package/symbol explorer could fit in though.

@Szabolcs Ok, thanks for the heads up, I will delve into git then to understand its workings, and do the updating with as little impact on the originakl as possible.

@rm-rf I suggested to include it in Spelunking because it's related, and I needed this functionality before when trying to find out what's going in OpenCLLink.

I mean trying to find what file a function is defined in.

hi @halirutan

@Szabolcs Oh, I agree... I thought Istvan's question was generic. I probably missed part of your discussion.

That's how we ended up here actually :)

@rm-rf Sure, I agree with you. The reason I bundled these functions (package, symbol and definition explorer) with each other is only because they are used most during development and introspection. Certainly they can be made more independent, but think of my SymbolButton as PasteButton: it is used throughout the interface all the time.

@IstvánZachar While this isn't a detailed intro to git, parts of my answer here might be useful:

@IstvánZachar Would you be willing to just integrate the new functions into Spelunking then? If you have technical questions about GitHub just email me. Given that you contributed to Spelunking recently, I think you'd be a better maintainer for it.

@rm-rf @IstvánZachar @rm-rf Guys, I think the problem being discussed is very deep and has to do with the absence of fine-grained module system. I do have the solution for that. It worked when I tried it last time. It is a custom loader with a different, much more light-weight module system.

@IstvánZachar Would you like to try it? I can set up the repo for it, with some examples. It might solve your project - coupling problem.

Hi @halirutan! How are you?

@Szabolcs Actually, I would rather add further package files to spelunking, to maintain their partial independency. I really don't want to put everything in one huge file, and also, they can work on their own quite nicely.

@Szabolcs @LeonidShifrin Hi. I'm fine. I have much PC time these days :-)

@LeonidShifrin That is a definite YES!

@halirutan Just like I suggested? Good! I hope that helps!

@IstvánZachar Then probably it would be good to keep them as separate packages. This also reduces the barrier to contribute to either of them. Smaller packages are less scary.

@Szabolcs Now you are talking! This is exactly the fine-grained vs. larger package thing.

@Szabolcs Yes, that is what I had in mind.

Hi @halirutan!

Ok, guys, I can construct a Github repo with this custom loader, and show the large data framework example, broken into small pieces, using that loader. Is anybody interested? It's an old code of mine, it works, it was supposed to be a part of a larger thing, but why don't we start trying to use it right now? @IstvánZachar seems to have a clear case where it may be useful for him.

Basically, I am talking about an alternative module system, much more light-weight than packages, with automatic namespace handling.

@LeonidShifrin or @halirutan Perhaps one of you can write a self-answered question on rules of thumb/good practices in writing large multi-file applications. There are some intricacies in the Get order, how to read in symbols from one sub-package to another (such as BeginPackage[{foo`, bar`}] vs. Needs[...] + AppendTo[$ContextPath,...]) . Each has its own purposes and it's useful to have this in a self-contained post.

@rm-rf Thanks, this will come handy!

You don't have to indicate the package name, it is inferred by the loader from the directory structure. And any new file can be just dropped in and used.

@rm-rf I could do that, but I think Istvan's case is a very clear case to try the light-weight module system I mentioned. It works already, I just need to put in on Github. It was supposed to be a part of the gist-related stuff, but can be used independently. The only problem is, the one who wants to use it needs the loader project installed - but it can be a part of the installation, and installed by a project installer, for instance.

@LeonidShifrin Please do so Leonid. Without seeing code, I usually only have a cloudy understanding of what it does.

@IstvánZachar Ok, will do right away ( I do have a bit of time left). Meanwhile, why don't we set up a meta thread for community projects, for now?

@LeonidShifrin I can write that post, if we agree on its form: each answer should be a single project; should contain a link for the project's site (github, specific SE thread, etc.) with a short description and perhaps an image?

I'll need to be going soon.

@IstvánZachar I would think so.

@IstvánZachar If it is a private beta / vaporware, might lack the links.

Ok, agreed then. Thanks all of you for the discussion, let's see how much spare time we'll have in the near future :)

@IstvánZachar @Szabolcs @rm-rf @halirutan All right, see you all soon then! @IstvánZachar Do keep an eye on this room, once I am done with this loader stuff, I will ping you. I do have a notebook with all code illustrated by examples, could sent it to you by email once I test it. Then, will place that stuff on Github, and let you know.

see you !

bye

@LeonidShifrin Am looking forward to that!

See you all

@Szabolcs I missed the start of the discussion. Your first message.. was there anything before that?

@halirutan We only started with Leonid a minute before Szabolcs has arrived, and the only thing detailed was the structure of my package to doing introspection/symbol/context exploration.

@IstvánZachar Ah, ok. I have to read through the discussion. Thanks

@halirutan The comment thread here might provide some context.

@IstvánZachar Ok, I have created a Github repo, where so far I have only put the zipped folder with the notebook with code / examples, and two sample projects for tests. The projects are my large data framework, split in two different ways. In the notebook, most functions are illustrated by examples, but you can also just skip those and run the code, and then only look at the final examples in the Test section.

@IstvánZachar It is however instructive to go through per-function examples, to see how this stuff works. Anyway, the repo is here, and the direct link to download the zipped folder is here. Have a look at how the projects LargeData and LargeDataAlt have been split.

@IstvánZachar WIthin the next few days, I will push the package version of this stuff, and add some instructions on readme on how it is used. In the mean time, you can see how it is used by looking at the "Test" section in the notebook. Note that the sample projects should be in the same folder with the code notebook, for that test code to work (they will be, when you unzip the main folder). Let me know if you have questions.

@IstvánZachar One last thing here: while for development, this fine-grained structure is very beneficial, I will add an option to "compile" the project into a standard package using standard namespacing / loading mechanism (Get/Needs, etc) soon, so deployment for those who don't have the loader runtime installed, won't be a problem

@IstvánZachar Note also the absence of any explicit namespace declarations in the split files in the sample projects. The namespaces / context names are assigned to them automatically, based on the name of the file (somewhat similar to Matlab). You can export more than one symbol, in which case you have to create a folder rather than a file, and the folder should also contain a dummy ExportedSymbols.m file, which is completely empty, but just hints that there are non-trivial symbol exports here

@IstvánZachar Imports are implicit (something I will change to also have them explicit) - during the dependency - resolution stage, all files from the same directory are considered, so as long as you declare symbols there as exported, they will be automatically imported when used in other files within the same directory. The dependency - resolution / import part should be further elaborated though.

@IstvánZachar Ok, that should be enough for an intro. One thing we could do some time soon is that you post your code somewhere (Github, bitbucket), and then we can work out the version of it which would be using this fine-grained system. Then you'll see if this would bring any benefits to you (which I believe it would), and I will have a chance to get a feedback and perhaps make some improvements to this stuff

@IstvánZachar One really cool feature of this module system is that you can run several versions of the same project without any conflicts, since the system will automatically assign different context names for the files in the projects, simply based on their location / top-level project folder / name.

@LeonidShifrin Thanks for the details, I will go through the code with great care. I also will try to finalize my packages of the workspace utilities and put it on github, possible as an extension of the spelunking package. I cannot promise hard dates, but will keep you updated and (more probably) will ask tons of questions.

@IstvánZachar Ok, sounds great. I am myself quite short of time these days, so I won't have lots of time to work on this, but it's good to get this started, and I will certainly answer any questions.