@Caleb My 'port' to pure JS is now doing basic searching. Where's the best place to discuss, here, Meta, or Github?

@curiousdannii I'm on mobile right now but a search prototype sounds great, thanks for coding.

Lets keep as much discussion as we can in issues. I think that will be most conducive to new hands if they ever want to pitch in. That being said some things are just easier in chat so maybe a dedicated SE room is in order.

room topic changed to Stack Verse Mapper development: Discussion related to github.com/alerque/stack-verse-mapper (no tags)

room topic changed to Stack Verse Mapper: Discussion related to github.com/alerque/stack-verse-mapper whether development or usage. (no tags)

← 82 messages moved from The Library

@curiousdannii Some of that look great and I'm definitely going to integrate it. Your XML handling looks more robust than my initial stab at it.

On the other hand I'm not sure what is to be gained my trying to make a monolithic JS app out of this.

For example not using make looses a lot of ground. It was providing us with nice logic for when not to do things. For example it only does the downloads if the upstream data dump has been updated. It could resume partial downloads if interrupted. It would only extract again if the dump was changed. It would only rebuild the index if the extracted dump data was updated. And so on.

If we're going to take this through to a browser extension build I don't think it makes sense to mix the logic of pulling all the pieces together into the same code as does the other lifting.

Also I'm going to be using the data generated for another project and keeping all the pieces modular so they can be used independently of each-other is going to make it a lot easier to re-use.

Tests are good though, thanks for starting in on that early!

@Caleb That's a fair question. I really just went with what I am comfortable with and can understand, which isn't make. It can be a very confusing format, and I can't follow the dependencies of tasks even in your rather short makefile. And if ant was needed because make couldn't do it all, then I'd prefer one system that can. I hadn't used gulp before yesterday, but I already quite like it.

But it's early days. Let's use what works out best, or maybe both can be used in parallel.

I didn't realise your download script did partial downloads or checks for updates. How does it do that, using ETags or last-modified or something? That is something I would like to do for mine too.

Also smarter rebuilds too. And making 7z only unzip the Posts.xml file, which should cut down the unzipping time by a fair amount

@Caleb I didn't really mean to start doing proper tests... I should've named the task "demo" instead :P

@Caleb What do you mean?

@curiousdannii That should've been that I can't follow the dependencies of tasks

Have I mentioned yet that this is a very cool initiative/development? No? Well, it's high time I did! Thanks, all, for the energy you're giving this. Will be great when it's done, I'm confident! :D

@curiousdannii What? My makefile is confusing? Never! Blasphemy!

In all honesty I don't blame you a bit, it's kind of black magic.

Dark arts at least. But to my knowledge nothing else really even comes close to the things it offers. It can tie disparate systems together in a way that just isn't easy to script or ones own. Not what you want to write software in but for getting software put together with different technologies involved its hard to beat.

@curiousdannii I mentioned ant because there are some prebuilt routines for it that take a userscript and build browser extensions for multiple different browsers.

@curiousdannii I left the heavy lifting there to curl. Pass it the right flags and it just takes care of it. But I had to operate on the correct URL, archive.org is passing the actual download off on mirrors so my first request just figures out the final target URL and whether its a valid site, etc. Then the second one can do proper resuming, and as a bonus when doing resuming it also figures out if it's even the same file so will know when an update is posted.

@curiousdannii I mean the code you run to use the system, the code you run to prepare the data, and the build logic that takes care of taking the source code and data and preparing the final package that people will use shouldn't be in the same code.

@Caleb Well I wasn't suggesting that the end user code should be part of the build system, but I don't really see why the code to prepare the data and build the packages need to be separate. They could be separate, but they could be brought together as two tasks in a task runner - surely you would have a make target for building the package?

@Caleb I looked at all the flags and can't see how it would know to use last-modified info... Also you should standardise between -I and --head