@curiousdannii How proficient with nodejs are you?

@Caleb I know certain things quite decently and haven't tried others, but can probably learn quite quickly

Have you figured out how to match the verse indexes to queries? Do you know of any other systems with such a thing whose database schema we could borrow? I think that could be the most complex part.

And maybe we could make the end user part a browser extension - easier to install (and maybe easier to keep updated?)

@curiousdannii Yes. That's easy to build from a user-script actually, but the user-script version is a lot easier to develop and debug so that's my first target.

@curiousdannii Other than knowing some Javascript, I don't at all. I'm tempted to go down the python road for the XML parsing and keeping track of data bits while we build the index. How comfortable are you in Python?

@curiousdannii That part is easy. At least in my head.

The table is basically just going to be [ post_id | osis_ref ] with some more columns eventually to make the search results more relevant.

@Caleb I've done some Python, back in the 2.6-7 era. Haven't really touched any 3, and I don't know the packaging system.

But XML should be very easy in JS too.

@Caleb How is the matching going to work though? If the post mentions Matt 5:11-18 and you search for Matt 5:16-25 how will it check if it matches? Does sqlite support stored functions?

@curiousdannii So we need a script that reads an XML file, parses out two fields and passes them on to the next piece. Can you do that?

@curiousdannii The index will have an expanded list of all verses in mentioned ranges (Matt.5.11, Matt.5.12, Matt.5.13, etc.). Search queries will be parsed and expanded the same way (the parser for this already exists) and all we have to do is cross reference the two.

I was going to use SQLite in the devel environment just to store the data in a way that's easy to hand off, but ultimately what I want to do is pass that data off to the Browser's local storage. We'll have very limited query capability, but we don't need much and the data will be local to the user.

@Caleb IndexedDB could be even better. And it's widely supported. But I haven't used it before.

@Caleb Ahh. What if you search for Matt 5-7? Will it still expand to every verse?

@curiousdannii That's what I was going to use. SQLite is just a placeholder while building the index before passing the data off to the browser when installing the user-script/extension, where it would be stored in an IndexedDB.

@curiousdannii Yes. Otherwise you have to iterate over every reference in the index looking for ones that are in the range. More efficient to expand all the items in the range you are searching and return a result set that has any one of them (and the more the merrier). Again all that parsing/expanding ranges is already taken care of.

@Caleb I've just been looking at IndexedDB, and it's so different to SQL. Querying it is weird...

@curiousdannii Ya, it's whacky. I'm going to suggest we wrapper it with DStore in order to get a sane query/filter interface.

I wonder if it might be better to not expand the references though. A function could compare strings to check if it matches and rank the post: exact match > strict subset > partial overlap

If the userscript/extension will never be changing the data, it would probably be faster and simpler to give the dataset as a JSON file.

@curiousdannii I think that would be too slow. Doing that means you have to iterate over the entire data set for every query rather than taking advantage of in being indexed. The ranking bit we can accomplish with another column in the table that tells us whether the expanded reference was a direct reference or indirect as part of an expanded range.

BH only has ~9300 posts - iterating through an array and checking with a function would only take a fraction of a second I think

Anyways, we can experiment and try a few different things

What's the next step? Are the issue numbers in order that they need to be developed?

@curiousdannii Christianity has 28k and Judaism has 47k posts. Factor in many of those posts have loads of references and I think we're going to be dealing with a big enough data set that running any functional code over it is going to take noticeably longer than asking for something that a DB index can return.

@curiousdannii No. And some parts of the system can be developed independently but the next logical step in the pipeline is issue #6. If you can do it go ahead and assign the issue to yourself, fork the repo, get yourself a clone and knock yourself out!

In case anybody cares, there are 315,301 verses referenced in posts on BH (including all verses in whole chapters or ranges mentioned, but counting each verse mentioned only once per post). By the same metrics, Christianity has 509,408 references, Judaism 121,012, Islam 29,245, History 23,285, and Skeptics 52,212.

@curiousdannii and @Caleb I'll contribute if we can get together a to do.

I have done some node.js scripting. Mostly web scraping and DB connection pooling stuff.

The nice thing about it though is if you know jQuery, then you know node.

Some jQuery libraries are portable unmodified. It's just server-side jQuery.

That being said, is that actually the direction y'all want to go.

My concern with scraping the site on a frequent basis with a browser extension is that becomes expensive and SE might get grumpy at the additional server load.

Hmm, those numbers appear somewhat inflated. For example Skeptics posts mentioning CO2 (the gas) are being parsed as "Colossians 2".

Good to know that we win the verse-per-post count. I would hope so! ("Psalm 119" gets more than its fair share of weight, but still.)

The other thing I am wondering about is if we should distinguish from verses in the question vs in the body.

@JamesShewey I hate jQuery. Thankfully jQuery is not JavaScript, and Node is JavaScript, not jQuery.

To me the references in the body are less valuable than those in the question itself.

@JamesShewey Scott's post on Meta discussed that.

@JamesShewey The to-do is a bit loose as we're just sketching out parts and wiring them up. I'm trying to keep issues up to date though. If you see a part you can handle let us know you're working on it and go for it.

@JamesShewey We're processing each post separately, so we can score answers differently that questions.

@JamesShewey We're not scraping anything. SE has full data dumps that we're processing offline to build an index.

Yeah, but my point is that dozens of users repeatedly dumping that dataset is costly - even if they do have an API for it. Doing it once daily and then letting users have access to the formatted data is much cheaper.

Do they only dump as xml files? Or is JSON an option?

(not a huge fan of parsing XML as a general rule)

@JamesShewey No it's not. It's hosted on archive.org, available as a torrent, and end users of what we're making won't even need that dump, they just need the pre-generated index that we build from the dump data.

But Scott looks to have a good start to an algorithm.

Got it.

@JamesShewey It's XML. But I'm already done parsing it to JSON and parsing out verse references. You should check out the Github project.

\o/

I'm excited for this. It's like having a crowd-sourced commentary... It would also be interesting to see a visualization of where analysis is concentrated and where it is missing.

And in case you were wondering there are 31,102 verses in the bible - I assume that is the Luther Canon, but wikipedia didn't specify.

@JamesShewey But given the number of times we've talked about John 1:1 and Gen 1:1, there's surely lots of good stuff in there we're missing.

@JamesShewey That's pretty easy. Visualization can be done later, but I cana tell you right now Isa is far and away the most referenced, with John having more diverse references.

@Susan John.1.1 = 118, Gen.1.1 = 97.

I would expect something like Psalms or Leviticus to have the least coverage...

@JamesShewey Numbers, 1st Chronicles.

Above analyses are on references in questions only by the way.

@Caleb - Nice. I'll have to see if I can chip-in later tonight. For today, I must code for the establishment for my daily bread.

@Caleb I guess not surprising, but that's kind of crazy about Chronicles in particular -- as if there were nothing to talk about! Wait, you're not counting each of the minor prophets separately and saying Chronicles still has fewer, are you?

@Susan Yes. I'm eyeballing a raw list of individual references. I didn't group them by book yet. I just looked at what book names showed up very low in the hits.

@Caleb Regarding "but counting each verse mentioned only once per post," I did just update my meta regarding that, as it would be ideal to have a "count" of how often a verse is mentioned within a post (and the set of posts related to the question) for ranking purposes. But I do not know if that adds a whole level of complexity best left untouched at present.

@ScottS No, that's an easy add. Could you add thing like that as issues though? That way I can track what code is related to each feature request.

At the moment I'm roughing out a proof of concept from start to finish of the pipeline so most features won't be implemented, but it's nice to have a a list of what they are so we make good choices of technologies to use along the way and coding structures so that it can be expanded to cope with how people expect to use it.

Grrrr. Why do site do stupid things like make Matt. 1:23 have a URL like http://www.biblestudytools.com/nrs/matthew/1-23.html? Hyphen is not an appropriate separator for number pairs that are not ranges.

The numbers above might be highly inflated!

@Caleb - because using a colon would violate the RFC and apparently that programmer never heard of URL encoding.

@JamesShewey Or underscores? Or slashes? Or dots? Anything but hyphens man!

Corner cases: bane of my existence.

Anybody have any brilliant ideas for why Isaiah 1 might be getting parsed a lot where it doesn't exist? For example this post is showing 10 hits but I don't see it anywhere.

This: "Is 1" thing that might do it...

@JamesShewey All but one instance of that are to "Genesis 1..." which should parse (thanks to greedy regex) to Gen.1... meanwhile the only real "Is 1" hit in the post is actually to "Is 13:10" and (again thanks to greedy regex) should parse to that.

sounds like maybe you are matching on the is 1 in Genisis 1?

Is your regex case insensitive?

@JamesShewey Yes it is case insensitive. But it's also greedy and tries to match long book names first and also checks that matches start at word boundaries, so that shouldn't be possible.

I think I'm done for the moment. As documented the current output is a flat text file per-site with verse references and URLs which you can grep and sort to your hearts content.

For anybody that can't git clone and run a few commands but still wants to see the data, you can download the text files I've generated to date from here.