@VitaliyKaurov Maybe

arrayCrop[a_, b_] :=
 Extract[a,
  Span @@@ MinMax /@
    Transpose@Position[a, Except@b, {-1}, Heads -> False]]

Has anyone built a spider using mathematica?

@William community.wolfram.com/groups/-/m/t/910902

That was my second version where I tried to speed it up with multiple asynchronous URLSubmit's. Asynchronous stuff tends to crash the kernel though after a while for me if I try anything more than the most basic "keep the UI responsive after clicking a button" use case.

The sequential one was more stable and simpler. Just replace the main loop with:

While[Length@toVisit > 0,
 current = First@toVisit;
 AppendTo[visited, current];
 toVisit = Rest@toVisit;
 response = URLRead[current, Interactive -> False];
 If[response["StatusCode"] == 200,
  html = response["Body"];
  links =
   ImportString[html, {"HTML", "Hyperlinks"}] //
     Map[Quiet@resolveURL[current, #] &] // Cases[_String];
  toVisit = Join[
      toVisit,
      links //
        Map@URLBuild(*clean up trailing /'s*)//
       Select[
        URLParse[#, "Domain"] == currentDomain &&

Couldn't use Import and it's automatic URL resolving of relative URLs because I needed the Interactive -> False option from URLRead/Submit.

Since the asynchronous one wasn't very stable I did a simple version in Python Scrapy. Due to server throttling though it didn't turn out to be any faster really.

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import Spider
from scrapy.shell import inspect_response
import csv

class CollegesSpider(Spider):
    name = 'colleges'
    id = 190150
    allowed_domains = ['columbia.edu']
    start_urls = ['http://www.columbia.edu/']
    link_extractor = LinkExtractor(allow_domains=allowed_domains)

    def parse(self, response):
        file = open("results columbia.csv","ab")
        csv.writer(file).writerow([self.id, response.url, response.text.encode("utf-8")])

I did think it was cool how the code for the Mathematica version "from scratch" wasn't much longer than using the Scrapy framework. Although for more advanced cases some of the features they've built in Scrapy would save some more time I'm sure.

In many cases though I think it would be easier to modify the Mathematica code to get exactly the behavior you want. I'm faster at writing logic to examine the URLs or do different things depending on the page content in Wolfram than Python.

@VitaliyKaurov Yeah, if I capitalize it and add in the default 0 case then it works as is on all of the tests in your posts. For the 5D case it is 2-3x faster, and the code is shorter so I'd say it's a good trade. Using Extract and Position just seemed like a more direct way of doing what you were doing with SparseArray.

ArrayCrop[a_, b_] :=
 Extract[a,
  Span @@@ MinMax /@
    Transpose@Position[a, Except@b, {-1}, Heads -> False]]

ArrayCrop[a_] := ArrayCrop[a, 0]

If you really needed it, you could get a little faster by not building the list of positions (just keeping track of max/min position in each dimension as you scan them), then being a little clever about keeping track of how deep you've gone in each dimension as you scan so you don't look at the same position twice, and then parallelizing that, but I would only start adding those complexities for performance if I ran into cases where I needed it.

Sorry, just woke up. I didn't mean to say you look at some positions twice right now. I meant to say that with some extra work you could do the scan without looking at some positions ever.

@MichaelHale this actually very cool solution. Could you post it as a comment on my Community post?

@VitaliyKaurov Sure thing. Done.

@MichaelHale Perfect, thank you! @MichaelHale

@MichaelHale It never stops running for me and doesn't populate the variables but I will continue messing with it. Did you ever set up a way to index the pages so they can be searched? Would indexing pages for search be an appropriate question for the site. I would guess not but thought I should ask.

@William The asynchronous one I have to abort and then restart because for some reason the first URLSubmit never returns. I can't remember if that problem happens with the synchronous version. I thought it didn't. I didn't work any on indexing. We put the project on hold shortly after I made that because it was going to take too long.

@MichaelHale thank you

Would an indexing question be acceptable for the site

@MichaelHale

I'm thinking it could be pretty simple like this

{"first block of text with random content","different block of text","1 2 3 4"}

now pick the block of text that is closest to "content random with"

So I asked a simplish question here mathematica.stackexchange.com/questions/125013/…