
update from:Basic search engine
I want to process multiple pages at once pulling url's from tier_list in the crawl_web method. The problem I am having is crawl_web already has 2 threads to crawl and index simultaneously, the .join() seems to prevent opening more than one page.
import urllib2, c...