Pokeycrawl Spider pseudocode

From __init__():

Spider
  Crawler.__init__
  Spider.__init__
    Set up: cached_ips,history,exclude,to_do,HTMLParser,tester,ip,cookie jars
  Spider.execute
    While (runs until stop.set() is run in any process)
      get next URL
        if urls are in to_do, choose one randomly: otherwise return arg url
      make the request
      process the response
      add the visited URL to Spider.history
        Spider.get_links
          extract links from html body for successful requests
          if link not in Spider.history/exclude/to_do
            append unique values to Spider.to_do
        stat collection
      Every 3rd iteration:
        # Send data from the stat_queue and flush the cache
        Crawler.s_queue_next
        Crawler._clear_stat_cache
    Spider.clean_up
    Crawler.s_queue_next = None (Signal execution completion)

To do:

Capture form data (in HTMLParser response parsing) and send it to the form_q for processing by the form crawler.

Need to build out the FormCrawler.__init__() method, this may not require a separate execution phase since it should just wait for data to appear in the form_q.