Simple collection of crawlers # 1. I only made a function to collect URLs. If you need to store the data into the database, you can write the processing code in the parseData function. # 2. Requires sqlite3 or pysqlite support # 3. Can run on DreamHost.com space # 4. User-Agent can be modified to pretend to be a search engine spider # 5. You can set the pause time and control the collection speed.
