Spiders

Wulfsta
Thu, 02 Dec 2021 05:13:51 +0000

Is anyone here familiar with crawling the web? I’m interested in broad crawling, rather than focusing on particular sites. I’d appreciate pretty much any information about how this is usually done, and things to watch out for if attempting it.

Comments URL: https://tildes.net/~comp/zg0/spiders

Votes: 10

Comments: 5