![]() faulttolerancepolicies |
Introduction to writing spiders and agents |
![]() dealingwithdata |
| Advice and warnings | ||
| 1. | Observe other people's intellectual property rights | |
| 2. | Adhere to web site's usage policies | |
| They can block your IP address forever | ||
| Don't spider eBay or Yahoo! | ||
| 3. | DNS Registries only allow a limited number of requests from a single IP address in a given day | |
| (or so I'm told) | ||
| 4. | Be responsible... | |
| Don't write spiders that collect email addresses for spam mailers | ||
| Don't write DOS (Denial of Service) agents | ||