![]() faulttoleranceproactive |
Introduction to writing spiders and agents |
![]() legalstuff |
| Fault Tolerance: Observe site policies | ||
| 1. | Stated usage policies | |
| 2. | Meta tags | |
| <HEAD> | ||
| <META NAME="ROBOTS" CONTENT="NOINDEX"> | ||
| <HEAD> | ||
| 3. | Machine readable policies referenced in headers | |
![]() |
||