![]() faulttoleranceheaders |
Introduction to writing spiders and agents |
![]() faulttolerancepolicies |
| Fault Tolerance: Be proactive | ||
| 1. | Watch for site layout changes | |
| Look for "Last Modified:" changes in headers | ||
| Look for "Location:" changes in headers | ||
| Write code that looks for anomalies | ||
| Compare content | ||
| Automate the above | ||
| 2. | Assume that people (and machines) write non-compliant HTML | |
| 3. | Assume that HTML comands will span lines of HTML | |
| 3. | Use protocol descripters | |
| http://www.site.com | ||
| https://www.site.com | ||