Hello all,

I’m writing these speaker notes because I know how frustrating it is for some one other than the speaker to make sense of a session by simply looking at a set of slides.    

If you were at this session, these notes should refresh your memory.  And it you were not at the session, these should give you a rough idea of what was covered.  You may want to use these notes in conjunction with the copy of the presentation, which is also on this CD.

These are very preliminary notes that I’m using to prepare my presentation, which is still a few weeks away.  While the data on this DEF CON X CD ROM reflects what I plan to present, it will not be exactly like the final presentation.  So please remember, these are only rough notes.

Sorry about the quality of the thumbnails, but I had to make this a relatively small file.

Introduction to writing spiders and web agents 
You can have a lot of fun with the Internet by ditching your browser in favor of writing special purpose programs that look for -- or do -- very specific things on the Internet. This session will equip you with techniques to extract and interact with data from web sites without a browser, parse and filter data, follow links, deal with encryption and passwords, and manage terabytes of information. You'll also learn why writing these programs is a useful activity, and walk away with ideas and abilities to write useful spiders or web agents of your own design. 
Michael Schrenk is a freelance Internet developer, instructor and writer. Much of his consulting business revolves around the creation of spiders, which search the Internet for information of value to his clients. He has also developed web strategies and online applications for Disney, Adidas, Nike and many others. 	
Figure 1 Description of session as stated on def con web site:

Thanks,

Mike Schrenk
July 9-10, 2002

My name is Michael Schrenk

I’m a freelance Internet strategist, developer, writer and adjunct instructor, who’s based in Minneapolis.

Much of my time is spent writing spiders and other web agents for clients.

I have created strategy and web applications for a variety of companies including; Adidas, Callaway Golf, Nike, Disney, Medtronic--and many others.		
The scripts used in this presentation were developed on a Windows ME platform, and should run fine on any similar system.

They may have to be tweaked to run on Win 2000/XP or Unix-like systems.

It was never my intent to provide a bunch of programs that could be down loaded and run without modification.

My goal is to provide you with enough knowledge and enough enthusiasm to take what I’m providing to write your own programs of your own design.

If you want to take a closer look at any of the scripts, reference the HTML presentation on the defconX conference CD ROM.

While we’re talking about platforms, I’d like to make a strong argument for running programs like this in Linux.  I have one particular client who ran about six to twelve spiders simultaneously on a Windows ME machine--largely because that’s all that was available.  	

This computer froze daily.  I converted the computer to a dual-boot Windows/Linux computer, tweaked the code only a little to run in a Unix-like environment, and the system has been running around the clock for many months now.  The only time it stumbles is when they have network problems.  Even then, the programs keep running.	
Once I got the nod to do this session, I put a lot of thought into what exactly I’d do with my time.  I only have 50 minutes, so instead of teaching a comprehensive class, I’m leaning more on teaching you what you need to know to explore this subject further.  The one thing I want you all to know is that these programs are a gas to write because once they’re launched, they can take on an almost organic nature, since their behavior is based almost entirely on information they happen to find on the Net.

So, I have set the following goals for this presentation about spiders and agents:

Session Goals:	

   1.	
Show relationship between agents and hacking & security	

   2.	
Provide basic background information	

	
Provide brief introduction to client/server protocols	

	
How to download pages	

	
Tips on parsing data	

	
How to handle password requirements and encryption	

	

	

   3.	
Convince you that writing these programs is a useful activity	

   4.	
Give you ideas and abilities to write useful spiders or web agents of your own design.	

 This topic is maybe a little different than other sessions here at DEF CON because it deals more with the classic definition of hacking.

The original definition of hacking is to tear a technology apart until you understand it.  And then reassemble it to perform tasks that were never originally contemplated.

It may be similar to the session I attended at DEF CON 5, where a guy built a rail gun out of root beer jugs and parts from a hardware store.

There are also more traditional security uses for agents and spiders in monitor in networks, access and error logs, and notifying administrators when something’s up		
What do agents and spiders do?

Simply put they:
1.	Download data
2.	Parse data
3.	Perform task with data		
We should probable start simply and define what an agent actually is.

An agent is simply any program that interfaces (or acts as a client)  with a server.

A web browser is a very general purpose agent. That people use to interact with web, email, gopher, and FTP servers.		

		
The client  / server environment works like this:  A client send a request to a server.  The server sends a response.

In the case of  the Internet, this usually means that a browser sends a URL to a  web server, which send an HTML file or graphic image back to the  browser.

A special set of HTTP HEADERS (not HTML <head></head>) is also sent, and we’ll talk about the importance of headers later.		
Once the HTML file is loaded into the browser, there may be other files that are needed to render the entire web page.  

In the case of the DEF CON home page there are nine separate images that need to be identifies and requested from the server, in the same way the original HTML file was requested.

There may be other files that are also required by the browser, including: CSS files, JavaScript files, include files,, Flash files, applets, and others.  Frames also require additional file transfers. 		
Types of agents include:

Spiders, which move from page to page depending on the links it finds.

Bots and worms, which physically move to a host system, perform some task, and then move or copy themselves to another host system,   The code red virus and Morris Worm fit this description.

Hybrid agents use multiple protocols and/or interface with multiple programs, like the “I LUV YOU” virus or Internet Explorer.  The “I LUV YOU” virus fits this description because it was transmitted via email and then used Windows Host  Scripting to destroy JPGs and replicate itself through Outlook and Exchange.		
Why write a spider or agent?    

1.	To automatically surf the web for you    
2.	To filter information    

3.	To download and repackage information for non-web devices         PDAs, Printers, FAX machines    

4.	 Security auditing    

5.	Speed.  Remember all those images requested and downloaded earlier?  You can avoid a lot of over-head with a specific-purpose agent. 		

Spider / agent ideas

Remember, one of my goals is to get you excited about writing agents and spiders on your own, so here are a few ideas to get you started.

Run a spider every 24-hours to receive notification when a Jack Russell Terrier is available for adoption at your local animal shelter.		
Spider / agent ideas

Automatically send an email to your mother every time there's a story about you on CNN's web site.            

 Automatically send a fax to your real estate agent when a house appears in your area at your price.             

Look for unauthorized servers on a company network		

Spider / agent ideas

Study things that browsers ignore             

Look for secret messages, which are encrypted in images             

Determine the web technologies used by each of the Fortune 500 companies, and receive an email when any of them change servers.		
Spider / agent ideas

Automatically compare the data on multiple web sites             

Scout all the top employment web sites for jobs that match your skills, and automatically send the contact person a resume.             

Write a web agent that returns the best price on a dozen roses delivered to Los Angeles.        		
Spider / agent ideas

Integrate multiple protocols into a single function             

Automatically analyze server logs for anomalies and send emails when found             

Hide a camera in your medicine cabinet, when the door opens take a digital picture and post it on a web site.             

Create an email to fax service.  I actually did this a long time ago using the Linux FAX utility   

Have your phone ring when a specific person on your buddy list signs onto Instant Messenger		
Spider / agent ideas

Interface completely separate web sites.           

Write a new front end to someone else’s online catalog.             

This makes more sense than it sounds.  Consider, for one, a custom interface to a retailer of some kind where you receive a commission.

Integrate one online service on a separate web site.		
Spider / agent ideas

Start a new business  

Start a digital clipping service that databases information and images about both your client, and their competition, by spidering web, ftp and news servers along with DNS records, and new domain name applications             

Index and store this information on a data-driven web site that your customer can access            
I’ll tell you why I’m not doing this near the end of this session.		

Not only can you use all these protocols, you can also integrate your agents and spiders with
   Email,
   FAX machines,
  Phones, 
  Motors, 
  Thermometers,
  Digital Cameras, 
  Or virtually any type of electronic hardware 		
I’ve personally written agents and spiders in each of these technologies.

Java works well when you need to provide a graphical user interface.  I’ve also used Tcl/Tk and PHP-GTK for this purpose.

My Tcl/Tk web agent experience is pretty much limited to an article “Writing Intelligent Web Agents”, which I wrote for the March 2000 issue of Web Techniques.

My favorite technology for this purpose is PHP used in conjunction with cURL
		
Like I said, I like to use PHP, but I use it primarily for processing, and use cURL to download files.

You can, however, download files from a number of server types with PHP on it’s own.		
You can also do much more complicated file transfers in PHP if you are willing to do direct socket programming, as in this example that performs a simple DNS look up, or WHOIS function.

.		
Here are the results of the previous script.

If you ever have the need to automate the process of obtaining DNS records, now you have the code.		

As useful as PHP is, I prefer to use it in conjunction with a program called cURL.  

cURL (client for URLs) is a command line tool for getting or sending files using URL syntax and a number of protocols.

This cURL should not be confused with the client-side programming language produced by the Curl Corporation.

cURL source (and binaries for about 50 platforms) are available from their web site.

CURL is available as a PHP module, and can be used directly within PHP.  

I prefer to use it from the command line because it’s a little simpler and I don’t have to worry about whether the module is installed.

I normally would not suggest running a shell script from PHP within a web space because you’re potentially giving a hacker access to the shell. 

But PHP can be run as a stand-alone program, just like Perl, and that’s typically how I use PHP when I write spiders and agents.

This eliminates the problems involved with running shell scripts from a web space--because there IS NO WEB SPACE in these programs.  Everything runs from a command line.  I might be fooling myself thinking that this is more secure, but I believe that it is.

The first two examples on this slide show how to download a web site with cURL.  

Example one downloads the HTML default page for the site and displays it on the screen.  The second example copies the default web page to a local file called “file”. 

  
The third example shows how PHP can run cURL from a shell.  This is pretty slick because the content of the entire downloaded web page goes into the variable $temp, where it is easily parsed later.

As I mentioned earlier, I like to run PHP as a stand-alone program, much like Perl.  Example 4 shows the syntax fro doing this.  I’ve also found that sometimes it’s handy to create batch files that run a number of PHP powered spiders/agents.  This is especially true if you’re launching them as scheduled tasks.	
CURL has a wide array of options.  These are all defined in the copy of the cURL manual, which is also on the CDROM.

One option is the “--head”  option.  This option downloads the header information.  Headers are intended for use by a browser for all kinds of purposes.  In an http header you may  find:
1.	Server:  identifies the server technology
2.	Date: the date of the download, can be used to determine time zones if GMT isn’t used
3.	Set-Cookie: the contents of any cookies set by the server
4.	Cache-control: & Pragma: here both used to turn off browser caching, which makes since as the content of the NY Times page probably changes often
5.	Content-type: tells the browser (or our client) what type of media we’re downloading.  If this was a gif image, the content-type would be “image/gif”	

The list of header variables varies from server to server.  Another one that is very useful is “Last-Modified:”.  If you keep track of this variable, you can tell if web content has changed since the last time you looked.

One of the spider ideas I mentioned earlier was to write an agent that recorded the web server technology used by the Fortune 500 and to contact you when it changed--can you see now how this might be a realatively easy programming task?	
OK, so now you know how to download web content.  What do you do with it?  The first thing you need to do is parse the content of the data and remove the information you find important.

In the case of spiders, you’ll want to find all the links in the page, so my first parsing example shows ONE WAY to parse hyperlinks.  (AGAIN, ONLY ONE WAY--NOT THE ONLY WAY).

I’ve been doing this type of thing for many years and I keep coming back to a technique I used in 1999, and one that I wrote about in my March 2002 WebTechniques article, “Writing Intelligent Web Agents”.   It’s something I call the “inclusive” parse because I state the starting and ending criteria that surrounds what I’m interested in, and saves that data inclusively.

I also occasionally use something I call an “exclusive parse” , which does the opposite.  This is useful for removing items from a downloaded page, although there are more effective ways of doing this in PHP with the str_replace(“find”, “replace” $org_string); function, and will not be used in these examples (read the before mentioned article for more information about the exclusive parse).

In addition to the start and end search criteria, I also define a maximum number of character happen between search criteria.  This is a safety feature that protects against poorly written HTML and cases where the ending criteria is never found.	
After the inclusive parse is finished, I remove all but the very data that describes the link, this discards the equal signs, excess white space, and delimiters.

This particular parse scheme requires either single or double quotes around the link destination.  Links may work without these delimiters, but they are non-compliant with HTML standards and would increase the complexity of this simple parse engine.

The routine returns an array containing everything it found, in this case links to other pages.	
This is how the inclusive parse is used in the real world.

(The code is available on the CD ROM in the example scripts directory.)		
Here is how we remove all but what’s between the delimiters is how the clean_link() function is used in the real world.

The “inclusive_parse” function may also work here for removing the delimiters, but I don’t like the idea of using the routine when the start and end search criteria is the same.  It should probably work, but from a fault tolerance point of view, it just sounds like a bad idea.

(The code is available on the CD ROM in the example scripts directory.)		
So, here’s the top-level code for creating an array called $link_array, which contains all the hyperlinks in the default page on www.example.com.

The last segment of code displays the links.

(The code is available on the CD ROM in the example scripts directory.)		
In actual experience, you usually know where the data is located in targeted web pages.  And the data is generally in tables, which vastly simplifies things because you can use the HTML table tags to parse the table data.

Suggestion when doing this:
1.	Parse on the closing tags (</td>) and not the beginning tags (<td>), because you’ll avoid any complications that are otherwise caused by tag attributes.  The closing tags have no attributes.
2.	Convert the closing tags to a known value and case.  For example convert </TD> and </th> to </td>. 

(The code is available on the CD ROM in the example scripts directory.)
		
At the start of this session we talked about interfacing to existing web applications that don’t even know about your spider or agent.  Here’s how it’s done.

It’s basically done the same why data is normally passed from a form to a program that takes action on the form data, except this time the data is coming from your program, and not the intended form.

1.	Use a browser to load the web page that contains the form you want to emulate and save it to a local file.
2.	If the form method isn’t “GET” change it, so that the form variables will be visible later.
3.	Remove the ACTION attribute, so the page will not look for the code that processes the form data when you press the submit button.
4.	Fill out the form and press submit.
5.	Use the browser’s location field to read the form’s data fields.	

It’s the data found in step 5. that you want to send as either a POST or GET method to the original ACTION script.	

CURL has the capability to send data to a web page as either GET or POST data.

To send data as GET data, simply place in the URL (after the “?”) as shown in the example.

To send POST data use cURL’s  -d option.

The difference between POST and GET is that GET variables are passed in the query string along with the URL.  POST values, in contrast, are send in a separate file.  Since servers generally limit query strings to about 250 characters, you can send more data with a POST method.

Remember that all your actions will appear in a log file somewhere.  So if a host admin is expecting to only see only POST requests to his web application and your sending GET requests to the same page, you might raise suspicions.  		
cURL has the capability to download and decrypt pages that are SSL encrypted.

Again, we talked about how all your actions will appear in a log file somewhere.  

One of the HTTP variables that is sent to every page you contact is the HTTP_USER_AGENT variable.

For most browsers it will look like:
Mozilla/5.0 (Windows; U; Win 9x 4.90; en-US; rv:1.0.0) Gecko/20020530

What you call your agent is totally up to you.	
	
CURL deals with log on credentials as shown.

This works with standard CODE 401 “Authentication Required” web server authentication.  

I’m not sure if it works with the improved “Digest Authentication”, but it doesn’t really matter since few sites use Digest Authentication.

If authentication is done with cookies, use cURL’s send cookie options, as described in the cURL manual.		
There are several ways to launch your agents and spiders:

1.	From a command line.
2.	As a scheduled service
3.	On start-up or log-in
4.	Run and loop continually

You can also run multiple spiders/agents simultaneously from the same computer.  Since most of the processing time is spent waiting for servers to respond and send data, this vastly increases the efficiency of the program(s).		
Since your often downloading data that is maintained by some one else, you want to write code that is as fault tolerant as possible.		
Observe case 
(as stated to the right)		
Look for and use server redirects that are in the header files.

(see the code sample in the CD ROM)		
In the past I’ve have customer’s spiders stop running when an extra column is added to a table.

Don’t wait for things to blow-up before you act on changes to web sites.

Observe the suggestions (stated at the right).		
Observe site policies and don’t spider sites that don’t want to be spidered.

Remember that spiders, when run automatically, may consume a great deal of bandwidth.  		
Be smart with your new skills.

Observe intellectual property rights.  THIS is the reason I’m not doing the digital clipping service.  I could not find enough available sources of information.

Adhere to site usage policies as sites can and will block your IP address.

Don’t even think of spidering eBay or Yahoo (for example)

DNS registries limit the number of requests in a given day--don’t be greedy

Be responsible
•	Don’t spam people
•	Don’t do DOS *Denial of service” attacks
		
Some times you just want to run an agent once in a while.

Sometimes you may want to run a agent and send output to either a FAX or email.

It you have a lot of data, you may want to to filter the data and store it in a database.

You can also use the GET and POST techniques discusses earlier (in a kind of home spun SOAP protocol) to upload the data to a remote web site, or remote web site.

Remember that automated agents can generate a lot of traffic.  I had a couple of simple spiders running for a client and was generating through several gigabytes of traffic every couple of days..  And that was only the data that was sent to a remote server, and did not include the actual page downloads!
		
Thanks,

If you do something interesting, let me know about it.

mikeSchrenk@yahoo.com