This script spiders a web site like the bots from search engines. It will collect all urls in a page and then visit the urls found for new urls. The script will stop if it has found and visited all urls in your site. It will not follow urls outside your web site.

Use this script to see if a search engine robot can find all your pages.

The first script lines define the settings for the script, like the base URL and what page extensions should be included. A debug switch will also produce a text file with an overview of all links on each page in your web site.

Note: This script uses the Djuggler List variable which is not available in the Djuggler Personal version. Use the Djuggler Lite or higher version.

Thanks! Great for finding 404 errors.

Creating your own Internet bots with Djuggler, cool!
Thanks for the script.

Nice web scraper example script.
Thanks.
J.