spider script to get all urls in a web site - Djuggler web scraping and data integration software - Djuggler Forum - Default Group - Script Repository

Unanswered Active Topics

Forums Search

Forums > Default Group > Script Repository

Reply To Topic

Topic: spider script to get all urls in a web site

Username:

Security Code:

Enter the code shown above:

Subject:

RE: spider script to get all urls in a web site

Message:

Submit

Cancel

Preview

Attachments
Current Attachments My Files Upload

Topic Review

Gleb

11 Oct 2010 09:58 AM
I can't find place where to put "Copy Text Between".

Umesh

11 Oct 2010 05:28 AM
Gleb, Simply use Copy Text Between control and in Properties set starting tag <h1>, End tag </h1>. Umesh

Gleb

10 Oct 2010 01:18 PM
Could you add to the script ability to extract specified data, for example, page title from <h1></h1>? Thank you.

Joey

16 Nov 2009 11:12 AM
Nice web scraper example script. Thanks. J.

Barry

11 Mar 2009 09:49 AM
Creating your own Internet bots with Djuggler, cool! Thanks for the script.

Amanda

11 Feb 2009 10:59 PM
Thanks! Great for finding 404 errors.

Tijn

02 Feb 2009 07:24 PM
This script spiders a web site like the bots from search engines. It will collect all urls in a page and then visit the urls found for new urls. The script will stop if it has found and visited all urls in your site. It will not follow urls outside your web site. Use this script to see if a search engine robot can find all your pages. The first script lines define the settings for the script, like the base URL and what page extensions should be included. A debug switch will also produce a text file with an overview of all links on each page in your web site. Note: This script uses the Djuggler List variable which is not available in the Djuggler Personal version. Use the Djuggler Lite or higher version.
Get_all_urls_in_a_site.djs

Powered by Active Forums

	Forum participation and optional registration You don't need to be registered to partcipate in the Djuggler forums, however if you want to subscribe to email notifications you need to register. You can also subscribe to the forum RSS feed.