contact  |  about  |  sitemap


contact  |  about  |  sitemap

Google web scraper example script
Last Post 29 Jun 2010 03:10 PM by Tijn. 3 Replies.
Sort:
PrevPrev NextNext
Author Messages
Tijn

--
16 Nov 2009 11:09 AM
This script shows how to scrape the results of a Google search.

The web scraper script uses the copy text between from source action to harvest each result.

The Read Next Link by Title action will loop all next pages in the Google search results. A simple If action will limit the result to 10 pages.

A Match and Replace Regex action is used to transform the Google data for representation.

This Djuggler example is a typically screen scraper script that can also be found in the Djuggler Demo directory as installed by the Djuggler setup.


Economic_Google_search_results.djs

Reedster

--
25 Jun 2010 01:32 AM
Could this script be adapted to a Google Image Search?

I find that using a keyword search in "Images" is somewhat more specific in identifying all the sites that use the keywords in their image ALT descriptions!

I also want to recover the thumbnail images presented in Google's results page. If I "SAVE AS" the individual returned Google page displayed using Mozilla, it creates a clean HTML page with an associated file with all the 20 images... which it names "image_001" through "image_020".

This does not work using Internet Explorer... which does not create a coherent HTML file to parse!


mazel

--
29 Jun 2010 09:45 AM
why dont you get the web page directly via Get Web page action? Then accumulate all the links to your thumbnails and donwload the images via Dowload File action...


Tijn

--
29 Jun 2010 03:10 PM
See demo script 8 'google web scraper example script' that is included in the Djuggler install.




Quick Reply
toggle
  Username:
Subject:
Body:
Security Code:
Enter the code shown above:

Submit

Powered by Active Forums

Forum participation and optional registration

You don't need to be registered to partcipate in the Djuggler forums, however if you want to subscribe to email notifications you need to register. You can also subscribe to the forum RSS feed.

Forum participation and optional registration

You don't need to be registered to partcipate in the Djuggler forums, however if you want to subscribe to email notifications you need to register. You can also subscribe to the forum RSS feed.