I'm testing Djuggler. I need it to collect data from some web pages.


My target website has some kind of "anti spider" protection, and after browsing around 50 pages, it blocks the requesting ip,

so data collection can't proceed further.

What can I do to solve this ?

The program has some built in functions to, for example, use proxies, or reset the modem to renew his public ip address ?

You could use a public anonymous proxy.
Here is a list:

http://www.publicproxyservers.com/page1.html


In the Open web page action, you will find the proxy settings.
You can typically switch them every 49 request in the script logic.

Would be great if you can upload copy of this script in action when you get it working

Hi,

I am also facing similar kind of problem, I used my script to extract data from a secure site i.e. Https sites, So how can I extract data from a secure website, may be a secure site generate a session or may be something else ( since I am new in this field, the question may be silly.)..can any one suggest me, is there any logic/way to extract data from a secure site.

Thanks
Umesh.

Djuggler can extract data from a secure web by simply using the Open Web Action with a https address.

Doing that via a proxy is also no problem.
When you want to use a secure proxy that will depend on the proxy.

Tijn

Hi Tijn,

Still facing same problem while retrieving data from a secure site. Here I do attach my script.

In this website when U go through a proper channel you will get some data i.e follow this path.

open web page: http://www.grad.cmich.edu/programs_alpha.htm

then go record having Title : "Hospitality and Tourism Administration **"
it has secure link i.e. https://bulletins.cmich.edu/display-degree.asp? yr=2006&level=gr&loc=onc&pid=21001040&pname=Certificate+in+Hospitality+%26+Tourism+Administration

but if I open direct this page (https://bulletins.cmich.edu/display-degree.asp? yr=2006&level=gr&loc=onc&pid=21001040&pname=Certificate+in+Hospitality+%26+Tourism+Administration) it shows error page due to security reason.

So still my question is same, Is it possible to retrieve data from a secure site, generally I used a loop which read a no of secure links, so that my script should retrieve data from all those links.

Thanks,
Umesh

Hi,

The problem was a space in the url. I used a replace text action to remove the space and now it seems to work.

See the attached script.

Regards,
Tijn

Thanks Tijn,

Sorry for disagree with your previous post. It was my mistake. :(

Regards,
Umesh