DataMystic

The prog seems to use a terribly outdated version of in-built brouser. My sniffer indicates it as IE 5.5.
The result is that many sites do not give their pages for download just because of the in-built brouser's signature.

P.S. Running evaluation version of TP Pro at OS WinXP SP3 with IE 8.0

Which in-built browser do you mean? Are you using a scripting filter, or something else?

I tried to use TP as web-parser. The tab "Files to process" encourages me to download pages. This is how the manual says "..You can specify an internet link to download and then process by specifying the link e.g. http://www.hotmail.com.
So I put a list of internet-links into "Files to process" for downloading and further processing. Nonetheless, just 30-40% of links really succeed. I analyzed the TP communication by http-sniffer. It's identified as IE 5.5 In GET-requests.
For many sites such user-agent header is enough to decline the whole request.

That's what I wanted to say.

Yes - this is exactly what is used. Would you like the user agent to be blank for the next release?

Hmm. No. Leaving it blank may be worse. IMHO better's to replace it by some kind of universal header that is not widely rejected by web-resources
for instance -
Googlebot/2.1
Mozilla/5.0

Done - we'll use Mozilla/5.0