DataMystic

Great product.
I can't find a way to get Webpipe to download successive pages of web database queries. This is a recurrent problem I have across a number of web-based databases. I'll explain with a specific example: Law Society of Singapore. There is a database to allow members of the public to look up the Law Society's online directory to find lawyers. I want to download this whole directory. Usually I can do this by giving Webpipe the urls associated with each query - eg http://58.185.2.219/lawsociety/practiti ... B&cat=name
searches all Bs in this particular database. So by giving Webpipe 26 urls to download (changing "letter=B" to "letter=A", etc) I can pull down the whole database ... except that each query returns many records, which do not fit on one page, and so there are hyperlinks at the bottom of the 20 odd files it does return, to take you to pages "1, 2, 3, etc". It's those buggers 1, 2, 3 etc that I want. I need Webpipe to follow the links and pull down each successive page. Is there a way to do this ? Setting the number of "levels" doesn't seem to help.

Sure - press F1 on the URl field - and you will see that there are a number of macros you can use to generate new urls to download.

The following macros are supported:

{:day} the current day number (1 - 31)
{:0day} the current day with a leading zero, when necessary (01 - 31)
{:shortday} the day as an abbreviation (Sun-Sat)
{:longday} the day as a full name (Sunday-Saturday)
{:month} the current month as a number without a leading zero (1-12)
{:0month} the month as a number with a leading zero (01-12)
{:shortmonth} the month as an abbreviation (Jan-Dec)
{:longmonth} the month as a full name (January-December)

{:year} the current year as a two-digit number (00-99)
{:longyear} the year as a four-digit number (0000-9999)
{:hour} the current hour without a leading zero (0-23)
{:0hour} the hour with a leading zero (00-23)
{:minute} the current minute without a leading zero (0-59)
{:0minute} the minute with a leading zero (00-59)
{:5..12} this is a special macro which will load URLs with numbers from 5 to 12. You may also use leading zeros in the first number.

{:0..100|25} the | separator defines enumeration step. In this example numbers 0, 25, 50, 75, 100 will be generated.
{:a..z} to load URLs with letters from a to z. You may use single letters from A to Z and a to z here.

More information on URL Macros is available in the Help menu | Contents | Advanced Features | Using URL Macros.

Thanks !