Question : Question about wget, https, and downloading from a CGI application...

Hey experts!

I need to download some documents from this website...there are 851 in all. I thought that I could use "wget" with an input file to do this, but all I am able to download is a page that tells me to login.

Is there a way to use wget to login to the site? Or is there some other way to get these files?

this is the URL

https://secure.website/cgi-bin/show_case_doc?1,21924,,,,1

the next url would  be like this:

https://secure.website/cgi-bin/show_case_doc?2,21924,,,,1

Please help!

Answer : Question about wget, https, and downloading from a CGI application...

Hi neomage23,

Try Accessing http://www.unicode.org/mail-arch/unicode-ml/y2002-m08/0067.html directly with your web browser and you'll be prompted for a login.

Run this sample script either on a unix command line or via your web server if you have a cgi enabled web server. It should give u an idea of what you need to do.  It accesses the same url as above, but automatically verifies with the server..

#!/usr/bin/perl
print "Content-type: text/html \n\n";

use CGI::Carp qw(fatalsToBrowser);
use LWP 5.64;
use strict;


my $browser = LWP::UserAgent->new;
$browser->credentials(  # add this to our $browser 's "key ring"
        'www.unicode.org:80',
        'Unicode-MailList-Archives',
        'unicode-ml' => 'unicode');

my $url = 'http://www.unicode.org/mail-arch/unicode-ml/y2002-m08/0067.html';
my $response = $browser->get($url);

die "Error: ", $response->header('WWW-Authenticate') ||
    'Error accessing', #  ('WWW-Authenticate' is the realm-name) "\n ", $response->status_line, "\n at $url\n Aborting"
   unless $response->is_success;

print $response->content;