LWP user agent grabs the intermediate wait page after POST intead of the actual result page

my $web_browser = LWP::UserAgent->new();
push @{ $web_browser->requests_redirectable }, 'POST';
$web_browser->timeout(300);

$web_response = $web_browser->post('http://blabla.com/travel/

'fromCity' =>

=> 'CVG'

die "Error: ", $web_response->status_line()
unless $web_response->is_success;

my @content = $web_response->content;

displays the progress bar using javascript.... => I matched the

> I wrote a small LWP based perl program to search the air fare from a
> travel website using POST.
>

>
> When I print the content, I see the "intermediate" wait page (where it
> displays the progress bar using javascript.... => I matched the
> content with the "view source" from IExplorer)
> I am unable to capture the final air fare page. It takes time for the
> website to do the search and then display the air fare result page.
> How do I make my program wait for the actual result and not grab the
> intermediate response.

> bhabs wrote:
> > I wrote a small LWP based perl program to search the air fare from a
> > travel website using POST.
> >
> [...code snipped]
> >
> > When I print the content, I see the "intermediate" wait page (where it
> > displays the progress bar using javascript.... => I matched the
> > content with the "view source" from IExplorer)
> > I am unable to capture the final air fare page. It takes time for the
> > website to do the search and then display the air fare result page.
> > How do I make my program wait for the actual result and not grab the
> > intermediate response.
>
> You have to simulate what the browser does, and from your
> description, this is most likely a repeated ajax request
> to the server. Analyze the behaviour of the javascript
> and see how it fetches the progress state and what it
> does once the result is calculated, then craft those
> actions yourself. You best chances to see exactly what is going
> on in the background is with a network sniffer like wireshark,
> or a browser plugin like Firefox' Live HTTP Headers.

> bhabs wrote:
>> I wrote a small LWP based perl program to search the air fare from a
>> travel website using POST.
>>
> [...code snipped]
>>
>> When I print the content, I see the "intermediate" wait page (where it
>> displays the progress bar using javascript.... => I matched the
>> content with the "view source" from IExplorer)
>> I am unable to capture the final air fare page. It takes time for the
>> website to do the search and then display the air fare result page.
>> How do I make my program wait for the actual result and not grab the
>> intermediate response.
> You have to simulate what the browser does, and from your
> description, this is most likely a repeated ajax request
> to the server. Analyze the behaviour of the javascript
> and see how it fetches the progress state and what it
> does once the result is calculated, then craft those
> actions yourself. You best chances to see exactly what is going
> on in the background is with a network sniffer like wireshark,

> or a browser plugin like Firefox' Live HTTP Headers.