How to download a file from a website that prompts a popup form?
The website requires you to have two cookies set: A valid JSESSIONID
and the abaDataCaptureCookie
called agreement cookie. The first can be obtained with your first wget
command, the second one needs to be added manually.
-
Obtain a valid
JSESSIONID
and save it tocookies.txt
:wget --post-data="agreementValue=Agree" frbservices.org/EPaymentsDirectory/submitAgreement --save-cookies cookies.txt --keep-session-cookies --delete-after
-
Add the
abaDataCaptureCookie
line:echo -e "frbservices.org\tFALSE\t/EPaymentsDirectory/\tFALSE\t0\tabaDataCaptureCookie\tabaDataCaptureCookie" >>cookies.txt
-
Download the file:
wget --load-cookies cookies.txt frbservices.org/EPaymentsDirectory/FedACHdir.txt
I'm not quite sure how long the first cookie stays valid, but I assume you won't download the list multiple times per hour, so obtaining a new one on every run seems OK to me – that's the failsafe approach after all.
Script version of the above steps, using a tempfile as the cookie file:
#!/bin/bash
cookiefile=$(mktemp)
wget --post-data="agreementValue=Agree" frbservices.org/EPaymentsDirectory/submitAgreement --save-cookies $cookiefile --keep-session-cookies --delete-after
echo -e "frbservices.org\tFALSE\t/EPaymentsDirectory/\tFALSE\t0\tabaDataCaptureCookie\tabaDataCaptureCookie" >>$cookiefile
wget --load-cookies $cookiefile frbservices.org/EPaymentsDirectory/FedACHdir.txt
Running this script will save the current FedACHdir.txt
to the current directory without leaving a cookies.txt
lying around, if there's already a file with this name wget
adds a number and saves it as e.g. FedACHdir.txt.1
.
On resolving this issue I found great help here: Format of cookies when using wget? · U&L