wget how to download file from a web page that prompts you to click on "Agree" condition

I'd like to download a file using the wget utility. I followed the instruction on "How do I use wget/curl to download from a site I am logged into?" and the download process works; however, the saved cookies expires after a certain time, so I'm unable to continue downloading the file the following day.

Here's the URL I'm trying to download the a file from:

https://frbservices.org/EPaymentsDirectory/FedACHdir.txt

The download page requires that I click the "Agree" button before I can proceed with the download.

Is there a way to include the "Agree" submission with the wget utility?

Thank you.

Solution 1:

You can submit a form using wget, using the --post-data option. First, look at the form that page uses:

<form name="acceptedForm" id="acceptedForm" action="submitAgreement" method="post">
...
        <button id="agree_terms_use" name="agreementValue" type="submit" value="Agree">Agree</button>  &nbsp; 
        <button id="disagree_terms_use" name="agreementValue" type="submit" value="Do Not Agree">Do Not Agree</button>

Typically, the value of the action attribute is used to get the target URL, which becomes https://frbservices.org/EPaymentsDirectory/submitAgreement. The names of the form elements become the parameters. Then you need to save the cookies that you get from accepting the agreement. From this SO post, we can construct the command needed:

wget --post-data="agreementValue=Agree" https://frbservices.org/EPaymentsDirectory/submitAgreement --save-cookies cookie.txt --keep-session-cookies --delete-after

Then, we can use these cookies to download the file:

wget --load-cookies=cookie.txt 'https://frbservices.org/EPaymentsDirectory/FedACHdir.txt?AgreementSessionObject=Agree'