How does one scrape a page with JavaScript?

[Philip J. Fry]

New to web scraping here. I need to scrape a government website for a list of Government approved retailers. Why they didn't provide a .csv file in the first place boggles me. They also coded the menu with JavaScript so I can't use http://webscraper.io/, as I found out.

The site requires you to input your zip code, press "go," and then choose a drop down to see your results (there's only ever one drop down each and every time - they could have removed this step but didn't). Then, and only then, are the retailers in your area are shown. The funny thing is that the pagination ( <-, 1, 2, 3, ... X -> ) is done in javascript. Why? Who knows.

My next go-to tool is curl and wget but I'm sure there's a better way. Please point me in the right direction. Thanks!

 

[turbin3]

http://phantomjs.org/
http://casperjs.org/
https://nicolas.perriault.net/code/2011/scrape-and-test-any-webpage-using-phantomjs/

 

[hehejo]

I don't see the problem. You can also extract data from JS scripts. If the site is doing ajax requests, you just do those requests yourself.

If that's too complicated, just use a headless browser as turbin3 suggested.