Unfortunately, loading a browser would take a lot of resources because it has to load a lot of other things like the toolbar and buttons.

The easiest way to manage these sites is to open a browser and load the site. The biggest is that it cannot handle dynamic sites – sites that are rendered using JavaScript. Though this is a fast method, it has its limitations. We covered this process in-depth in our JavaScript web scraping tutorial. This can then be parsed using packages like Cheerio. It directly sends a get request to the web page and receives HTML content. The first method uses packages e.g., Axios. Generally, there are two methods of accessing and parsing web pages. There are a few methods to accessing and parsing web pages, but in this tutorial we will be covering how to do it with Google Puppeteer. I ultimately just need to get the data somewhere accessible by Google AppScript.Web scraping and automation with JavaScript has evolved a lot in recent years. My main issue is I don't understand how to actually handle file downloads in Puppeteer, I'm currently just editing the default download location for the google chrome instance puppeteer creates. So is there a way I could pass the file download in Puppeteer directly into Google AppScript somehow? Or use the Drive API to directly receive the file? IF I were to do that, my Drive for Desktop work-around would not be viable. Thing is, I'd like this to be some type of scheduled cloud function not having to be manually run on my computer. I don't yet understand all too well how to handle file downloads through puppeteer, but I currently have a work around by setting the default chrome download path to a folder on my computer, and that folder is actually Drive for desktop, so easy way to get files directly in drive. I currently have a Puppeteer script which automates CSV downloads into google drive which then allows me to programmatically access the data and move it into google sheets.Ĭurrently, I run this locally on my computer on VS code. Would appreciate if anyone can suggest ideas:

Download google puppeteer for free