![\"\"](\"/images/content/5b6/8e4c91039eb155f8b4577/1200_-1086-)
Have you ever needed to gather all the information from a web page? Here's how to write a tool in PowerShell that will do that for you.
\nWeb scraping is the art of parsing an HTML web page and gathering up elements in a structured manner. Since an HTML page has a particular structure, it's possible to parse through this and to get a semi-structured output. I've intentionally used the word \"semi\" here because, if you begin playing with web scraping, you'll see that most web pages aren't necessarily well-formed. Even though the page doesn't adhere to \"well-formed\" standards, they will still appear as such in a browser. This is what most webmasters care about.
\r\nBut, using a scripting language like PowerShell, a little ingenuity and some trial and error, it is possible to build a reliable web-scraping tool in PowerShell to pull down information from a lot of different web pages.
\r\nNote: Web pages can vary wildly in their structure and once even a tiny amount is changed, it can cause your web scraping tool to blow up. Focus on the basics for this tool and build more specific tools around particular web pages, if you're so inclined.
\r\nThe command of choice is Invoke-WebRequest. This command should be a staple in your web scraping arsenal. It greatly simplifies pulling down web page data allowing you to focus your efforts on parsing out the data that you need therein.
\r\nTo get started, let's use a simple web page that everyone is familiar with; google.com and see how a web scraping tool sees it. To do this, I'll pass google.com to the Uri parameter of Invoke-WebRequest and inspect the output.
\r\n$google = Invoke-WebRequest –Uri google.com
\r\nThis is a representation of the entire google.com page all wrapped up in an object for you. Let's see what we can pull from this web page. Perhaps I need to find all the links on the page. No problem. I simply need to reference the Links property.
\r\nThis will enumerate various properties about each link on the page. Perhaps I just want to see the URL that it links to.
\r\nHow about the anchor text and the URL? Since this is just an object, it's easy to pull information like this.
\r\nWe can also see what the infamous Google.com form with the input box looks like under the hood.
\r\nLet's now take this another step and actually download some information from the web page. A common requirement is to download all images from a web page. Let's go over how to use PowerShell to enumerate all images on a page and download them to your local computer.
\r\nThis time, we'll only get the basic information using the –UseBasicParsing parameter. This is quicker because Invoke-WebRequest doesn't crawl the DOM. It's almost necessary when querying massive web pages. For example, let's download all of the images on the cnn.com website.
\r\n$cnn = Invoke-WebRequest –Uri cnn.com –UseBasicParsing
\r\nNow I'll figure out each URL that the image is hosted on.
\r\nOnce we have the URLs, it's just a simple matter of using Invoke-Request again only, this time, we'll use the –OutFile parameter to send the response to a file.
\r\n@($cnn.Images.src).foreach({
$fileName = $_ | Split-Path -Leaf
Write-Host \"Downloading image file $fileName\"
Invoke-WebRequest -Uri $_ -OutFile \"C:$fileName\"
Write-Host 'Image download complete'
})
In this case, I saved the images directly to my C: but you can easily change this to whatever directory you'd like here. If you'd like to test the images directly from PowerShell, you can use the Invoke-Item command to pull up the image's associated viewer. You can see below that Invoke-WebRequest pulled down an image from cnn.com with the word \"bleacher.\"
\r\nUse the code I went over today as a template for your own tool. Build a PowerShell function called Invoke-WebScrape, for example, with a few parameters like –Url or –Links. Once you have the basics down, you can easily create a customized tool to your liking that can be applied in many different places.
\n","docType":"article","guid":"articles_fa8dccb7c57e31beb3ecc983ebf24355","lastModified":"2018-09-05T09:00:00.000Z","published":"2018-09-05T09:00:00.000Z","structuredData":null,"subtitle":"Write a tool in PowerShell that will gather all the data from a web page.","template":"v1","title":"How to Create a Web Scraping Tool in PowerShell","type":"article","author":{"avatar":null,"bio":"Adam Bertram is a 20-year veteran of IT and experienced online business professional. He's an entrepreneur, IT influencer, Microsoft MVP, blogger, trainer and content marketing writer for multiple technology companies. Adam is also the founder of the popular IT career development platform TechSnips. Catch up on Adam's articles at adamtheautomator.com, connect on LinkedIn, or follow him on Twitter at @adbertram or the TechSnips Twitter account at @techsnips_io.","name":"Adam Bertram","image":null,"slug":null,"social":{"facebook":"","linkedin":"https://www.linkedin.com/in/adambertram/","twitter":"https://twitter.com/adbertram","website":"https://www.adamtheautomator.com/"},"status":"enabled","title":null,"type":"Contributing Writer","user":{"imagePath":"https://staging.business.com/images/undefined","profile":{"company":"eGain Communications","jobTitle":"Product Management Analyst","website":""}}},"breadcrumbs":null,"channels":{"primary":{"name":"Technology","slug":"technology"},"sub":{"name":"Technology Solutions","slug":"technology-solutions"}},"meta":{"robots":"index, follow","description":"Have you ever had the need to gather up all of the information from a web page? Here's how to write a tool in PowerShell that will do that for you.","title":"How to Create a Web Scraping Tool in PowerShell - business.com"},"parsedContent":{"toc":[{"id":"the-command-of-choice","title":"The command of choice"},{"id":"gathering-images","title":"Gathering images"}],"json":[{"component":"p","children":["Have you ever needed to gather all the information from a web page? Here's how to write a tool in PowerShell that will do that for you."]},{"component":"p","children":[{"component":"picture","props":{"className":"aligncenter full"},"children":["Perhaps this is part of some larger automation routine, an API might not be available to get to the data the \"right\" way, or maybe you just need an email every time your favorite Pokemon character gets updated. In any case, all of these scenarios are possible with web scraping."]}]},{"component":"p","children":["Web scraping is the art of parsing an HTML web page and gathering up elements in a structured manner. Since an HTML page has a particular structure, it's possible to parse through this and to get a semi-structured output. I've intentionally used the word \"semi\" here because, if you begin playing with web scraping, you'll see that most web pages aren't necessarily well-formed. Even though the page doesn't adhere to \"well-formed\" standards, they will still appear as such in a browser. This is what most webmasters care about."]},{"component":"p","children":["But, using a scripting language like PowerShell, a little ingenuity and some trial and error, it is possible to build a reliable web-scraping tool in PowerShell to pull down information from a lot of different web pages."]},{"component":"p","children":["Note: Web pages can vary wildly in their structure and once even a tiny amount is changed, it can cause your web scraping tool to blow up. Focus on the basics for this tool and build more specific tools around particular web pages, if you're so inclined."]},{"component":"h2","props":{"id":"the-command-of-choice"},"children":["The command of choice"]},{"component":"p","children":["The command of choice is Invoke-WebRequest. This command should be a staple in your web scraping arsenal. It greatly simplifies pulling down web page data allowing you to focus your efforts on parsing out the data that you need therein."]},{"component":"p","children":["To get started, let's use a simple web page that everyone is familiar with; google.com and see how a web scraping tool sees it. To do this, I'll pass google.com to the Uri parameter of Invoke-WebRequest and inspect the output."]},{"component":"p","children":["$google = Invoke-WebRequest –Uri google.com"]},{"component":"p","children":[{"component":"picture","props":{"className":"aligncenter full"},"children":[{"component":"source","props":{"srcSet":"/images/content/5b6/8e4c91039eb155f8b4577/1200_-1086- 1200w, /images/content/5b6/8e4c91039eb155f8b4577/668_-604- 668w, /images/content/5b6/8e4c91039eb155f8b4577/320_-289- 320w","sizes":"668px, 100%"}},{"component":"img","props":{"src":"/images/content/5b6/8e4c91039eb155f8b4577/930-842-","alt":"","width":"930","height":"842","data-content-img-id":"5b68e4c91039eb155f8b4577"}}]}]},{"component":"p","children":["This is a representation of the entire google.com page all wrapped up in an object for you. Let's see what we can pull from this web page. Perhaps I need to find all the links on the page. No problem. I simply need to reference the Links property."]},{"component":"p","children":[{"component":"picture","props":{"className":"aligncenter full"},"children":[{"component":"source","props":{"srcSet":"/images/content/5b6/8e4e351ee473a658b45a4/1200_-390- 1200w, /images/content/5b6/8e4e351ee473a658b45a4/668_-217- 668w, /images/content/5b6/8e4e351ee473a658b45a4/320_-104- 320w","sizes":"668px, 100%"}},{"component":"img","props":{"src":"/images/content/5b6/8e4e351ee473a658b45a4/934-304-","alt":"","width":"934","height":"304","data-content-img-id":"5b68e4e351ee473a658b45a4"}}]}]},{"component":"p","children":["This will enumerate various properties about each link on the page. Perhaps I just want to see the URL that it links to."]},{"component":"p","children":[{"component":"picture","props":{"className":"aligncenter full"},"children":[{"component":"source","props":{"srcSet":"/images/content/5b6/8e4f91039eb6d708b457a/1200_-872- 1200w, /images/content/5b6/8e4f91039eb6d708b457a/668_-485- 668w, /images/content/5b6/8e4f91039eb6d708b457a/320_-232- 320w","sizes":"668px, 100%"}},{"component":"img","props":{"src":"/images/content/5b6/8e4f91039eb6d708b457a/770-560-","alt":"","width":"770","height":"560","data-content-img-id":"5b68e4f91039eb6d708b457a"}}]}]},{"component":"p","children":["How about the anchor text and the URL? Since this is just an object, it's easy to pull information like this."]},{"component":"p","children":[{"component":"picture","props":{"className":"aligncenter full"},"children":[{"component":"source","props":{"srcSet":"/images/content/5b6/8e52b92a3a47a578b459a/1200_-556- 1200w, /images/content/5b6/8e52b92a3a47a578b459a/668_-309- 668w, /images/content/5b6/8e52b92a3a47a578b459a/320_-148- 320w","sizes":"668px, 100%"}},{"component":"img","props":{"src":"/images/content/5b6/8e52b92a3a47a578b459a/932-432-","alt":"","width":"932","height":"432","data-content-img-id":"5b68e52b92a3a47a578b459a"}}]}]},{"component":"p","children":["We can also see what the infamous Google.com form with the input box looks like under the hood."]},{"component":"p","children":[{"component":"picture","props":{"className":"aligncenter full"},"children":[{"component":"source","props":{"srcSet":"/images/content/5b6/8e5411039eb1b728b4580/1200_-579- 1200w, /images/content/5b6/8e5411039eb1b728b4580/668_-322- 668w, /images/content/5b6/8e5411039eb1b728b4580/320_-154- 320w","sizes":"668px, 100%"}},{"component":"img","props":{"src":"/images/content/5b6/8e5411039eb1b728b4580/646-312-","alt":"","width":"646","height":"312","data-content-img-id":"5b68e5411039eb1b728b4580"}}]}]},{"component":"p","children":["Let's now take this another step and actually download some information from the web page. A common requirement is to download all images from a web page. Let's go over how to use PowerShell to enumerate all images on a page and download them to your local computer."]},{"component":"h2","props":{"id":"gathering-images"},"children":["Gathering images"]},{"component":"p","children":["This time, we'll only get the basic information using the –UseBasicParsing parameter. This is quicker because Invoke-WebRequest doesn't crawl the DOM. It's almost necessary when querying massive web pages. For example, let's download all of the images on the cnn.com website."]},{"component":"p","children":["$cnn = Invoke-WebRequest –Uri cnn.com –UseBasicParsing"]},{"component":"p","children":["Now I'll figure out each URL that the image is hosted on."]},{"component":"p","children":[{"component":"picture","props":{"className":"aligncenter full"},"children":[{"component":"source","props":{"srcSet":"/images/content/5b6/8e55b51ee47e3298b458a/1200_-402- 1200w, /images/content/5b6/8e55b51ee47e3298b458a/668_-224- 668w, /images/content/5b6/8e55b51ee47e3298b458a/320_-107- 320w","sizes":"668px, 100%"}},{"component":"img","props":{"src":"/images/content/5b6/8e55b51ee47e3298b458a/936-314-","alt":"","width":"936","height":"314","data-content-img-id":"5b68e55b51ee47e3298b458a"}}]}]},{"component":"p","children":["Once we have the URLs, it's just a simple matter of using Invoke-Request again only, this time, we'll use the –OutFile parameter to send the response to a file."]},{"component":"p","children":["@($cnn.Images.src).foreach({",{"component":"br"},"$fileName = $_ | Split-Path -Leaf",{"component":"br"},"Write-Host \"Downloading image file $fileName\"",{"component":"br"},"Invoke-WebRequest -Uri $_ -OutFile \"C:$fileName\"",{"component":"br"},"Write-Host 'Image download complete'",{"component":"br"},"})"]},{"component":"p","children":[{"component":"picture","props":{"className":"aligncenter full"},"children":[{"component":"source","props":{"srcSet":"/images/content/5b6/8e5741039ebd9798b4576/1200_-690- 1200w, /images/content/5b6/8e5741039ebd9798b4576/668_-384- 668w, /images/content/5b6/8e5741039ebd9798b4576/320_-184- 320w","sizes":"668px, 100%"}},{"component":"img","props":{"src":"/images/content/5b6/8e5741039ebd9798b4576/674-388-","alt":"","width":"674","height":"388","data-content-img-id":"5b68e5741039ebd9798b4576"}}]}]},{"component":"p","children":["In this case, I saved the images directly to my C: but you can easily change this to whatever directory you'd like here. If you'd like to test the images directly from PowerShell, you can use the Invoke-Item command to pull up the image's associated viewer. You can see below that Invoke-WebRequest pulled down an image from cnn.com with the word \"bleacher.\""]},{"component":"p","children":[{"component":"picture","props":{"className":"aligncenter full"},"children":[{"component":"source","props":{"srcSet":"/images/content/5b6/8e58c51ee47c8578b459c/1200_-680- 1200w, /images/content/5b6/8e58c51ee47c8578b459c/668_-378- 668w, /images/content/5b6/8e58c51ee47c8578b459c/320_-181- 320w","sizes":"668px, 100%"}},{"component":"img","props":{"src":"/images/content/5b6/8e58c51ee47c8578b459c/942-534-","alt":"","width":"942","height":"534","data-content-img-id":"5b68e58c51ee47c8578b459c"}}]}]},{"component":"p","children":["Use the code I went over today as a template for your own tool. Build a PowerShell function called Invoke-WebScrape, for example, with a few parameters like –Url or –Links. Once you have the basics down, you can easily create a customized tool to your liking that can be applied in many different places."]}]},"resource":null,"sponsorship":null,"thumbnail":{"path":"https://staging.business.com/images/content/5d0/06c899c658acc288b458a/1500-0-","caption":"kikujungboy/Shutterstock","isBlurred":false},"bestPicks":null},"related":{"relatedArticles":[{"title":"PowerShell: Everything You Need to Know","canonical":"https://www.business.com/articles/powershell-everything-you-need-to-know/","slug":"powershell-everything-you-need-to-know","thumbnail":"https://www.business.com/images/content/5b8/998581039eb8f738b457b/1500-0-","dek":"What you need to know to unlock Microsoft PowerShell's full capabilities.","lastModified":"2018-09-05T17:00:00Z","published":"2018-09-05T17:00:00Z","sponsored":false},{"title":"Essential PowerShell Cmdlets For Auditing And Maintaining Storage","canonical":"https://www.business.com/articles/powershell-for-storage/","slug":"powershell-for-storage","thumbnail":"https://www.business.com/images/content/5d0/18e065a215ea2498b4588/1500-0-","dek":"PowerShell v5 offers some new features for managing storage.","lastModified":"2018-09-04T11:00:00Z","published":"2018-09-04T11:00:00Z","sponsored":false},{"title":"Essential PowerShell Cmdlets For Managing Active Directory","canonical":"https://www.business.com/articles/powershell-active-directory-cmdlets/","slug":"powershell-active-directory-cmdlets","thumbnail":"https://www.business.com/images/content/5cf/90cab7b4374ab068b456f/1500-0-","dek":"Administering Active Directory is one of the most critical roles in any Windows network.","lastModified":"2018-09-04T08:00:00Z","published":"2018-09-04T08:00:00Z","sponsored":false},{"title":"Using Lean Strategies to Achieve Team Synergy and Efficiency","canonical":"https://www.business.com/articles/lean-strategies-team-efficiency/","slug":"lean-strategies-team-efficiency","thumbnail":"https://www.business.com/images/content/5be/480e49c658a51108b458c/1500-0-","dek":"Learn more about lean manufacturing strategies and how to implement them in your business.","lastModified":"2018-11-08T08:00:00Z","published":"2018-11-08T08:00:00Z","sponsored":false},{"title":"Microsoft Azure vs. Amazon Web Services: Cloud Comparison","canonical":"https://www.business.com/articles/azure-vs-aws-cloud-comparison/","slug":"azure-vs-aws-cloud-comparison","thumbnail":"https://www.business.com/images/content/5cf/ff3445a215ed86a8b459d/1500-0-","dek":"Which one is right for your business?","lastModified":"2018-05-25T07:00:00Z","published":"2018-05-25T07:00:00Z","sponsored":false},{"title":"From COVID-19 to Hurricane Season: Disaster Preparedness for Small Business","canonical":"https://www.business.com/articles/business-disaster-prep/","slug":"business-disaster-prep","thumbnail":"https://www.business.com/images/content/5e7/a495e5a215eb11d8b4569/1500-0-","dek":"Natural disasters – whether a hurricane, tornado or outbreak like COVID-19 – can have a disastrous effect on small businesses, but they don't have to. Here's how to weather a disaster successfully.","lastModified":"2020-03-25T15:19:00Z","published":"2018-09-26T07:00:00Z","sponsored":false},{"title":"Your Guide to Creating a Small Business Marketing Plan","canonical":"https://www.business.com/articles/sample-marketing-plan-outline-and-template/","slug":"sample-marketing-plan-outline-and-template","thumbnail":"https://www.business.com/images/content/5f3/3e1765a215eb62d8b4567/1500-0-","dek":"Follow these templates and guidelines to get started on creating your business's marketing plan.","lastModified":"2020-08-12T12:28:00Z","published":"2018-03-02T13:00:00Z","sponsored":false},{"title":"Use PowerShell to Make WSUS Suck Less","canonical":"https://www.business.com/articles/powershell-windows-software-update-services/","slug":"powershell-windows-software-update-services","thumbnail":"https://www.business.com/images/content/5b8/eca2d92a3a4ee5a8b4574/1500-0-","dek":"Windows Software Update Services can use PowerShell to work with servers and Windows.","lastModified":"2018-09-04T10:00:00Z","published":"2018-09-04T10:00:00Z","sponsored":false},{"title":"How To Manage User Profiles With PowerShell","canonical":"https://www.business.com/articles/powershell-manage-user-profiles/","slug":"powershell-manage-user-profiles","thumbnail":"https://www.business.com/images/content/5cf/ffdf07b4374ac348b4576/1500-0-","dek":"Managing Windows user profiles can be time consuming and tedious. Here are some tips.","lastModified":"2018-09-06T07:00:00Z","published":"2018-09-06T07:00:00Z","sponsored":false},{"title":"How and When to Create and Use PowerShell Modules","canonical":"https://www.business.com/articles/powershell-modules/","slug":"powershell-modules","thumbnail":"https://www.business.com/images/content/5cf/a71415a215e501f8b457a/1500-0-","dek":"Step-by-step instructions on creating and using modules.","lastModified":"2018-09-05T08:00:00Z","published":"2018-09-05T08:00:00Z","sponsored":false},{"title":"What Is the Triple Bottom Line?","canonical":"https://www.business.com/articles/triple-bottom-line-defined/","slug":"triple-bottom-line-defined","thumbnail":"https://www.business.com/images/content/5e8/c6d029c658a03718b456a/1500-0-","dek":"The triple bottom line is a sustainability-based accounting method that focuses on people, profit and planet. Here's how it works for businesses.","lastModified":"2020-04-10T12:17:00Z","published":"2018-07-26T17:00:00Z","sponsored":false},{"title":"How to Use PowerShell to Work with JSON Data","canonical":"https://www.business.com/articles/using-powershell-with-json-data/","slug":"using-powershell-with-json-data","thumbnail":"https://www.business.com/images/content/5db/c5acc9c658aaa778b457a/1500-0-","dek":"When REST APIs return data in JSON format, you can get at it through PowerShell.","lastModified":"2018-09-04T07:01:00Z","published":"2018-09-04T07:00:00Z","sponsored":false}],"relatedReviews":[{"doc_type":"category","thumbnail":"https://staging.business.com/images/rz/5df908cc5a215e8c4a8b4568/conferencecall_Antonio_Diaz_Getty.jpg","canonical":"https://staging.business.com/categories/best-conference-call-services/","title":"The Best Conference Call Services of 2021","dek":"Which conference call service is right for your business in 2021? We review the top-rated services.","slug":"best-conference-call-services","lastModified":"2021-03-17T18:17:00Z"},{"doc_type":"category","thumbnail":"https://staging.business.com/images/rz/5df3d8695a215e58428b4570/POS_GETTY_DERO2084.jpg","canonical":"https://staging.business.com/categories/best-pos-systems/","title":"The Best POS Systems of 2021","dek":"Before you choose a POS system, check out our reviews of the best picks for 2021. ","slug":"best-pos-systems","lastModified":"2021-03-17T17:33:00Z"},{"doc_type":"category","thumbnail":"https://staging.business.com/images/rz/5e00d4787b43745d6a8b456e/Laptop_undrey_getty.jpg","canonical":"https://staging.business.com/categories/employee-monitoring-software/","title":"The Best Employee Monitoring Software of 2021","dek":"We've narrowed down the best employee monitoring software for businesses. Check out our best picks and reviews.","slug":"employee-monitoring-software","lastModified":"2021-03-05T15:40:00Z"}],"relatedSponsoredArticles":null}},"pageContext":{"displayModified":"2018-09-05T09:00:00.000Z","template":"v1","title":"How to Create a Web Scraping Tool in PowerShell","meta":{"robots":"index, follow"},"docType":"articles","slug":"create-web-scraping-tool-in-powershell","canonical":"https://staging.business.com/articles/create-web-scraping-tool-in-powershell/","skipTOC":false}},"staticQueryHashes":[]}![\"\"](\"/images/content/5b6/8e4e351ee473a658b45a4/1200_-390-)
![\"\"](\"/images/content/5b6/8e4f91039eb6d708b457a/1200_-872-)
![\"\"](\"/images/content/5b6/8e52b92a3a47a578b459a/1200_-556-)
![\"\"](\"/images/content/5b6/8e5411039eb1b728b4580/1200_-579-)
![\"\"](\"/images/content/5b6/8e55b51ee47e3298b458a/1200_-402-)
![\"\"](\"/images/content/5b6/8e5741039ebd9798b4576/1200_-690-)
![\"\"](\"/images/content/5b6/8e58c51ee47c8578b459c/1200_-680-)