extract urls from websites

This is a quick guide to help you pull down a list of URLs or a list of data on a web page into Excel using Octoparse. Is this the URL extractor you are looking for? Let’s see.

URL Extractor / List Extractor

I am not sure if you have an idea about what is a roundup article, but you must have read one, and most likely you have read something that you want to save for future use.

Take this article’s 100 infographic submission sites as an example. If I am an SEO marketer and one day I come across this roundup post, what would come to my mind is:

“Hey, look at this. I can pull these websites’ URLs down to a table and every time I have created a new infographic, I am going to submit it to these websites. This definitively could help boost my website traffic, or at least the number of backlinks.”

Yea, this is what the URL extractor can do. I am going to do this with a web scraping tool, Octoparse, in a few seconds.

Scrape URLs in a Web Page

This is a simple example of how you can scrape a list of URLs from a web page into Excel. Octoparse can scrape all kinds of structured data from web pages efficiently.

If you are looking to scrape other than URL data, more cases will be introduced in a video later. The video would help too if you find this textual tutorial boring.

Prerequisites

Download Octoparse and install
Sign up and log in
A target URL (example) to scrape a list of URLs from

When you enter the target URL into Octoparse, the web page will be rendered in the built-in browser. You will be able to browse it as if you are surfing on Chrome. One thing that differs from it is you can click and build a scraper while you are browsing.

Step-by-step Guide

Enter the target URL into Octoparse
Click the first hyperlink in the list
Click the second hyperlink in the list (The whole list of infographic websites will be selected in green)
Click “Extract both text and URL of the link” (Now data can be previewed in the table)
Click “Create Workflow”
Click the blue button “Run” above

That’s it. After a few clicks, you have built and run your URL extractor and get all of the 100 links into Excel for your use.

Use Auto-detection

If you find that after clicking a few pieces of data, the whole list on the web page is not selected automatically by Octoparse, maybe you need to find another method to do this.

You can try Octoparse’s auto-detection feature and let the AI algorithm select the data for you. If this is not working as well, well, the website you are scraping from is unique. It is not an average type. It has a structure, not recognizable to the bot.

In this case, you need to amend the Xpath and locate the data accurately. Curious about how to write an XPath? You are getting onboard web scraping then.

Hey, don’t worry. Just assume your website is well-structured and test it with auto-detection.

Maybe you can get more than you expect. That’s possible. The AI algorithm is not omnipotent but it is powerful enough to cover most types of web pages.

Octoparse: Boost Your Working Efficiency

If you are a digital marketer and have no idea about web scraping, this is a good chance for you to learn something new. I am a marketer and as I get hold of this web scraping tool, I collect data at a rate that I can never do manually.

That means:

You can grab articles and news for your content creation.
You can bulk download data from your competitors, always keep yourself informed.
You can pull valuable resources down to Excel and make it an actionable working plan (https://medium.com/dataseries/a-long-story-how-i-found-youtube-kols-for-marketing-purposes-5e8ef88bbb71).