Can you believe that 70% of Internet traffic was created by spiders*? It is shockingly true! There are a lot of spiders, web crawlers, or searching bots busy with their jobs on the Internet. They simulate human behavior, walking around websites, clicking buttons, checking data, and bringing back information. They are powerful, but how to make money with them?
With so much traffic generated, they must have achieved something magnificent. Some of them may sound familiar to you, like price monitoring in e-Commerce businesses, social media monitoring in public relations, or research data acquisition for academic studies.
Here, we’d like to dig into the 3 web-scraping applications that are rarely known but are surprisingly profitable that many freelancers are making money with them.
1. Making Money Through Ticketing
Budget Airline platforms are very popular among web scrapers because of the unpredictable promotion of cheap tickets.
The airline’s original intention is to offer cheap tickets at random to attract tourists*, but scalpers found a way to make a profit out of this. The geek scalper uses web crawlers to refresh the airline website continuously. Once a cheap ticket is available, the crawler will book the ticket. Is Scalping Legal?
AirAsia, for example, only keeps the reservation for an hour and if the payment is not made by then, the ticket gets sent back to the ticket pool and is ready to sell. Scalpers will buy the ticket again at the millisecond after the ticket returns to the pool, and so on and so forth. Only when you order the ticket from a scalper, then the scalpers will use the program to abandon the ticket in the AirAsia system and in 0.00001 seconds later, book it for you with your name.
It doesn’t sound like an easy job for me. Maybe the intermediary fee is quite fair.
2. Making Money through Saving
There must be plenty of price comparison sites and cashback websites in a smart shopper’s favorites. No matter what their names are, “price comparison platforms”, “e-commerce aggregate websites” or “coupon websites”, the ideas are the same, earning money by saving your money.
They scrape the prices and pictures from e-commerce websites and display them on their own websites.
E-Commerce moguls like Amazon know that it’s hard to reverse the trends of open source*. So they start a business to sell their API with a long pricing list. It sounds like the pricing comparison websites need to do all the jobs, write codes to scrap, provide a sales channel, and pay the bills!
Not so fast. Let me tell you how these aggregate e-commerce platforms make their profits:
- Suppose there is a platform that aggregates several stores selling Okamoto. When people search for the product, they get to decide which store ranks first and which comes last. So long story short, whoever pays more gets ranked at the top.
- If the bid for ranking is so painful (@Google AdWords), you can also take the easy way out – buy ads on the web page. Each time a visitor clicks, the website makes money.
- The website can also act as an agent to earn commission fees. It’s quite easy to understand as they help stores to sell goods. Based on the belief of “the more, the merrier”, there we have cashback websites coming into being.
To sum up, they are doing quite well. In fact, a price comparison website can be categorized as an aggregator website and their business model is similar in most cases.
3. Making Money with Data Services
For two companies who agree to share databases through API, they might still need web scraping to turn data on web pages into structural data reports.
Let’s take the big three as an example. Equifax, Experian, and TransUnion hold the credit files of 170 million American adults and sell more than 600 million credit reports each year, generating more than $10 billion in revenue*.
“It’s a pretty simple business model, actually. They gather as much information about you from lenders, aggregate it, and sell it back to them,” said Brett Horn, an industry analyst with Morningstar*. They receive your personal behavior data from your banks, employer, local court, shopping malls, and even a petrol station. With so many reports to analyze, web scraping is a big helper to organize the data. They can turn web pages into structural data reports.
There are many ways you can scrape from websites. If you want to scrape data at scale from a lot of websites, a web scraping tool comes in handy. Here is a list of the top 10 web scraping tools as per preference.
Web scraping is an incredible way to collect data for your business. If you are looking for a reliable web scraping service to scrape data from the web, you can try to start with Octoparse.
