Black Friday is around the corner, and the deals are rolling. You might notice that many retailers have been promoting their early Black Friday deals since the end of October. Products prices are fast-changing at this time.
How do you make sure your pricing is the most appealing among all your rivals to customers? The first thing you need to do is keep an eye on the prices of your competitors.
In this article, we’ll show you how to collect products’ prices on Amazon with Octoparse. As a no-code data scraper, Octoparse users can collect data from webpages with only a few clicks regardless of coding skills.
Why Monitor Prices on Amazon
The quick answer is that Amazon, which is one of the biggest e-commerce platforms today, has countless product listings, making it a huge product database for all kinds of market research. You can find almost everything you need in daily life, and gain price information along with products’ size, materials, textures, prices, etc., there.
Speaking of Black Friday, Amazon had won the event back in 2021 by capturing 17.7% of all sales during the shopping frenzy. This year Amazon launched Black Friday deals in October with the Prime Early Access Sale. The Black Friday sales will last for more than a month, so there is a higher chance that products will be re-priced during this period. Naturally, if you want your products to succeed, pricing monitoring becomes even more crucial.
5 Steps to Scrape Data from Amazon
Taking Christmas mugs as an example, we’ll use Octoparse to collect data for this particular product type on Amazon, including basic information from the listing page, and additional information from product’s detail page. Here is the URL of the target page we’ll be using:
https://www.amazon.com/s?k=christmas+mugs&ref=nb_sb_noss
To download Octoparse, you can go to octoparse.com to get the latest version. When the installation is complete, sign up for a free account to start your data scraping journey with Octoparse.
Step 1: Create a new task with the target link
Enter the target URL into the search bar, then click the Start button. The page will be loaded in the Octoparse built-in browser after seconds.
Step 2: Select the data we need from the listing page
After the page has finished loading, click the “Auto-detect website data” button in the Tips panel. Octoparse will scan the page for data that you are probably looking for. You can click “switch auto-detect results” on the Action Tips to see if Octoparse has successfully detected the data you need. The detected data will be highlighted in red and also shown in the Data Preview pane below. In this case, you can keep the fields for product title, price, rating, and the number of reviews.
Meanwhile, Octoparse automatically detects the next-page buttons on the target page and asks you if you would like to paginate to scrape more pages. Go ahead and leave the option checked so you’ll get data from page 1 to the last page. Then, click Create workflow on the Action Tips.
Step 3: Go to the product detail pages
To grab additional data from detail pages, click “Click on link(s) to scrape the linked page(s)”. It will ask you to choose a list from data fields to click on. In most situations, you can select the URL field directly to proceed, for Amazon, however, things are a bit trickier and we’d need to add a Click action manually to the workflow.
Hover under the Extract Data action, click on the + sign and add a Click action. Rename the action to “Click URLs in the list”, select Relative XPath to the Loop Item under Matching XPath and paste the XPath below into the bar. Switch to Options menu and select Open in a new tab under Before action is performed. Click the Apply button to save the change.
The correct XPath to locate the product URLs on the listing page:
//div[contains(@class,’s-card-container’)]//*[contains(@class,’title’)]//a
Quick Tips:
1. You can also build steps to go to detail pages manually by adding an action to click on the product title URL without finding or writing XPath by yourself.
2. Web pages structures can be complicated and ever-changing, which might lead to inaccurate data detection. You can use XPath to locate the exact elements you want on the page. It is a query language which needs some basic knowledge of HTML language. To learn more, please check XPath using guide.
Step 4: Grab data from the detail pages
You can launch the auto-detection process as Step 3, or select the required data on the page and then click “Extract text of the selected element” in the Tips panel. We select brand and capacity for this step.
Till this step, we’ve built the complete workflow which is shown on the right-hand side. You should click through the steps of the workflow to check if every step works as desired.
Step 5: Scrape and export data
After verifying all the details (select every data we need and check the workflow runs properly), click Run to start the scraping process. Then Octoparse will handle the data extraction for you. Once the run is finished, you can export the data as an Excel, CSV, or JSON file.
A quick tip: because products’ prices are fast-changing, scraping data frequently will be helpful in making your strategy more dynamic. You can reuse the task saved in your Dashboard to run the workflow and grab the same data, or you can schedule to run the task automatically.
How the Scraped Data Can Help in Pricing
Now that you’ve got the data in structured format, it’s time to do some quick data analysis. For pricing purposes, there are a couple of things we can try.
Track product variants pricing
There are diverse factors that can affect a product’s price. Products of different sizes, materials, textures and even brands often have different prices. With the scraped data, you can easily carry out an analysis with a lot of data to monitor the prices of the various products based on a particular set of factors.
For example, as we’ve scraped products’ sizes and brands for the mugs, by comparing prices of mugs that share the same size but not the same brand, you can find out which brands have set the highest and lowest prices. If you need, you can also scrape other data like colors and material from detail pages to compare. The more details you get, the more specific you will achieve with your analysis, and can better help you price your products correctly to boost sales.
Track prices over time
This approach can be used against a single product or a group of products during a specific period. For instance, Black Friday deals this year will last over a month on Amazon. Your competitors may start out with higher pricing before dropping them later with a discount to catch people’s eyes. How can you keep track of the price changes and make sure your products are not overpriced/underpriced? Scraping price data frequently and doing horizontal analysis is the lighthouse.
Besides only looking at the price trend of various goods separately, we still have room to improve. For instance, you can group products into different brands to observe the changing trends of mug prices under the same brand. It also gives you a new perspective to monitor prices and develop a more targeted pricing plan.
Go beyond Amazon
Besides scraping data from Amazon, you can also apply data scraping to other e-commerce platforms. There are a variety of data available for scraping, not just names and product prices, but also detailed information like sizes, shelf life, producers, reviews, etc. Even data like the number of sales and watches are extractable on some platforms. You can then identify more attractive pricing based on this information.
Wrap-up
Pricing is dynamic in most situations. Depending on the season, market trends, and even the weather, product prices are always changing. Price monitoring gives you a good basis to improve your pricing strategies and gain insights into your market and competitors.
