Four steps to build a Glassdoor scraper

Known as a leading platform for job seekers, employers, and recruiters alike, Glassdoor provides a wealth of information about companies and the labor market. With over 50 million unique monthly visitors and 2.4 million employer clients, it’s a valuable database for market insights. This article will walk you through the process of Glassdoor scraping and utilizing tools like Octoparse for a smooth scraping experience.

What Glassdoor Data is Worth Scraping

When it comes to Glassdoor, there’s a plethora of data available for extraction. A few of the most valuable data points include:

Company reviews and ratings

Company reviews on Glassdoor are generated by current and former employees of these companies. Thus, such information can let you develop trustworthy insights into the company’s work culture.

Salary information

Glassdoor provides detailed information on salary and compensation for various positions within companies. The salary information can help you understand the trends of the labor market and even identify emerging industries.

Interview questions and reviews

Users on Glassdoor have an interest in sharing their interview experiences. You can gain insights into the company’s interview process, the questions asked, and the overall experience of applicants.

Job listings

The job listings on Glassdoor contain detailed information about job openings, including roles, job titles, locations, job descriptions, etc., which is valuable for both job seekers and employers.

Does Glassdoor Have an API

The answer is YES! Glassdoor has an official API but it is now only available to their API partners. If you’re interested in using it, you need to contact them for permission. The Glassdoor API can be used for a variety of purposes.

Job search and recruitment: You can use the API to retrieve job listings and company information on Glassdoor, then integrate such information into job aggregators;
Employee engagement and retention: The API can access employee reviews and ratings, which can help you and your company understand job seekers and employees, and eventually identify where to improve;
Market research: The API can be applied to access company and industry data. With such data, you can develop a deep into the market and analyze the data to identify your competitors;
Branding and reputation management: Brand images are important for most companies. The API allows businesses to collect company reviews and ratings, then track companies’ reputations and identify areas for improvement.

However, using the Glassdoor API might take a long time to apply for an API key. Writing a script or building a scraper may be more effective for most users.

How to Scrape Glassdoor Data Using Python

Thanks to many free Python libraries, people can now build Glassdoor scrapers with Python at ease. If you’re good at coding, using Python to pull data from Glassdoor can always be a considerable and affordable choice. Taking Glassdoor salary information as an example, below is a sample of code to extract Glassdoor data with a Python library called BeautifulSoup.

import requests

from bs4 import BeautifulSoup

url = 'https://www.glassdoor.com/Salaries/data-scientist-salary-SRCH_KO0,16.htm'

response = requests.get(url)

soup = BeautifulSoup(response.text, 'html.parser')

salaries = soup.find_all('div', class_='salarySnippet')

for salary in salaries:

name = salary.find('h3').text

location = salary.find('span', class_='location').text

salary_range = salary.find('span', class_='value').text

print(f'Job Title: {name}')

print(f'Location: {location}')

print(f'Salary Range: {salary_range}')

print(f'Scraped {len(salaries)} salaries from Glassdoor')

Scrape Glassdoor Salary Data with Octoparse

If you’re not good at coding, writing a script to scrape Glassdoor might have a steep learning curve. Well, don’t worry, there is an easy-to-use and no-coding web scraping solution for you. Octoparse is the AI web scraping assistant for anyone regardless of coding skills.

If you don’t have Octoparse on your computer, download Octoparse for free and install it on your device before we begin. When you open the software for the first time, you’ll need an account to log in. You can sign up for a new account or log in directly with your Google or Microsoft account. You can then follow the step-by-step guide below to scrape Glassdoor salary in minutes.

Step 1: Create a new task to scrape Glassdoor salary data

Copy the URL of the page you want to pull salary information from and paste it into the search bar on Octoparse. Next, click the “Start” button to create a task.

create a new task for scraping salary data on Glassdoor

Step 2: Auto-detect Glassdoor salary data

After creating a task successfully, the built-in browser will navigate to the target page. Wait for the page to finish loading, then click “Auto-detect webpage data” in the Tips panel. Auto-detection is an advanced feature to scan the page and detect extractable data automatically. Therefore, you don’t need to select the desired data on the page or read HTML files to locate data.

The auto-detection process will take seconds, then it’ll highlight all the detected data in the browser. You can check if your desired data is selected on the page, or check all extractable data in the “Data Preview” panel at the bottom. You can also edit detected data fields and remove unwanted fields or change their names in this panel. For example, we only need job titles and salary information now, so we only keep these two data fields.

Step 3: Create and modify the workflow

Click “Create workflow” after selecting all the wanted data. Then a workflow will show up on your right-hand side. It’s a flow chart that shows how this scraper runs. Before launching the scraper, you can click on each action on the chart to see if it runs as expected.

Step 4: Run the task and export scraped salary data

Click “Run” to launch the scraper after verifying all the details, such as selecting all the needed data fields. Next, select the option to run the task. If you’re working on a small project, running it on your device will be a good choice. But if the project is large and you need up-to-date data from the project, you can run it on Octoparse cloud servers because the cloud platform can work for you around the clock.

When the scraping process is complete, you can export scraped salary data as an Excel, CSV, or JSON file, even to a database like Google Sheets for further use.

Preset template for Glassdoor scraping

Another easy way to scrape data from Glassdoor is using a data scraping template. Octoparse provides online preset template to extract job listing data like description, title, salary, location, etc. from Glassdoor. Just click on the link below and enter asked parameters to start scraping.

https://www.octoparse.com/template/glassdoor-scraper

Wrap Up

Glassdoor is a gold mine of data. For people who want to gain insight into companies, it allows them to get first-hand information in various aspects. By following this post, we believe you’ve built a Glassdoor salary scraper. Of course, this guide can also be applied to pull data from many other websites. Try Octoparse now and simplify your web scraping process from now on!