How to Scrape Google Product Reviews Results with Python

s the digital wave sweeps the world, Google, as one of the world's largest search engines, hides a huge amount of valuable data in its search results page (SERP). This data is not just a simple list of information, but also the key to insight into market dynamics, analysis of competitors, and understanding of consumer behavior.

However, it is not easy to efficiently and effectively crawl Google search results. The page structure of Google Product is complex and changeable, and it has a powerful anti-crawler mechanism, which makes traditional data crawling methods often difficult to work. In order to meet these challenges, a powerful, scalable and flexible data pipeline that can adapt to Google's structural changes is particularly important. Whether you plan to build your own large language model (LLM) or want to get first-hand consumer insights from the market, a reliable Google search crawler is indispensable.

In this article, we will focus on how to use Python and the BeautifulSoup library to build a Google product review result crawler from scratch. With this tool, you will be able to automatically extract valuable data and dig out insights that can be directly used for decision-making from the search engine's ocean of data.

Use Cases of Scraping Google Product Results

Scraping Google product results has become an important means for companies and researchers to gain market insights. By analyzing this data, companies can understand market trends, consumer preferences, and competitor dynamics, thereby developing more effective business strategies. The following are some common usage scenarios:

• Market research and trend analysis.
• Competitor analysis
• Product development and optimization
• Consumer behavior insights
• Price monitoring and adjustment
• Brand protection and reputation management
• E-commerce and online retail optimization
• Academic research and data analysis
• Public policy formulation and supervision

Difficulties in crawling Google product reviews

Extracting Google product reviews presents several challenges, including the following:

Anti-Scraping Measures

• Google uses CAPTCHA and IP blocking to prevent bots.
• Requires proxies and user-agent rotation to bypass.

Dynamic & JavaScript-Rendered Content

• Reviews load dynamically via JavaScript.
• Needs tools like Puppeteer or Selenium to extract data.

Frequent DOM Structure Changes

• Google updates its HTML structure regularly.
• Scrapers require constant maintenance.

Method 1: Scrape Google Product Reviews Results with Scrapeless

Scraping Google product reviews in Python requires handling challenges like dynamic content loading and anti-scraping mechanisms. This section explores effective methods and tools to extract review data efficiently.

Step 1: Build a Google Product data crawling environment

First, we need to build a data crawling environment and prepare the following tools:

1. Python: This is the core software for running Python. You can download the version we need from the official website link, as shown in the figure below, but it is recommended not to download the latest version. You can download 1-2 versions before the latest version.

2. Python IDE: Any IDE that supports Python will do, but we recommend PyCharm, which is an IDE development tool software designed specifically for Python. Regarding the PyCharm version, we recommend the free PyCharm Community Edition.

3. Pip: You can use the Python Package Index to install the libraries you need to run your programs with a single command.

Note: If you are a Windows user, don't forget to check the "Add python.exe to PATH" option in the installation wizard. This will allow Windows to use Python and commands in the terminal. Since Python 3.4 or later includes it by default, you don't need to install it manually.

Through the above steps, the environment for crawling Google Product data is set up. Next, you can use the downloaded PyCharm combined with Scraperless to crawl Google Product data.

Step 2: Use PyCharm and Scrapeless to scrape Google Product data

1. Launch PyCharm and select File>New Project… from the menu bar.

2. Then, in the window that pops up, select Pure Python from the left menu and set up your project as follows:

Note: In the red box below, select the Python installation path downloaded in the first step of environment configuration

3. You can create a project called python-scraper, check the "Create main.py welcome script option in the folder" and click the "Create" button. After PyCharm sets up the project for a while, you should see the following:

4. Then, right-click to create a new Python file.

5. To verify that everything is working correctly, open the Terminal tab at the bottom of the screen and type: python main.py. After launching this command you should get: Hi, PyCharm.

Step 3: Get Scrapeless API Key

Now you can directly copy the Scrapeless code into PyCharm and run it, so you can get the JSON format data of Google Job. However, you need to get the Scrapeless API key first. The steps are as follows:

• If you don’t have an account yet, please sign up for Scrapeless. After signing up, log in to your dashboard.
• In your Scrapeless dashboard, navigate to API Key Management and click on Create API Key. You will get your API Key. Just put your mouse on it and click it to copy it. This key will be used to authenticate your request when calling the Scrapeless API.

We firmly protect the privacy of the website. All data in this blog is public and is only used as a demonstration of the crawling process. We do not save any information and data.

Join Scrapeless and Get 20,000 Free Queries!

Sign up for Scrapeless today and enjoy 20,000 free queries on us! Start crawling Google product reviews and unlock valuable insights with ease. Don’t miss out—register now and experience the power of Scrapeless for free!

Step 4: How to integrate the Scrapeless API into your scraping tool

Once you have the API key, you can start integrating the Scrapeless API into your own scraping tool. Here is an example of how to call the Scrapeless API and retrieve data using Python and requests.

Sample code for crawling Google Product information using the Scrapeless API:

import json
import requests

class Payload:
    def __init__(self, actor, input_data):
        self.actor = actor
        self.input = input_data

def send_request():
    host = "api.scrapeless.com"
    url = f"https://{host}/api/v1/scraper/request"
    token = "your_token"

    headers = {
        "x-api-token": token
    }

    input_data = {
        "engine": "google_product",
        "product_id": "4172129135583325756",
        "gl": "us",
        "hl": "en",
    }

    payload = Payload("scraper.google.product", input_data)

    json_payload = json.dumps(payload.__dict__)

    response = requests.post(url, headers=headers, data=json_payload)

    if response.status_code != 200:
        print("Error:", response.status_code, response.text)
        return

    print("body", response.text)


if __name__ == "__main__":
    send_request()

Based on the output of this code, you can find all the information about Google product reviews.

You can also refer to the official Scrapeless API parameter documentation

The following is a partial list of results, which includes user comment information.

body {"product_results":{"product_id":"4172129135583325756","title":"Apple iPhone 12 Pro - 128 GB - Silver - AT\u0026T","price":["$314.84","$298.00","$325.33"],"reviews":14303,"rating":4.4,"extensions":["Smartphone","Dual SIM","5G","With Wireless Charging","With Fast Charging","AT\u0026T","Dual Lens","iOS","GSM","CDMA"],"description":"5G goes Pro. A14 Bionic rockets past every other smartphone chip. The iPhone 12 Pro features a 6.1” Super Retina XDR display, LiDAR scanner for ultrafast and accurate depth maps of whatever space you're in and MagSafe wireless charging. The Pro camera system takes low-light ...More5G goes Pro. A14 Bionic rockets past every other smartphone chip. The iPhone 12 Pro features a 6.1” Super Retina XDR display, LiDAR scanner for ultrafast and accurate depth maps of whatever space you're in and MagSafe wireless charging. The Pro camera system takes low-light photography to the next level with Night mode available in both the Wide and Ultra Wide cameras, so it’s better than ever at capturing incredible low-light shots. And Ceramic Shield delivers four times better drop performance.Less","media":[{"type":"image","link":"https://encrypted-tbn0.gstatic.com/shopping?q=tbn:ANd9GcStw-jdTZtGmdXcVKCqweq6wxzU5tpRTTbl6stPV97GpGVR6XY\u0026usqp=CAY"},{"type":"image","link":"https://encrypted-tbn1.gstatic.com/shopping?q=tbn:ANd9GcR0wJ1fsUOPAGDMtjdtx1zsd5ZWUXwnNe70fmZszERkEihkYCKnZoGJ3Y4lqSQTyR4soiTVWFVzllzYTHJBTRXegTR7Pj83RA\u0026usqp=CAY"},{"type":"image","link":"https://encrypted-tbn1.gstatic.com/shopping?q=tbn:ANd9GcR4L66Gss9O5HSL00NLxaHu0pl5huMUojbC9tO9FKCRpCQObUqdHWsSPYZJ4lU8eETn-MlJx4Hni_oc_l5mxIs_l-Z2htBiaA\u0026usqp=CAY"},{"type":"image","link":"https://encrypted-tbn0.gstatic.com/shopping?q=tbn:ANd9GcQLv5xOi-9b-Mka7jfFnQzlXkTrEAsjPAzumbUB2D6Ddgl3FHGZOQXAUGQAv6WkUeZsbsdvKA2NRF1-h8EOBSQPLmuPMLPQ2Q\u0026usqp=CAY"}],"sizes":{"128 GB":{"link":"https://www.google.com/shopping/product/4172129135583325756?gl=us\u0026hl=en\u0026sourceid=chrome\u0026ie=UTF-8","product_id":"4172129135583325756"},"256 GB":{"link":"https://www.google.com/shopping/product/1700752269234454309?gl=us\u0026hl=en\u0026sourceid=chrome\u0026ie=UTF-8\u0026prds=opd:11579479524734831751,rsk:PC_14243855303706753583\u0026sa=X\u0026ved=0ahUKEwjxhNXT2ZeMAxVcK7kGHUh1ErMQlIUHCEQoAQ","product_id":"1700752269234454309"},"512 GB":{"link":"https://www.google.com/shopping/product/14752474427020499512?gl=us\u0026hl=en\u0026sourceid=chrome\u0026ie=UTF-8\u0026prds=opd:11579479524734831751,rsk:PC_14243855303706753583\u0026sa=X\u0026ved=0ahUKEwjxhNXT2ZeMAxVcK7kGHUh1ErMQlIUHCEUoAg","product_id":"14752474427020499512"}},"highlight":["5G transforms iPhone with accelerated wireless speeds and better performance on congested networks","A14 Bionic: generations ahead of any other smartphone chip","Night mode comes to both the Wide and Ultra Wide cameras, and it's better than ever at capturing incredible low-light shots",

Why use Scrapeless to crawl Google product reviews

• Affordable Pricing: At just $0.1 per 1,000 queries, Scrapeless offers a highly cost-effective solution for businesses that need to scale data collection without breaking the budget.
• Fast and Reliable: With a response time of under 3 seconds, Scrapeless delivers real-time results, ensuring you get the data you need quickly and efficiently—essential for fast-paced business environments.
• Easy to Use: Scrapeless features an intuitive interface that allows teams with limited technical expertise to get started with minimal setup, reducing the need for extensive training.
• Scalable Data Collection: Whether you're collecting data for a small project or enterprise-level analysis, Scrapeless scales with your business needs, allowing you to handle high volumes of queries without a hitch.
• Customizable Scraping: Scrapeless allows you to tailor your data extraction to specific needs—whether you’re targeting particular products, regions, or review types, giving you flexibility to gather the most relevant insights.
• Enhance Decision-Making: Scrapeless helps businesses gain valuable insights from Google product reviews, aiding in customer sentiment analysis and product improvement decisions. This is particularly crucial for businesses looking to stay competitive.
• Seamless Integration: With options to integrate Scrapeless with other business tools, your team can directly analyze collected data within your existing systems, making it easier to act on insights quickly.

Join Our Discord Community and Connect with TOB Clients!

Join our Discord community to network with other TOB customers, share insights, and discuss strategies. Plus, get direct access to our team for personalized support—whether you need custom solutions, data testing, or guidance. Don’t miss this opportunity to enhance your Scrapeless experience!

How to scrape Google product reviews (using Scrapeless Playground)

Step 1: If you haven't registered Scrapeless yet, you can register now and log in to the Scrapeless Dashboard.

Step 2: Click Deep Serpapi and select Google products.

In addition, Scrapeless Serpapi also supports the crawling of information such as Google Hotels, Google Search, Google Maps, Google Job, and Google Shopping.

Step 3: Set the Google product ID. The product ID is obtained as follows:

Step 4: Click "Start Search" and the crawling results will be displayed on the right. You can get user review information through the "ReviewsResults" field.

Method 2: Google Product Reviews Scraping using Python and Selenium

Prerequisites

Before we begin, ensure you have the following:

• Python installed (3.x recommended)
• Google Chrome and ChromeDriver
• Required Python libraries:

pip install selenium beautifulsoup4 pandas

Step 1: Set Up Selenium WebDriver

Selenium allows us to automate web interactions. First, we need to initialize the Chrome WebDriver:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
import time
service = Service('path/to/chromedriver')  # Update with the correct path
options = webdriver.ChromeOptions()
options.add_argument('--headless')  # Run in headless mode
driver = webdriver.Chrome(service=service, options=options)
driver.get('https://www.google.com/shopping/product/1234567890/reviews')  # Example URL

Step 2: Extract Review Data

We use BeautifulSoup to parse the page source and extract review details.

from bs4 import BeautifulSoup
def extract_reviews(driver):
    soup = BeautifulSoup(driver.page_source, 'html.parser')
    reviews = []

    for review in soup.find_all('div', class_='sh-dgr__content'):
        rating = review.find('div', class_='sh-dgr__rating')
        text = review.find('div', class_='sh-dgr__review-text')

        if rating and text:
            reviews.append({
                'rating': rating.text.strip(),
                'review': text.text.strip()
            })
    return reviews
data = extract_reviews(driver)
print(data)

Step 3: Automate Pagination

If there are multiple pages of reviews, we need to automate pagination.

def scrape_multiple_pages(driver):
    all_reviews = []
    while True:
        all_reviews.extend(extract_reviews(driver))

        try:
            next_button = driver.find_element(By.XPATH, '//a[@aria-label="Next page"]')
            next_button.click()
            time.sleep(2)  # Wait for the new page to load
        except:
            break
    return all_reviews
data = scrape_multiple_pages(driver)
print(f'Total Reviews Scraped: {len(data)}')

Step 4: Save Data to CSV

Once we have collected the data, we can save it for further analysis.

import pandas as pd

df = pd.DataFrame(data)
df.to_csv('google_reviews.csv', index=False)
print('Reviews saved to google_reviews.csv')

Comparison: Scrapeless vs. Selenium + BeautifulSoup for Scraping Google Product Reviews

Here’s a comparison table highlighting why Scrapeless is the better choice for scraping Google product reviews:

Conclusion

Scrapeless, Selenium, and Scrapy are all viable solutions for scraping Google product review data, but each has its own pros and cons:

• Scrapeless is the simplest and most efficient option, especially for scenarios where you need to quickly obtain structured data. It can automatically handle anti-crawling mechanisms and dynamic content loading.
• Selenium provides powerful dynamic content processing capabilities, but has low performance, high maintenance costs, and is easy to detect.
• Scrapy is an efficient crawler framework that is suitable for large-scale data crawling, but it is difficult to handle dynamic content and has a steep learning curve.

All in all, if you want to quickly, efficiently, and stably obtain Google product review data, Scrapeless is the best choice. It not only simplifies the scraping process, but also avoids complex development and maintenance work. When using any tool, make sure to comply with Google's Terms of Service to avoid unnecessary legal risks.

Ready to Take Your Data Scraping to the Next Level?

Don’t let your business fall behind! Register for Scrapeless today and start scraping Google product reviews effortlessly. With just a few simple API calls, you can access valuable insights to improve your product development and customer engagement. Plus, Scrapeless isn’t just limited to reviews—use it to gather data across various platforms, analyze competitors, track trends, and more!

Join now and get 20,000 free queries to explore all the powerful features Scrapeless has to offer. Whether you’re in e-commerce, marketing, or research, Scrapeless is your go-to tool for efficient, scalable, and customizable data extraction.

Sign up now and see the difference it makes for your business!

More Resources

You may also be interested in the following data capture, welcome to discuss with us.

• How to Scrape Google Play Store App in Python
• How to Use Undetected ChromeDriver for Web Scraping
• How to Scrape Google Finance Ticker Quote Data in Python
• How to Bypass Cloudflare With Puppeteer