How to Scrape a Website with Python: A Practical Guide to Requests, BeautifulSoup, and Playwright

Overview

A Python web scraper usually has four jobs: request a page, locate the fields you need, normalize the results, and save them in a useful format. The right implementation depends on how the target site delivers its content.

• Static HTML: Use requests to download the response and BeautifulSoup to parse it.
• JavaScript-rendered content: Use a browser automation tool such as Playwright when the required data appears only after scripts run.
• Repeated collection: Add pagination, timeouts, retry limits, logging, and a clear output schema.
• Production workflows: Separate fetching, parsing, validation, and storage so each part can be tested or replaced.

Before scraping, check the website's terms, access instructions, and applicable requirements. Prefer an official API or downloadable dataset when it provides the information you need. Keep requests limited to the pages and fields required for your legitimate use case, and avoid attempting to bypass access controls.

This process is deliberately modular. If the page layout changes, you should be able to update a selector or parser without rewriting the entire pipeline.

Step-by-step workflow

1. Define the output before writing the scraper

Start with a small data contract. For example, a product listing workflow might require name, price, url, and collected_at. Defining the fields first prevents the scraper from collecting large amounts of unstructured page content that is difficult to use later.

Also record the page type, expected pagination method, and whether the target values are present in the initial HTML. Inspect a few representative pages rather than assuming every page follows the same pattern.

2. Install the basic tools

Create an isolated Python environment and install the libraries needed for the first version:

python -m venv .venv
# Activate the environment using the command for your operating system
pip install requests beautifulsoup4 lxml

For browser-based extraction, add Playwright and install its supported browser separately:

pip install playwright
playwright install

Use a browser only when a normal HTTP request cannot provide the required content. It is generally more resource-intensive and introduces additional timing and browser-state concerns.

3. Fetch and parse a static page

Keep the request and parsing steps explicit. A short timeout, a descriptive user agent, and a check for unsuccessful responses make failures easier to diagnose.

from datetime import datetime, timezone
import requests
from bs4 import BeautifulSoup

url = 'https://example.com/articles'
headers = {'User-Agent': 'ExampleDataCollector/1.0'}
response = requests.get(url, headers=headers, timeout=20)
response.raise_for_status()

soup = BeautifulSoup(response.text, 'lxml')
rows = []
for card in soup.select('article.card'):
    link = card.select_one('a.card-link')
    title = card.select_one('.card-title')
    if not link or not title:
        continue

    rows.append({
        'title': title.get_text(' ', strip=True),
        'url': link.get('href'),
        'collected_at': datetime.now(timezone.utc).isoformat()
    })

print(rows)

The selectors in this example are placeholders. Replace them after inspecting the target markup, and keep selectors as specific as necessary without coupling them to unstable presentation details.

4. Add pagination carefully

Pagination can use a next link, numbered URLs, an offset parameter, or a cursor returned by an endpoint. A next-link loop is a useful starting point. Always include a stopping condition and track visited URLs so a malformed page cannot create an endless loop.

from urllib.parse import urljoin

session = requests.Session()
session.headers.update({'User-Agent': 'ExampleDataCollector/1.0'})
results = []
visited = set()
url = 'https://example.com/articles'

while url and url not in visited and len(results) < 500:
    visited.add(url)
    response = session.get(url, timeout=20)
    response.raise_for_status()
    soup = BeautifulSoup(response.text, 'lxml')

    for card in soup.select('article.card'):
        title = card.select_one('.card-title')
        link = card.select_one('a.card-link')
        if title and link:
            results.append({
                'title': title.get_text(' ', strip=True),
                'url': urljoin(url, link.get('href', ''))
            })

    next_link = soup.select_one('a[rel="next"]')
    url = urljoin(url, next_link['href']) if next_link and next_link.get('href') else None

The record limit is a safety guard, not a substitute for understanding the site's pagination. If the site exposes an API or structured endpoint that is appropriate for your use case, evaluate that route before crawling many rendered pages. See when an API beats a crawler for the trade-offs.

5. Handle JavaScript-rendered pages with Playwright

If the data is absent from the response HTML, a browser automation workflow may be appropriate. Playwright can load the page, wait for a meaningful element, and then pass the rendered HTML to BeautifulSoup.

from bs4 import BeautifulSoup
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto('https://example.com/catalog', wait_until='domcontentloaded', timeout=30000)
    page.locator('article.card').first.wait_for(timeout=15000)

    soup = BeautifulSoup(page.content(), 'lxml')
    items = []
    for card in soup.select('article.card'):
        title = card.select_one('.card-title')
        if title:
            items.append({'title': title.get_text(' ', strip=True)})

    browser.close()

Wait for a content selector rather than relying only on a fixed sleep. A selector-based wait expresses what the scraper actually needs and is usually easier to maintain. For a deeper Playwright workflow, read this guide to JavaScript-rendered websites.

Tools and handoffs

A maintainable scraper benefits from clear boundaries between tools:

1. Requests or a session: Fetches static pages and reuses connection settings.
2. BeautifulSoup: Converts HTML into a searchable tree and extracts fields.
3. Playwright: Handles pages that require browser execution, interaction, or rendered state.
4. Python data structures: Hold normalized records before export.
5. CSV, JSON, SQLite, or a database: Stores results for analysis and downstream automation.

For static pages, keep the handoff simple: fetch HTML, parse it, validate records, and write output. For browser workflows, isolate browser startup and page interaction from the parser. This lets you reuse the same parsing logic when a site offers both rendered pages and a server response with equivalent markup.

JSON is convenient for nested records and API handoffs, while CSV works well for flat tables. SQLite is useful when you need local querying, deduplication, or incremental runs without managing a separate database. Review storage options for scraped data before choosing an output format.

When a project grows, put selectors in a dedicated parser module, configuration such as start URLs and limits in a settings file, and credentials outside the source code. Store logs with the run date, URL, response status, and parsing counts. These details make a failed scheduled job much easier to investigate.

Quality checks

A scraper that runs without an exception can still produce poor data. Add checks that compare the output with reasonable expectations:

• Confirm that each required field exists and is not just whitespace.
• Check that URLs are absolute and use the expected scheme.
• Parse numeric values consistently and preserve the original text when conversion may lose context.
• Track the number of pages requested, records found, and records rejected.
• Detect duplicate URLs or identifiers before saving.
• Save a small sample of raw HTML when a parser failure needs investigation.
• Compare record counts with previous runs and flag large unexplained changes.

Use retries selectively. A temporary connection failure may justify another request, but repeated parser failures usually indicate a markup or workflow change and should be surfaced rather than hidden. Add a delay between requests and use conservative concurrency. Rate limiting is part of reliability: it reduces load on the target and helps keep your own pipeline predictable. If your use case requires proxy rotation, understand its operational and compliance implications; the proxy rotation guide covers implementation considerations.

Test parsers against saved HTML fixtures. A fixture-based test can verify that a selector still extracts the expected fields without making a live request every time the code changes. After extraction, apply a separate cleaning step for trimming, deduplication, normalization, and validation. The Python data-cleaning guide provides a useful next stage.

When to revisit

Revisit this workflow whenever the target site's structure, delivery method, or data requirements change. A page redesign may invalidate CSS selectors; a move from server-rendered HTML to client-rendered content may require Playwright; and a new pagination method may change how the crawler discovers pages.

Set a review trigger around observable signals rather than an arbitrary schedule. Investigate when a run returns zero records, required fields become empty, response times change sharply, duplicate rates rise, or the number of pages differs substantially from the normal range. Also review the workflow when you add a new field, change the destination schema, or move from a small manual run to scheduled automation.

For your next implementation, begin with one page and three or four fields. Save the raw response, write a parser test, add a small page limit, and export a sample JSON file. Then add pagination, validation, logging, and storage one stage at a time. This sequence gives you a working baseline and makes future updates safer than building a large crawler before you know how the target behaves.