Python Web Scraping Tutorial: Requests, Beautiful Soup, and Playwright

Overview

The quickest way to make web scraping harder than it needs to be is to use browser automation for every task. The quickest way to make it unreliable is to ignore dynamic content and modern frontend behavior. A good scraper sits between those extremes.

For most projects, Python gives you a clear progression:

• Use requests when the page HTML already contains the data you need.
• Use Beautiful Soup to parse that HTML and extract structured fields.
• Use Playwright when the target relies on client-side rendering, authenticated sessions, scrolling, or user interaction.

That progression is useful because it keeps your approach proportional. Not every page needs a browser. Not every browser task needs a full scraping framework. If your goal is maintainability, start simple and move up only when the site demands it.

A basic Python scraping stack often looks like this:

pip install requests beautifulsoup4 lxml playwright
playwright install

Once installed, think of the stack in layers:

• Fetching: getting a response from a URL
• Parsing: turning raw HTML into searchable elements
• Extraction: selecting fields like title, price, date, link, or body text
• Validation: checking whether the data is complete and still matches expectations
• Persistence: saving to CSV, JSON, a database, or an API

Here is the simplest possible example using requests and Beautiful Soup:

import requests
from bs4 import BeautifulSoup

url = "https://example.com"
headers = {
    "User-Agent": "Mozilla/5.0"
}

response = requests.get(url, headers=headers, timeout=20)
response.raise_for_status()

soup = BeautifulSoup(response.text, "lxml")
page_title = soup.title.get_text(strip=True) if soup.title else ""

print(page_title)

This pattern is enough for many documentation pages, blogs, listings rendered on the server, and simple directories. The main beginner mistake is to stop here and assume every site behaves the same way. Before writing extraction logic, inspect the response body and confirm that the data you want is actually present in the HTML returned by requests.

If the content is missing, loaded after page render, or appears only after a click or scroll, move to Playwright. A minimal Playwright Python scraping example looks like this:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com", wait_until="networkidle")
    title = page.title()
    print(title)
    browser.close()

The difference is significant: Playwright runs a real browser context, waits for scripts to execute, and can interact with the page. That makes it more powerful, but also slower and more complex. In practice, the best Python web scraping guide is one that helps you know when not to use it.

As you build, keep a small checklist nearby:

• Can I get the data with plain HTTP?
• Is the HTML stable enough for CSS selectors?
• Does the page load content dynamically?
• Do I need login, clicks, scroll, or pagination?
• How will I know when the scraper breaks?

That last question matters more than many tutorials admit. A scraper that succeeds once is a script. A scraper you can trust next month is a maintained tool.

If you are evaluating broader options beyond Python libraries, see Best Web Scraping Tools in 2026: Features, Pricing, and Use Cases for a wider tool-selection view.

Maintenance cycle

The most useful way to think about scraping is not as a one-time extraction, but as a workflow with a review cycle. Libraries evolve, sites redesign layouts, selectors drift, and access patterns change. A maintenance-oriented scraping setup reduces emergency fixes.

A sensible maintenance cycle has five parts.

1. Capture the page contract

Before you write too much code, define what your scraper assumes:

• The target URL pattern
• The selectors used for each field
• Required request headers or cookies
• Pagination rules
• Expected output schema

This can live in comments, a README, or a small config file. The goal is to document what “working” means. If the page title comes from h1.product-title today, write that down. If listings are loaded by an API call the browser makes after page load, note that too.

2. Separate fetching from parsing

In a healthy scraper, network logic and HTML parsing are not tangled together. That makes debugging much easier.

import requests
from bs4 import BeautifulSoup

def fetch_html(url):
    headers = {"User-Agent": "Mozilla/5.0"}
    response = requests.get(url, headers=headers, timeout=20)
    response.raise_for_status()
    return response.text

def parse_article(html):
    soup = BeautifulSoup(html, "lxml")
    return {
        "title": soup.select_one("h1").get_text(strip=True) if soup.select_one("h1") else None,
        "description": soup.select_one("meta[name='description']")["content"] if soup.select_one("meta[name='description']") else None
    }

html = fetch_html("https://example.com")
data = parse_article(html)
print(data)

With this structure, you can save sample HTML locally and test parsing without hitting the website every time. That is faster, cleaner, and more respectful to the target server.

3. Store fixtures and sample outputs

Even lightweight scrapers benefit from example files:

• A saved HTML snapshot for one representative page
• A sample JSON or CSV output file
• A short note describing edge cases

When the site changes, these fixtures help you compare old assumptions against new structure. They also make onboarding easier if another developer inherits the scraper.

4. Add basic failure checks

A scraper should fail loudly when extraction quality drops. At minimum, check for:

• HTTP errors and timeouts
• Missing required fields
• Sudden drops in item count
• Unexpected redirect or login pages
• CAPTCHA or block-page markers in HTML

For example:

data = parse_article(html)
if not data["title"]:
    raise ValueError("Missing title: selector may have changed")

This is simple, but it turns a silent data-quality problem into an actionable alert.

5. Review on a schedule

If the scraper matters to a report, product, or pipeline, put it on a review schedule. Monthly is reasonable for volatile sites; quarterly may be enough for stable ones. During review, confirm:

• The target still renders as expected
• Selectors are still valid
• Dependencies still install cleanly
• Output fields still match downstream needs
• Runtime and failure rate are acceptable

This is where a “living” guide matters. The tools stay familiar, but best practices shift. A Python web scraping tutorial should not just show syntax. It should help you maintain your judgment about when to use HTTP, when to parse HTML, and when to automate a browser.

Signals that require updates

You do not always need to wait for a scheduled review. Some signals mean a scraper should be revisited immediately.

The page suddenly returns less data

If a listing scraper usually finds 50 items and now finds 8, the issue may be:

• A selector change
• Lazy-loaded content not being captured
• A changed pagination pattern
• A block page replacing the expected HTML

Count-based validation is one of the easiest ways to spot trouble early.

The HTML structure changes

Frontend teams regularly rename classes, wrap content in new containers, or replace semantic markup with component-generated HTML. Fragile selectors like deeply nested class chains tend to break first. Prefer selectors tied to stable attributes, semantic tags, or nearby labels when possible.

The site becomes more dynamic

If a page that once worked with requests now loads key content from XHR or fetch calls, you may need to inspect the browser network panel and adapt your strategy. Sometimes you can call the underlying JSON endpoint directly. Other times you will need Playwright.

Access behavior changes

New login requirements, CSRF tokens, region-specific responses, session cookies, or rate limiting can all break a scraper that previously worked. When that happens, avoid random trial and error. Reproduce the page behavior in your browser, inspect requests carefully, and document what changed.

Your output requirements change

Many scraper updates are not caused by the target site at all. Internal changes matter too. If downstream systems now need normalized dates, canonical URLs, deduplicated records, or richer metadata, update the extraction and transformation steps together. Scraping is only one part of the data pipeline.

This becomes especially important when scraped data feeds other systems or decision workflows. The same maintenance discipline used in broader integration work—clear contracts, thin-slice testing, and careful schema handling—also applies to scrapers. For a related engineering mindset, see Thin-Slice Prototyping for EHR Integrations: A Scraper-Engineer’s Playbook to Ship Safely.

Common issues

Most scraping problems are familiar once you have seen them a few times. The challenge is recognizing them quickly and responding with the least complicated fix.

Issue: requests returns HTML, but the data is missing

Likely cause: the page is rendered in the browser after the initial response.

What to do:

• Open developer tools and inspect the Network tab
• Look for JSON or API responses containing the data
• If needed, switch to Playwright and wait for the relevant selector

Example with Playwright waiting for content:

from playwright.sync_api import sync_playwright
from bs4 import BeautifulSoup

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com", wait_until="domcontentloaded")
    page.wait_for_selector(".product-card")
    html = page.content()
    browser.close()

soup = BeautifulSoup(html, "lxml")
items = soup.select(".product-card")
print(len(items))

Issue: Selectors break often

Likely cause: your CSS selectors are too tightly coupled to presentation classes.

What to do:

• Prefer IDs, data attributes, semantic tags, and stable landmarks
• Use shorter selectors where possible
• Write fallback extraction logic for key fields

For example, rather than a long chain of utility classes, target a product link inside a known card container or a heading near a label.

Issue: The scraper gets blocked or rate-limited

Likely cause: requests are too frequent, too uniform, or inconsistent with expected browsing patterns.

What to do:

• Reduce request rate
• Respect robots directives where relevant to your use case
• Use retries sparingly and with backoff
• Cache pages when testing
• Avoid sending unnecessary traffic

There is no universal anti-bot bypass recipe that is both reliable and appropriate for every context. The evergreen principle is simpler: be deliberate, cautious, and respectful of technical and legal constraints.

Issue: Pagination stops after one page

Likely cause: the next-page URL pattern changed, infinite scroll replaced numbered pages, or state is maintained in JavaScript.

What to do:

• Check whether a “next” link still exists in the HTML
• Inspect network calls during scrolling or clicking
• Capture and test page parameters separately

Beginners often hardcode page URLs without validating how the site actually paginates. A few minutes in the network panel can save a lot of guesswork.

Issue: Encodings and text cleanup are messy

Likely cause: mixed whitespace, HTML entities, hidden elements, or inconsistent source encoding.

What to do:

• Use get_text(strip=True) thoughtfully
• Normalize whitespace after extraction
• Store raw values when debugging transforms
• Keep parsing and cleaning as separate steps

That separation matters when you later need to audit why a field changed.

Issue: Browser automation is too slow

Likely cause: Playwright is being used for work that could be done via direct HTTP or API requests.

What to do:

• Use Playwright to discover how the site works
• Then move stable data collection to direct requests when possible
• Reserve browser automation for pages that truly require it

This hybrid approach is often the most maintainable. Use the browser as a diagnostic and fallback tool, not automatically as the default engine.

When to revisit

Revisit this tutorial—and your own scraper implementation—when any of the following is true:

• You are starting a new target and need to choose between requests and Playwright
• A scraper that used to work now returns incomplete or empty data
• The target site has undergone a redesign or frontend migration
• You are turning a one-off script into a recurring job
• You need to improve data quality, logging, or downstream integration
• You want to reduce browser automation costs by simplifying the stack

To make that revisit practical, use this action checklist:

1. Re-test the raw fetch. Confirm whether the initial HTML still contains the target data.
2. Inspect the browser network panel. Look for underlying JSON endpoints, lazy-loading behavior, or changed request patterns.
3. Review selectors field by field. Start with required outputs, not every optional element.
4. Validate output against a sample. Compare current extraction with a known-good result.
5. Decide whether to simplify or upgrade. Move from Playwright to requests if the browser is unnecessary, or from requests to Playwright if the page is now dynamic.
6. Document what changed. Update your assumptions, fixtures, and checks so the next revision is faster.

If you keep a personal or team scraping playbook, this article works best as a recurring reference rather than a one-time read. The core libraries are stable enough to learn once, but the real skill is knowing how to adapt them to shifting page behavior. That is why a maintenance-first Python web scraping guide remains worth revisiting: not because the syntax is hard, but because the web keeps moving.

As your projects mature, you may also want to connect scraped data into larger pipelines, APIs, or validation workflows. For that broader systems perspective, articles like Mapping the Healthcare API Landscape: A Practical Decision Matrix for Engineers can be useful reminders that extraction is only one part of dependable data delivery.

The practical takeaway is simple. Start with the smallest tool that works. Parse carefully. Validate aggressively. Save examples. Review on a schedule. And when the site changes—as it eventually will—update the scraper by re-checking assumptions instead of patching blindly.