Clean scraped data becomes useful when identical values are represented consistently, invalid records are visible, and the original source remains available for review. This guide presents a repeatable workflow for normalizing whitespace, Unicode, dates, prices, numbers, categories, and missing values, with practical Python and JavaScript examples, validation rules, and simple metrics for estimating data quality before import.
Overview
Web pages are designed for people, not databases. A product price may appear as $1,299.00, 1 299,00 €, or From 1299. The same publication date might be written as 2026-04-07, 07/04/2026, or April 7, 2026. Blank fields can mean “not supplied,” “not applicable,” “temporarily unavailable,” or “the scraper failed.” Treating all of these cases as ordinary strings creates unreliable comparisons and difficult-to-debug reports.
A dependable pipeline separates extraction from normalization. First, preserve the value exactly as it was collected. Then create a normalized field, record how it was parsed, and flag anything that does not meet the expected rule.
| Raw field | Normalized field | Recommended handling |
|---|---|---|
" Café Noir " | "Café Noir" | Normalize Unicode, replace non-breaking spaces, trim, and collapse internal whitespace. |
"07/04/2026" | 2026-04-07 | Parse with an explicit locale or format; do not guess between day and month. |
"€1.299,00" | 1299.00 plus EUR | Parse separators using the source locale and store currency separately. |
"N/A" | null | Map known missing markers to null and retain the original marker in the raw field. |
"Small", "S", "small " | "small" | Trim, case-normalize, and apply a controlled category mapping. |
For each record, useful operational fields include source_url, scraped_at, raw_value, normalized_value, parser_version, and review_required. This design makes later corrections possible without crawling the source again.
How to estimate
Before loading cleaned records into a database or report, estimate whether the result is complete enough for its intended use. A small quality summary is more useful than a vague claim that the data was “cleaned.” Calculate these measures for each important field:
- Completeness:
non-null normalized values ÷ total records × 100. - Parse success:
values passing the parser ÷ non-empty raw values × 100. - Review rate:
records flagged for review ÷ total records × 100. - Duplicate rate:
duplicate normalized keys ÷ total records × 100, after defining what makes a record unique.
Set acceptance thresholds according to the downstream decision. A dashboard may tolerate some missing descriptions, while a price comparison or financial import may require every price to have a valid numeric value and currency. Do not silently convert a failed parse to zero: zero is a meaningful number, whereas an unparsed value is an error or an unknown.
A practical estimate can also include manual workload. If a batch contains 10,000 records and the review rate is 2%, expect approximately 200 records to inspect. If a reviewer can assess 40 records per hour, the first-pass estimate is five hours. This is only a planning assumption; measure the actual rate and time for the first batch, then use those observations to revise future schedules.
Inputs and assumptions
Normalization rules are only reliable when their assumptions are explicit. Document the following before writing a parser:
- Locale: Identify the expected language, decimal separator, thousands separator, and date order. A slash-separated date is ambiguous without this context.
- Timezone: Decide whether dates represent the publisher’s local date, UTC, or the time at which your crawler observed the page. Store timezone-aware timestamps when time matters.
- Currency: Keep the currency code separate from the numeric amount. Do not convert currencies unless the conversion date, rate source, and rounding policy are part of the data model.
- Missing-value vocabulary: List markers such as an empty string,
N/A,—,unknown, andnot available. Decide whether each means null, not applicable, or a parser failure. - Category mapping: Use an explicit dictionary for values such as sizes, regions, or product types. Preserve unmapped values for review instead of forcing them into the nearest category.
- Validation boundaries: Define reasonable ranges, required fields, allowed currencies, and expected formats. A price below zero may be invalid, but a discount percentage could legitimately be zero.
For text fields, normalize Unicode before trimming. Replace non-breaking spaces and line breaks where appropriate, then collapse repeated whitespace. Avoid removing punctuation globally: punctuation may distinguish product identifiers, addresses, or versions. Case folding is useful for matching, but preserve the display form separately when capitalization matters.
Python’s standard library is sufficient for basic cleaning:
import re
import unicodedata
MISSING = {"", "n/a", "na", "-", "—", "unknown"}
def clean_text(value):
if value is None:
return None
value = unicodedata.normalize("NFKC", str(value))
value = value.replace("\u00a0", " ")
value = re.sub(r"\s+", " ", value).strip()
return None if value.casefold() in MISSING else value
Use a date or locale-aware parsing library when formats vary, but require the expected locale or format whenever possible. In JavaScript, avoid relying on implementation-dependent date parsing for ambiguous strings. Parse a known pattern into year, month, and day, then construct a deliberate UTC or local representation.
Worked examples
Parsing prices without losing meaning
Suppose a scraper collects "€1.299,00". In this format, the period is a thousands separator and the comma is the decimal separator. A safe result is an amount of 1299.00 and a currency of EUR. Store the raw string as well. If the source later changes to "1,299.00 €", the parser can be updated without erasing the evidence that exposed the change.
Do not infer currency from a symbol alone when the source can display multiple currencies. Prefer a currency code from structured data, a page-level setting, or a documented source rule. Flag a price when the currency is missing, the separators are contradictory, or extra text prevents a confident parse.
Standardizing dates
For a source documented as day-month-year, 07/04/2026 should become 2026-04-07. For a source documented as month-day-year, the same raw value means a different date. The normalized storage format can be ISO 8601, but the conversion rule must be recorded. If a page says “updated today,” save the observation timestamp and mark the value as relative; do not pretend it is an exact publication date.
Handling categories and missing values
A controlled mapping might convert S, Small, and small to small. An unfamiliar value such as Petite should remain unmapped until its meaning is confirmed. Likewise, distinguish null from not_applicable and parse_error if downstream users need to know why a value is absent.
After transformation, run validations such as:
- Required identifiers are present and unique within the intended scope.
- Normalized dates match the expected range and timezone policy.
- Amounts are numeric, non-negative where appropriate, and paired with a valid currency.
- Categories belong to the approved vocabulary or are explicitly flagged.
- Raw and normalized fields are both retained for records that need review.
When to recalculate
Revisit the workflow whenever the source changes its labels, locale, markup, currency display, date format, or missing-value conventions. A sudden increase in nulls, parse errors, duplicate keys, or review flags is a signal to inspect recent raw samples. Website change detection and scraper regression tests can help identify whether the issue is a selector change or a data-format change.
Recalculate completeness, parse success, review rate, and duplicate rate after every substantial crawl or import. Also recalculate when adding a new country, supplier, language, or source domain, because assumptions that worked for one locale may not apply to another. Keep a small fixture set containing representative prices, dates, whitespace, Unicode, missing markers, and invalid values, then run it whenever parsing code or dependencies change.
Before production import, use this checklist:
- Raw values and source URLs are preserved.
- Locale, timezone, currency, and missing-value assumptions are documented.
- Dates, prices, numbers, and categories have explicit validation rules.
- Failed parses are flagged rather than silently coerced.
- Quality metrics are compared with the previous successful run.
- A sample of normalized records has been reviewed manually.
Normalization is not a one-time cleanup step. It is a versioned part of the data pipeline. By preserving source values, measuring outcomes, and recalculating after source or rule changes, you can turn inconsistent scraped fields into comparable data without hiding uncertainty.