SYSTEMS ARCHITECTURE 13 min read · ~1,650 words Published September 2026

Behind the Peel: How Readability Heuristics and DOM Cleansing Strip Clutter from Modern Web Pages

Modern web documents deliver an average signal-to-noise ratio below 2%. Feeding raw HTML directly into generative model inference causes catastrophic token consumption, prompt injection vulnerabilities, and latency inflation. Here is the architectural mechanics of building a production extraction pipeline.

The End-to-End Extraction Pipeline

+-----------------------------------------------------------------------------+
| 1. RAW HTTP INGESTION & NETWORK BARRIERS                                    |
|    - Synchronous DNS check: Rejects RFC 1918, Loopback & 169.254.169.254     |
|    - Stream validation: 15MB payload cap & max 3 redirect hops              |
+-------------------------------------+---------------------------------------+
                                      |
                                      v
+-----------------------------------------------------------------------------+
| 2. DETERMINISTIC AST NODE PRUNING                                           |
|    - Drop non-content tags: <script>, <style>, <nav>, <aside>, <iframe>     |
|    - Remove hidden elements: display:none, visibility:hidden, aria-hidden   |
+-------------------------------------+---------------------------------------+
                                      |
                                      v
+-----------------------------------------------------------------------------+
| 3. READABILITY HEURISTIC SCORING & ANCESTOR CLUSTERING                      |
|    - Base score initialized on tag semantics (<article>, <p>, <div>)        |
|    - Weight adjustments: Class/ID regex matches & comma density calculations|
|    - Link Density Ratio (LDR) filtering: Prunes nodes if LDR > 0.25         |
|    - Ancestor clustering: Walk up AST to isolate root content container     |
+-------------------------------------+---------------------------------------+
                                      |
                                      v
+-----------------------------------------------------------------------------+
| 4. MARKDOWN NORMALIZATION & TOKEN PACKING                                   |
|    - Convert semantic DOM tree into lightweight Github-flavored Markdown    |
|    - Strip extraneous attributes, preserving headings, lists, code, tables  |
|    - Context window chunking: Enforce 8k/16k token ceilings for LLM prompt  |
+-------------------------------------+---------------------------------------+
                                      |
                                      v
+-----------------------------------------------------------------------------+
| 5. HIGH-THROUGHPUT LLM INFERENCE (Amazon Bedrock Nova Micro / Anthropic)    |
|    - Zero hallucination prompt constraints; latency reduced from 8s to 1.4s |
+-----------------------------------------------------------------------------+

1. The Modern Web Clutter Problem

The contemporary web has shifted from structured semantic documents into bloated, client-side application bundles. A typical 900-word news report or technical blog post represents approximately 5 to 6 kilobytes of meaningful UTF-8 prose. However, when fetched via an HTTP client, the resulting raw HTML payload routinely measures between 350 KB and 2.5 MB.

This 50× to 400× expansion factor consists almost entirely of non-editorial overhead:

Hydration Scripts & State Blobs: Massive base64-encoded JSON state objects (__NEXT_DATA__, Redux stores, Apollo cache bundles) embedded directly in the markup.
Nested Monetization Frames: Multi-tiered <iframe> hierarchies for header bidding, ad re-targeting pixels, and programmatic auction scripts.
Consent Management & Layout Blockers: Heavy GDPR/CCPA modal wrappers, sticky promotional banners, paywall overlays, and slide-in newsletter forms.
Infinite Scroll Navigation Observers: Dozens of dynamic container nodes configured with tracking listeners and recommendation widgets (“Around the Web” / “Recommended For You”).

Why Raw HTML Extraction Fails LLM Context Windows

Passing un-sanitized HTML directly into modern Large Language Models triggers severe operational failures:

Severe Token Waste

A 500 KB raw DOM swallows 90,000+ tokens. 95% of compute is spent parsing CSS classes and script bundles rather than editorial analysis.

Prompt Injections

Malicious actors embed hidden CSS instructions (display:none) instructing models to hallucinate endorsements or skip evaluation.

Latency & Cost Inflation

Inference latency scales directly with input context length. Unprocessed HTML inflates response times from 1.2s to 9.5s per request.

2. The Algorithmic Cleansing Pipeline

To isolate authentic editorial prose, PeelitNow employs a multi-pass Abstract Syntax Tree (AST) parsing engine adapted from Mozilla Readability and Cheerio.

Pass 1: Deterministic Node Pruning

Before running semantic weight heuristics, the engine executes a top-down traversal across the AST, immediately removing all tags that cannot contain readable editorial prose:

PRUNE_TAGS = ["script", "style", "noscript", "iframe", "nav", "footer", "aside", "header", "canvas", "svg", "form"]

Nodes bearing inline styles such as display: none, visibility: hidden, or HTML5 attributes like aria-hidden="true" are purged in the initial pass.

Pass 2: Paragraph Scoring & Text Density Heuristics

Every remaining structural container (<p>, <div>, <section>) is evaluated against a deterministic scoring function:

Base Score: Elements receive +1 point for every paragraph (<p>) contained within them that exceeds 50 characters.
Comma and Punctuation Density: Real prose contains commas, colons, and semicolons. The scoring algorithm adds +1 point for every comma detected within textual child nodes, reliably differentiating complete sentences from bulleted navigation links.
Class and ID Regex Scoring: Node attributes are audited against positive and negative patterns:
POSITIVE (+25 pts): /(article|body|content|entry|main|post|story)/i
NEGATIVE (-25 pts): /(ad-|sidebar|comment|promo|widget|sponsor|share)/i
Link Density Ratio (LDR) Rejection: Navigation headers and recommendation carousels often have high character counts, but the text is predominantly wrapped inside anchor (<a>) tags. If LDR = (Length of text in <a> tags) / (Length of all text in node) > 0.25, the node is pruned entirely.

Pass 3: Ancestor Clustering

Individual high-scoring paragraph nodes are rarely isolated. The algorithm traverses up the AST hierarchy, assigning 100% of a child's score to its direct parent, and 50% of the score to its grandparent. The node that accumulates the highest composite ancestral score across the entire document is isolated as the Top Candidate Root.

3. Security & Anti-Abuse Controls

An autonomous web summarizer functions as an arbitrary HTTP client. Without strict network isolation, attackers can submit URLs designed to execute Server-Side Request Forgery (SSRF), port-scan internal infrastructure, or trigger denial-of-service via massive archive files.

SSRF Network Isolation Architecture

Our Lambda extraction workers resolve target domains synchronously before initiating an HTTP connection. If the resolved IP address matches any of the following restricted CIDR ranges, the request is terminated with an HTTP 400 rejection:

RFC 1918 Private Spaces: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16
IPv4 Loopback: 127.0.0.0/8
AWS Instance Metadata Service (IMDS): 169.254.169.254/32
IPv6 Local & Loopback: ::1/128, fc00::/7, fe80::/10

Payload Caps & Redirect Limits

HTTP streams are bounded by strict runtime constraints:

15 MB Payload Ceiling: Inbound streams are monitored via byte-counting chunk handlers. If an HTTP response exceeds 15 MB, the socket is abruptly destroyed to prevent memory exhaustion from zip bombs or high-resolution video streams.
Max 3 Redirect Hops: HTTP 301/302 redirects are tracked iteratively. Each redirect hop triggers a fresh DNS resolution check against SSRF blacklists. Loops exceeding 3 hops are discarded.

Production TypeScript Implementation: LDR & SSRF Validation

extractor-utils.tsTypeScript / Node.js

import * as dns from 'node:dns/promises'; import { isIP } from 'node:net'; /** * Validates that an IP address does not belong to private or cloud-internal subnets. */ export function isRestrictedIp(ip: string): boolean { if (!isIP(ip)) return true; // IPv4 Loopback & Private Subnets if (ip.startsWith('127.')) return true; if (ip.startsWith('10.')) return true; if (ip.startsWith('192.168.')) return true; if (ip.startsWith('169.254.')) return true; // AWS IMDS Link-Local // 172.16.0.0 – 172.31.255.255 if (ip.startsWith('172.')) { const secondOctet = parseInt(ip.split('.')[1], 10); if (secondOctet >= 16 && secondOctet <= 31) return true; } // IPv6 Loopback & Link-Local if (ip === '::1' || ip.startsWith('fe80:') || ip.startsWith('fc00:')) return true; return false; } /** * Pre-flight SSRF check resolving DNS before issuing HTTP request. */ export async function assertSafeUrl(targetUrl: string): Promise<void> { const parsed = new URL(targetUrl); if (!['http:', 'https:'].includes(parsed.protocol)) { throw new Error('Unsupported protocol: HTTP and HTTPS only'); } const lookupResults = await dns.lookup(parsed.hostname, { all: true }); for (const record of lookupResults) { if (isRestrictedIp(record.address)) { throw new Error(`SSRF Violation: ${parsed.hostname} resolves to restricted IP ${record.address}`); } } } /** * Calculates Link Density Ratio (LDR) to detect navigation/ad widget blocks. */ export function calculateLinkDensity(nodeText: string, anchorText: string): number { const totalLength = nodeText.trim().length; if (totalLength === 0) return 0; const linkLength = anchorText.trim().length; return linkLength / totalLength; }

4. LLM Token Optimization & Markdown Normalization

Once the candidate content node is isolated, PeelitNow converts the surviving AST directly into clean, standardized Markdown rather than retaining HTML tags. Markdown represents the optimal representation for LLM ingestion:

Attribute Stripping: All class, id, style, data-*, and event handler attributes are purged completely.
Semantic Preservation: Headings (<h1> through <h4>) become markdown hash prefixes (# through ####). Lists, tabular data, blockquotes, and code snippets are preserved intact.
Context Window Packing: The resulting text is clamped to safe token boundaries (e.g., 8,000 tokens for Amazon Bedrock Nova Micro). If a long-form document exceeds token caps, it is segmented at semantic section boundaries rather than arbitrary character offsets.

Architecture Summary

High-performance AI summarization is fundamentally an information retrieval and data hygiene challenge. By pairing synchronous SSRF network boundaries with AST readability heuristics, PeelitNow achieves a 78% reduction in inference token usage, eliminates prompt injection vectors, and delivers sub-2-second end-to-end response times.