Respect the source.
Honor robots.txt, paywalls, and publisher policies. Per-domain rate limits, honest identification, and backoff protect both the source and the dataset.
01 / METHODOLOGY · FROM ARTICLES TO EVIDENCE
From messy news pages to a story you can inspect.
A chart is the final step, not the starting point. Explore the crawling queues, broken HTML, data-quality checks, NLP labels, event clustering, and statistics behind the film.
FILM SNAPSHOT690.828articles in the published dataset
FILM SNAPSHOT08tracked urban issues
PROCESSING08stages from discovery to export
DISCOVERY03RSS · crawling · GDELT index
01
THE BIG PICTURE
Eight stages. A trail of evidence.
Select a stage to see its decisions, tools, and outputs. Not every step needs an AI model: fetching, validation, duplicate detection, and statistics rely on reproducible code.
STAGE 01 / 08 · NEWS PROCESSING
Discovery.
Finding and accessing relevant reporting
RSS feeds and sitemaps are the first discovery layer, supported by direct news crawling and the GDELT index. Jobs follow robots.txt and per-domain rate limits.
OUTPUT OF THIS STAGECanonical article URLs, source metadata and discovery timestamps
feedparser · httpx · Crawl4AI · Trafilatura
Scheduled fetching, retries and source-health monitoring
1 OF 8 STAGES
02
BEHIND THE CRAWLER
Crawling is more than fetching a URL.
Every publication has its own feeds, templates, JavaScript, access rules, and failure modes. Explore how URLs are discovered, jobs are scheduled, pages are fetched, and usable text is extracted.
CRAWLER / FINDING NEW ARTICLE URLS
Discovery.
Start with RSS and sitemaps, then check publication sections and the GDELT discovery index. Keep per-source cursors so previously seen URLs are not processed repeatedly.
THE WORK INVOLVED
- 01Read feeds, sitemaps, and supporting GDELT records
- 02Strip tracking parameters and resolve canonical URLs
- 03Check the queue before creating a new fetch job
FAILURES TO EXPECT
Delayed feeds, stale links, or the same article discovered through multiple channels.
OUTPUT / CONTRACTsource_id, canonical_url, discovered_at, discovery_method
COMPONENTSfeedparser · sitemap · GDELT
Failures must be visible.
Validate schemas, timestamps, empty articles, and repeated URLs. Quarantine failed records instead of quietly turning missing coverage into zeros.
Keep the paper trail.
Each record retains its source domain, canonical URL, publishing and crawl times, retrieval method, and extractor version.
03
UNSTRUCTURED → STRUCTURED
HTML has no topic, sentiment, or actor column.
Raw text needs extraction, normalization, and NLP enrichment. A single article may discuss several issues, quote multiple people, and express different attitudes toward different subjects.
ILLUSTRATIVE EXAMPLE / NOT A REAL ARTICLE OR MODEL OUTPUT
From messy HTML to searchable fields.
Follow a fictional Indonesian article through extraction, NLP enrichment, and indexing.
● ● ● ARTICLE / RAW HTML
<nav>Berita | Kanal | Iklan</nav> <article> <h1>Genangan terjadi di Jakarta</h1> <p>Hujan menyebabkan genangan di ruas jalan.</p> <p>Pemprov DKI menyiapkan pompa.</p> </article> <aside>Berita lainnya ...</aside>
WHAT CHANGED?
Menus, ads, and recommendations are not article content. This input deliberately uses Indonesian news text.
Sentiment is an inference, not a fact hidden in HTML. It needs evaluation against human-labeled examples.One article. Multiple issues.
A story about floodwater and pumping operations can be about both flooding and local government. Store each assigned label with a confidence estimate and model version.
Negative toward whom?
Article-level tone is different from an opinion about an actor. A resident’s complaint and an official’s response may appear in the same story.
“Jakarta” is not enough.
A dateline can be misleading. District and neighborhood names, plus contextual mentions, help determine whether an event actually concerns Jakarta.
The examples above are illustrative, not real articles, measured model predictions, or benchmark scores. Thresholds and accuracy figures require evaluation against a labeled dataset.
04
DEDUPLICATION & EVENT CLUSTERING
Similar words do not always mean the same event.
One incident may be syndicated across dozens of outlets; two similar headlines may describe different incidents. That is why the workflow separates exact duplicates, near-duplicates, and articles reporting a shared event.
CONCEPTUAL EXAMPLE / NOT MEASURED SIMILARITY
Similar headlines are not always the same event.
Compare exact copies, lightly edited syndication, and reports about the same event. Select a case to explore the difference.
ARTICLE A
≈Genangan terjadi di Jakarta
ARTICLE B
Genangan terjadi di Jakarta
SHA-256 + canonical URL
Exact duplicates can be consolidated without a semantic model. A different tracking parameter does not make an article new.
A matching headline alone is not proof that two full articles are identical.Hashes & canonical URLs
Find identical documents and remove tracking parameters so the same article is not counted again under a different link.
MinHash + LSH
Five-word shingles and estimated Jaccard similarity surface syndication copies without comparing every possible pair of articles.
Embeddings + context
multilingual-e5-base captures semantic similarity. Time, place, and source evidence help decide whether articles belong to the same real-world event.
05
MULTI-STAGE CLASSIFICATION
From labels to defensible decisions.
The local model assesses topics, tone, place relevance, and entities. Ambiguous cases and high-impact stories may be escalated to a structured LLM review, with the reasoning recorded and the labels evaluated over time.
STAGE 1 / LOCAL
GLiNER2
Topics · area relevance · sentiment · entities
First pass for clear casesWHEN UNCERTAIN→
STAGE 2 / REVIEW
LLM
Structured reassessment with a concise explanation
Pydantic · audit trailWHEN TO ESCALATE?
- 01Top topic score below 0.60
- 02Gap between two top scores below 0.15
- 03Location relevance remains uncertain
- 04A major story covered by 10+ outlets
MODEL QUALITY CHECKS
Accuracy alone is not enough.
Changes to models, prompts, or thresholds should be evaluated on a human-labeled reference set. The figures beside this text are evaluation criteria, not achieved performance scores.
1,000reference articles for annotation
κ ≥ 0.7inter-annotator agreement criterion
≤ 2 ptsmaximum acceptable drop in cascade F1
06
STATISTICS THAT HOLD UP
From “this is trending” to measurable change.
Only after records are structured can we assess issue share, historical baselines, distinct outlets per event, co-occurrence, and uncertainty. The goal is to measure coverage, not mistake it for reality.
Share, not volume
Is an issue growing, or are there simply more articles?
Articles about an issue ÷ all relevant articles per week.
Makes comparisons less sensitive to changes in total coverage.
Detecting surges
When does coverage rise above its usual level?
This week’s share ÷ median share of the preceding 8 weeks.
Requires at least 30 articles and 4 comparable weeks.
Stories with reach
Which events were covered across the most outlets?
Rank event clusters by distinct news sources.
Each cluster links back to its source articles.
The media half-life
How quickly does attention fade after a crisis?
Time for excess coverage above baseline to fall by half.
Align event peaks before comparing attention curves.
Topic co-occurrence
Which issues appear together more often than expected?
P(A and B) ÷ [P(A) × P(B)].
Lift above 1 suggests an association in coverage, not causation.
Block bootstrap
How robust is a shift between periods?
4-week block resampling, 4,000 iterations, 95% interval.
Accounts for dependence between neighboring weeks.
Wilson intervals
How uncertain is a proportion from a small sample?
Report a 95% Wilson interval alongside the proportion.
Especially important when international coverage is sparse.
07
LIMITATIONS & TRANSPARENCY
What the evidence can—and cannot—tell us.
Every insight needs its source, denominator, missing-data flags, model version, uncertainty, and links to supporting articles. A precise-looking chart is not automatically a reliable finding.
Coverage is not reality
An increase in flood coverage does not prove an equivalent increase in actual flooding.
Missing is not zero
Two weeks in June 2025 are missing from the film snapshot. We keep those gaps visible rather than treating them as zero.
Models can be wrong
Topic, sentiment, and entity labels need evaluation against human-annotated examples, with ambiguous cases explicitly tracked.
Original reporting belongs to its publishers
The workflow respects access restrictions and paywalls; full text is handled only where permitted, and public excerpts remain limited.
DATA SOURCES & SCOPE
Jakarta News Intelligence combines RSS, sitemaps, and responsible news crawling, with The GDELT Project ↗ as a supplementary discovery and coverage index. The film presents a precomputed snapshot exported as story.json. Opening the page does not trigger crawling or AI analysis.