Text is not a keyword.
Title and body use text analyzers. URLs, IDs and categories use keyword fields; timestamps use dates, and confidence uses numeric fields. The mapping affects search and aggregation quality.
02 / THE ARCHITECTURE BEHIND THE STORY
The site stays light. The pipeline does the heavy lifting.
Behind the film are resilient crawlers, structured records, search indexes, semantic embeddings, AI agents, and evidence checks. Visitors receive a reviewed data snapshot—not a live model or crawler session.
DISCOVERY03RSS · crawling · GDELT
WORKFLOW07coordinated agent roles
PROCESSING08stages from discovery to export
DELIVERYJSONprecomputed website snapshot
01
HOW THE SYSTEM FITS TOGETHER
Complex behind the scenes. Fast in the browser.
RSS, crawling, and a supplementary GDELT index support discovery. Extraction, normalization, semantic grouping, classification and evidence checks happen before the website receives its precomputed JSON.
01 / DATA WORKFLOWJakarta News Intelligence
RSS · sitemap · GDELT→Crawl & extract→Normalize & deduplicate→GLiNER2 / LLM→Analyst + Verifier→JSON
Feeds and sitemaps help identify new URLs. Crawling retrieves permitted article content, while GDELT provides supplementary discovery and coverage checks. Each stage retains its source references.
02 / STATIC PRESENTATIONJakarta, 730 Hari
Processed data→Validated JSON→Next.js→Browser
The published film reads story.json. It does not crawl the news, query a production database, or call a language model when someone visits.
SOURCE OF TRUTH
PostgreSQL + pgvector
Articles, labels, events, evidence references, and checkpoints.
SEARCH INDEX
Elasticsearch
Text retrieval, filters and aggregation. Rebuildable from source records.
OBJECT STORAGE
MinIO / S3
Compressed source HTML, subject to access and retention rules.
02
STORAGE & RETRIEVAL
Why not just put everything in Elasticsearch?
News begins as unstructured HTML. Once the text is extracted, normalized, deduplicated, and enriched with NLP labels, it can be stored as an authoritative record and projected into search indexes.
STORAGE / DATA ARCHITECTURE
One article. Four different storage needs.
The database preserves authoritative records, search indexes enable retrieval, vectors capture similarity, and object storage retains raw inputs. Select a layer to see why.
SOURCE OF TRUTH
PostgreSQL
Articles, sources, labels, model versions, events, runs, and evidence IDs are stored in auditable relationships. Constraints help maintain integrity.
WHY KEEP IT SEPARATE?
Transactions, audit trails, checkpoints, and reproducible index rebuilding.
EXAMPLE STRUCTURE
articles → labels → stories → insightsElasticsearch does not infer topics, sentiment, or event identity on its own. The enrichment and clustering stages produce those fields before indexing, so they can later be searched, filtered, and aggregated.
Words have context.
Indonesian tokenization, stop words, and local synonyms—such as Transjakarta, TJ, and busway—improve recall across different phrasing.
Search is a projection.
PostgreSQL remains authoritative. Versioned monthly indexes can be rebuilt and switched behind an alias without changing search clients.
03
DATA CONTRACTS / RELATIONSHIPS
Every insight should lead back to a source.
IDs and versions make it possible to investigate a mislabeled article, retrieve supporting coverage, and reproduce an analysis with the same rules and model versions.
| ENTITY | IMPORTANT FIELDS & RELATIONSHIPS | WHY IT EXISTS |
|---|---|---|
| sources | domain, media_name, country, feed_urls, last_ok_at | Track source health, geography and coverage gaps. |
| articles | id, source_id, url_canonical, published_at, body_key, content_hash, story_id | Provide an article identity, fetch lineage and event connection. |
| labels / entities | article_id, task, label, score, model_version, role | Keep topics, sentiment, entities, and model confidence inspectable. |
| stories / embeddings | story_id, n_sources, vector(768), similarity | Support event clustering and semantic retrieval. |
| insights | period, numbers, evidence_ids, verdict, verifier_note | Trace narrative claims and metrics to their supporting articles. |
| runs / eval_runs | stage, n_in, n_out, cost_usd, metrics, model_version | Record batch quality, cost and model changes. |
01ARTICLE
Publisher, URL, timestamp, raw object key, content hash
↓02LABEL
Topics, sentiment, location, entities, confidence, model version
↓03STORY
Event cluster, distinct sources, embeddings and evidence
↓04INSIGHT
Statistics, intervals, evidence IDs, verifier verdict
↓05EXPORT
Schema checks, versioned JSON, review, snapshot
04
DIVIDING THE WORK
Seven roles. Not seven unnecessary LLM calls.
Scout monitors sources; Crawler and Dedup run reproducible steps; Classifier handles labels; Analyst drafts findings; Verifier checks evidence; Publisher produces the snapshot. Select an agent to inspect its role.
AGENT 01 / 07
Scout.
Contextual decisionsReviews new sources, failed feeds and changes in the publisher registry so discovery stays reliable.
TOOLS / COMPONENTSRSS · sitemap · GDELT · sources
OUTPUTReviewed source list
LLM USAGEWhen source assessment needs judgment
05
ELASTICSEARCH / RETRIEVAL
Search is how evidence is found—not how meaning appears.
A search index makes it fast to find the original coverage behind an insight: article text, people mentioned, dates, topics, and labels. Those labels were produced and checked upstream.
01 / VALIDATEClean articleExtracted body text, canonical metadata, regional relevance, and quality flags.
02 / ENRICHNLP labelsMulti-label topics, article or actor sentiment, entities, event ID, confidence, and model version.
03 / INDEXMapping + analyzerBulk indexing into articles-YYYY.MM behind an alias; Indonesian text analysis plus keyword/date filters.
04 / RETRIEVEFind evidenceFull-text search filtered by topic, outlet, time and sentiment; returned article IDs support claim verification.
Illustrative query: search for Indonesian-language coverage of “banjir Jakarta” within a chosen period, filter topic = flooding and sentiment = negative, then retrieve article IDs to inspect the original sources. This describes the backend retrieval pattern, not live search on the static film website.
06
THE ENGINEERING TOOLKIT
Choose tools for their responsibilities, not their logos.
Fetching, orchestration, NLP, transactional storage, search, and snapshot delivery solve different problems. The modules above show where each belongs.
Getting the news
Finding stories is different from extracting them.
Browser rendering only for permitted JavaScript-heavy pages.Keeping work accountable
Scheduling, retries, checkpoints, and review.
Agent decisions can be traced back to their inputs.The right model for the task
Local inference first; escalate exceptions.
Not every article needs an expensive model call.One source of truth
Transactional records and search indexes have different jobs.
Search indexes can be rebuilt from authoritative records.Publish, then render
Reviewed JSON becomes a static experience.
Visitors never need a live database connection.07
RELIABILITY IN PRACTICE
A diagram is not an operating system.
A robust pipeline distinguishes missing sources, extraction failures, model drift, and unsupported claims. Retries are only one response; some failures require quarantine, manual review, or rollback.
01 / SOURCE SAFETY
Crawl responsibly.
Respect robots.txt, apply per-domain rate limits, avoid bypassing paywalls, and track extraction failures.
Rate-limit policy: 1 request per 2 seconds per domain02 / QUALITY & EVIDENCE
Every claim needs a source.
Insights retain supporting article IDs. Verifier reviews the narrative and deterministic checks confirm numeric claims against computed data.
Only supported findings are exported03 / COST & PRIVACY
Set limits—and keep them visible.
Daily LLM budgets, caching, personal-data minimization and retention rules reduce unnecessary risk.
Documented HTML retention policy: 180 days08
RELIABILITY / QUALITY GATES
What must pass before a finding is published?
Quality checks span the entire lifecycle: source discovery, text extraction, similarity, model evaluation, evidence verification, and safe publishing.
Source health
Source registry, feed status, robots.txt, crawl schedule.
CHECKDead feeds, blocked sources, and drops in coverage are visible.
QUALITY GATEExtraction quality
Fetch, parse, dates, language, location, schema, quarantine.
CHECKFailed extracts are flagged instead of silently counted as zero.
QUALITY GATESimilarity & dedup
Canonical URLs, MinHash, embeddings, event windows.
CHECKCopies and independent events are separated; thresholds are evaluated.
QUALITY GATEModel quality
GLiNER2, confidence, LLM review, per-topic evaluation.
CHECKF1, disagreement, drift, and costs can be monitored.
QUALITY GATEEvidence checks
Aggregates, denominators, evidence IDs, independent verifier.
CHECKClaims are checked against source articles before publishing.
QUALITY GATERecovery & publishing
JSON validation, versioning, review, backup and rollback.
CHECKThe published snapshot remains available if the pipeline fails.
QUALITY GATE