hirspektrum.hu — clustering a national news feed

Context

hirspektrum.hu reads the Hungarian news feed continuously, groups the articles that cover the same event, and compares how different outlets tell that story.

The goal is media literacy rather than adjudication. The interesting question is usually not which outlet is lying — it is which facts appear in one version of a story and quietly vanish from another, and which actors get named versus described. That shape is visible in the data if you group the coverage first, and invisible if you read one article at a time.

I designed and built all of it: scraping, ingestion, clustering, graph analysis, the LLM pipelines, and the infrastructure underneath.

Constraints

Self-hosted, on a personal budget. No managed services and no per-token cost that scales with how busy the news cycle happens to be. This is the constraint that decided most of the architecture, and I consider it a feature: a media analysis tool whose running cost spikes during exactly the weeks it matters most is not a tool you can rely on.

The feed never stops. Scraping, clustering, and analysis run on completely different clocks — scraping is continuous and I/O-bound, clustering is periodic and compute-bound, LLM analysis is slow and expensive. None of them may block another.

Clustering has no ground truth. There is no labelled dataset that says "these eleven articles are the same story," and no fixed number of stories per day. The grouping has to be unsupervised and has to discover the cluster count from the data, which rules out the entire family of approaches that want k up front.

Architecture and decisions

RabbitMQ over Kafka. Kafka's strengths are durable replay, long retention, and partitioned throughput at a scale I do not have. What I actually needed was work-queue semantics: hand a job to exactly one consumer, acknowledge it when it completes, redeliver it when the consumer dies. That is RabbitMQ's native shape, it runs as one container, and one person can operate it. Choosing the smaller tool here was a capacity decision, not a technical preference.

Four data stores, because there are four genuinely different access patterns. Polyglot persistence is usually a smell, so the reasoning matters:

• MongoDB holds raw scraped documents. Every outlet's markup is different and the shape of what I can extract varies per source; forcing that into a schema at ingest time would mean losing data I have not yet learned to want.
• PostgreSQL holds the structured core — articles, outlets, clusters, and the relationships between them — where I want constraints and joins.
• Neo4j holds the entity graph. "Which people and organisations co-occur, across which outlets, over what period" is a traversal, and expressing it in SQL means writing recursive CTEs that nobody will maintain, including me.
• A cache in front of the read path, because the site's queries are overwhelmingly repeated and the underlying analysis changes on the order of hours, not seconds.

Independent Python services, one queue each. A scraper that wedges on a misbehaving site must not stall analysis, and an outlet changing its markup should degrade one consumer rather than the pipeline. Each service does one thing and can be restarted alone.

The LLM sits at the end, deliberately. Classical processing and unsupervised clustering do the grouping; the language model only ever runs over material that has already been grouped. Keeping it off the hot path is what makes the cost bounded and predictable — analysis scales with the number of stories, which is roughly constant, rather than the number of articles, which is not. It also keeps the non-deterministic component out of the part of the system I need to be able to reason about.

Outcome

The platform runs continuously and entirely self-hosted: ingesting the Hungarian news feed, clustering coverage dynamically as stories develop, maintaining the entity graph, and generating automated comparative analysis across outlets.

What I would do differently: I built the LLM analysis stage before I built any way to evaluate it. For a while the honest answer to "is this output good?" was that I had read a few and they seemed fine — which is not an answer. Retrofitting evaluation onto a generative stage is significantly harder than designing the stage around it, and it is the piece of this project I would reorder if I started again.