OpenCitizen

Even with the volume of reporting that surrounds an electoral campaign, few media outlets present that information in a way that lets readers actually compare candidates' policy proposals side by side. The 2025 NYC mayoral primary produced 11 candidates, dozens of policy PDFs, hundreds of news articles, and debate transcripts. No two candidates published their plans in the same format. Below are some of the source types used in this project:

OpenCitizen was built to be a structured reference for primary voters, synthesizing candidate positions so readers could vote with a clearer picture of what was actually proposed. The pipeline made that publication rate possible: rather than manually assembling context for each post, the structured dataset meant that drafting a comparison piece or debate recap started from organized, citable sources rather than a pile of raw articles. I wanted to organize each story by topic so a reader could follow one issue over time.

When pulling material in from across the web, it was important to keep track of where it came from and when it was sourced, as politicians often change their position on policy issues. The data pipeline splits the work into three stages: collection, enrichment, and comparison, each with its own stage boundaries, configs, and logs. Each artifact was kept in its original state, and transformed information was stored at each stage of the pipeline to make the work auditable. Storing raw source material separately from enriched versions allowed the final output to be citable.

The dataset splits into two layers: the candidate reference collection (campaign websites, Ballotpedia pages, PDFs, and full-page web scrapes, collected once per candidate and left unchanged) and the newsletter content (news coverage and debate material, collected as the race progressed). Both feed the same pipeline.

The pipeline stores each document in its original form before it is processed. That way, changes to a campaign website over time can be tracked and surfaced in later coverage, especially if the page or policies change.

To get all of the original material into a comparable format, suitable for both AI analysis and publishing, the project uses a three-stage medallion structure: cleaning, enrichment, and canonicalization, with files stored in separate folders at each stage to preserve provenance. Keeping each stage, its config, and its log separate means a bad NER (named entity recognition) run doesn't require redoing earlier work, and a bad cleaning pass shows up before it ever reaches the next stage.

The content_hash generated at bronze lets the pipeline skip files it's already processed on rerun. That matters once the collection spans hundreds of documents on different refresh schedules.

The pipeline's ability to compare candidates depends entirely on a fixed taxonomy applied consistently at the bronze stage. Without it, "compare housing positions" turns into a research project, because what counts as "housing" shifts depending on the source. With it, every document from every candidate carries the same tag set, and cross-candidate queries become possible.

The taxonomy stays fixed on purpose. Add a topic tag mid-dataset and you break comparability for any query that spans documents from before and after the change.

The second Democratic primary debate was held June 12, 2025 at John Jay College of Criminal Justice. Seven candidates, thirteen topics, roughly two hours of live television. The raw material is a video file and a transcript that reads nothing like prose: crosstalk, moderator interjections, incomplete sentences, candidates talking over each other, and a lot of filler that gets cut from the version people actually read.

Processing it meant feeding both the transcript and audio through the pipeline to separate each candidate's responses by topic, strip the crosstalk and filler, and produce a clean structured output: one row per candidate per topic, tagged with the speaker name, topic label, and a short summary of their position. The voice analysis step helped resolve the transcript where speaker attribution was ambiguous, which came up more than expected in the sections where candidates interrupted each other directly (the Cuomo/Lander exchange on sexual harassment, the Israel section near the end).

The reason this became one of the clearest examples in the project is that the candidates had no control over the questions. Campaign websites and position papers let you stay vague on anything inconvenient. A debate moderator asking "Would you vote for a rent freeze this year?" forces a yes or no from all seven candidates in the same room. A debate forces every candidate to answer the same question, so the outputs line up. A few of the positions from the debate are below.

The full analysis covers all thirteen topics. The table above is a sample. The point the pipeline makes visible is that the debate produced a kind of comparable data you can't get from press releases: same question, same room, every candidate on record.

The housing comparison below is the most direct example published during the project, and it covers seven candidates in a single post. Each candidate is presented in the same structure, and doing this by hand would mean reading dozens of source documents and manually pulling out unit targets, timelines, and track records. With the pipeline, every candidate's housing position got tagged with the same taxonomy label at bronze, chunked and keyed at silver, and entity-resolved at gold, so a consistent side-by-side extraction is just a query against the dataset.

NYC needs to build 50,000 to 70,000 new units per year to make a meaningful dent in the housing crisis. In 2024, the city completed 33,974 units. Here's where each candidate's plan lands against that baseline:

Stringer's plan leaves out a unit total on purpose, focusing instead on financing mechanisms and asset identification. That gap is itself a data point the structured schema makes visible: every other candidate committed to a number, and he didn't.

opencitizen.net/p/the-housing-crisis

The Mamdani piece takes a different approach from the debate analysis. Instead of extracting candidate positions, it pulls coverage of a single event from 23 outlets, tags each chunk with a source, sentiment score, and topic label, then uses that structure to write one article that accounts for what the press actually said across the board. National coverage of his win was mostly positive. Local coverage was more cautious. The piece holds both without picking a side.

The storytelling follows a fixed path: context, reaction, analysis, counterpoint. That structure doesn't change article to article. What changes is how the writer weights each section and which citations to surface. Writing stays consistent even when the source material isn't, because the organization is decided before the writing starts.

The underlying goal of OpenCitizen was to show how local events can be tracked and made relevant to the people most affected by them. The information can be published anywhere (a blog, newsletter, or social media app), but the key difference from the traditional news cycle is what's underneath it: a structured, queryable record rather than a pile of one-off posts that disappear into a feed. Building the database first, categorizing events, candidates, and positions as they happened, meant each new piece could layer on top of accumulated context instead of reconstructing it from scratch. Tracking everything by topic makes it possible to hold politicians accountable over time, because the record is persistent rather than scattered across individual stories.

The pipeline and the blog were the same project. The publication rate OpenCitizen achieved during the primary was a direct function of what the data infrastructure made possible.

Tracking the NYC mayoral race taught me a great deal about the challenges of publishing real-time content, citing information reliably, and building something that actually helps people make sense of what's happening in their city. The data pipeline and publishing system was ultimately a proof of concept for something larger: a state management system for cities, designed to track and surface the events that matter most to each resident. The goal is a digital twin of NYC that tracks and publishes news as it happens, and turns the city's record into a searchable database rather than something that scrolls past and disappears. As the dataset grows, the planned next phases include more complex cross-topic queries, scenario analysis, and causal modeling of how policy decisions play out across neighborhoods.