Election Pipeline Upgrades — Smarter Enrichment, Website Scraping, and Compass Filtering

What changed

Several updates to the election 2026 pipeline shipped today. The headline improvements:

• Smarter enrichment — the AI enricher now auto-fetches candidate website content and uses it for scoring, instead of relying solely on pre-collected evidence.
• --only flag works — you can now enrich a single candidate without triggering a full run of all 80+ candidates.
• Compass filtering — the Political Compass now hides candidates who haven't been scored yet, with a toggle to show them.
• Candidate withdrawals — three candidates have been marked as withdrawn and are no longer shown.
• Peter Shimmin manifesto update — full 64-page manifesto (V6 Digital) analysed with 13 policy positions.

The enrichment problem

The election enricher takes each candidate's bio, evidence, and manifesto data, sends it to the AI, and gets back a political compass position, policy scores, and a manifesto summary. The problem: many candidates had announced but hadn't been fully researched yet. They had a website URL in the system but no scraped evidence. The enricher would see "no evidence, no bio" and skip them entirely.

Worse, the --only flag — meant to target a single candidate for re-enrichment — didn't actually exist in the code. Running election-enrich --only james-cochrane would silently ignore the flag and enrich every un-enriched candidate in the database. For a campaign with 80+ candidates, that's expensive and slow.

How it works now

Website auto-fetch

When a candidate has a links.website URL (added via the quick-add candidate-links.json file or their seed data), the enricher now:

1. Fetches the page HTML with a 15-second timeout
2. Strips scripts, styles, navigation, and footers
3. Extracts up to 6,000 characters of clean text
4. Includes the text in the AI prompt alongside any existing evidence

This means a candidate who has nothing but a campaign website can still get a meaningful compass position. The AI sees the actual content of their site — their priorities, background, policy statements — and scores accordingly.

Manifesto pages are handled separately if the URL differs from the main website, and PDF manifesto URLs are noted in the prompt even though they can't be auto-fetched.

Single-candidate targeting

The --only <candidate-id> flag now works correctly:

pnpm --filter scraper election-enrich --only james-cochrane

This:

• Loads the fully merged candidate data (including links from candidate-links.json)
• Filters to just the named candidate
• Implies --force — re-enriches even if already scored
• Saves the enrichment to a per-candidate override file, not the base seed

The override-file approach means targeted enrichment doesn't risk overwriting the full 80+ candidate seed file. The results merge in automatically on the next election-import run.

Compass filtering

The Political Compass was showing all candidates, including those at position (0, 0) — the default for candidates who haven't been analysed yet. This was misleading because it made unscored candidates look like centrists.

Now:

• Candidates at (0, 0) are hidden by default on the compass chart, candidate directory, and positions table
• A "Show unscored" checkbox in the filter bar lets you see them if you want
• The positions table shows a count of how many candidates are still awaiting analysis

Candidate updates

Withdrawn candidates are now filtered from all election pages:

• John Wannenburgh (Ramsey) — deceased
• Kate Lord-Brennan (Douglas South) — not standing
• Tim Glover (Arbory, Castletown & Malew) — not standing

Peter Shimmin (Ayre & Michael) has had his full 64-page manifesto analysed, covering economy, cost of living, healthcare, housing, education, transport, immigration, environment, governance, digital policy, assisted dying, culture, and young people's issues. His key position: opposition to the proposed Mooir Vannin offshore wind farm.

Constituency normalisation now handles the inconsistency between "and"/"or" and "&" in constituency names, ensuring candidates aren't split across duplicate dropdown entries.

What's next

The enrichment pipeline is now in a state where adding a candidate's website link to candidate-links.json is enough to get them scored on the next run. The daily cron will handle the rest — fetch their site, score them, and rebuild the compass.

Candidates with published manifestos or detailed campaign websites will get the richest scoring. Those with minimal online presence will still appear but with fewer policy positions filled in. As the campaign heats up and more candidates publish manifestos, the compass will fill in automatically.