Performance Overhaul, AI Fluency Rubric Matrix, and 65 Automated Pipelines

Performance Overhaul, AI Fluency Rubric Matrix, and 65 Automated Pipelines

A second round of updates today, focused on performance, a major redesign of the AI Fluency page, and documenting the automation that keeps the platform ticking.

Performance: 72 Pages Converted to ISR

The single biggest change today: 72 pages have been converted from force-dynamic (rendered fresh on every request) to Incremental Static Regeneration with a 5-minute revalidation window.

Previously, every page load hit the server and waited for a full render. Now, pages are pre-built and served instantly from cache, regenerating in the background every 5 minutes. The result is dramatically faster load times across the platform.

What changed

• 32 data pages (vehicles, aircraft, crime, companies, weather, economy, etc.) — now ISR with 5-minute revalidation
• 17 biosphere and observatory pages — same treatment
• 19 election, advisor, and employer pages — ISR
• Blog pages — ISR with 1-hour revalidation (content changes less frequently)
• Sitemap and RSS feed — hourly regeneration

Pages that genuinely need real-time data (job listings, individual job pages, API routes) remain dynamic.

Server-side improvements

• Standalone output mode enabled in Next.js — smaller deployment footprint, faster cold starts
• PM2 cluster configuration prepared for dual-process mode (the server has 2 vCPUs)

AI Fluency Rubric Matrix

The AI Fluency page has been completely redesigned as a full-screen immersive experience with a task-level rubric matrix.

The four levels

Inspired by Zapier's AI Fluency Rubric, every task and knowledge area in a role is now assessed across four levels of GenAI fluency:

• Unacceptable — uses GenAI occasionally but core process unchanged; no evidence of improvement
• Capable — uses GenAI to operate at a meaningfully higher level; standard practice, not occasional
• Adoptive — orchestrates GenAI and builds systems that elevate how they work; creates reusable workflows others can use
• Transformative — re-engineers how work happens; processes run largely automatically, humans work from exceptions

What the page shows

• Job metadata — title, employer, salary, location, O*NET classification, automation risk
• Quick wins and tools — immediate actions candidates can take this week, plus specific GenAI tools to learn
• Full rubric matrix — every task and knowledge area as a row, with descriptions of what each fluency level looks like for that specific item
• Employer signals — interview questions and red flags for hiring managers
• Previous/Next navigation — browse between jobs without returning to the listing
• GenAI caveat — the rubric uses "GenAI" throughout, with a note that this applies to Microsoft Copilot, ChatGPT, Claude, Gemini, or any organisational AI platform

The matrix is generated per O*NET occupation code, so all jobs sharing the same occupation get the same rubric — no redundant AI calls. New occupations are enriched automatically as part of the daily pipeline.

Behind the Scenes: 65+ Automated Pipelines

SmartIsland runs 65+ automated pipelines on a structured schedule to keep 30+ datasets current. Here is what happens without any human intervention:

Daily (every day, 05:00–08:00 UTC)

Every 3 hours

Weather data refresh — IoT sensor readings and forecast updates.

Nightly (22:00 UTC)

4 parallel vehicle scrapers run for 12 hours each, incrementally updating the 80,000-vehicle register.

Weekly (Sunday, 04:00–14:00 UTC)

• Vehicle processing: normalisation, AI narrative, precompute (04:00–05:00)
• 11 dataset precomputes: aircraft, property, crime, companies, FSA, GSC, ships, weather, World Bank, MUA, telecoms (05:00–09:00)
• 7 weekly imports: parliament, grants, OSM, heritage, nature, marine, FIU (07:30–10:00)
• Open data downloads: regenerates all CSV/JSON download bundles (10:00)
• Market analysis: weekly snapshot, 5 AI advisors (careers, economics, employer, strategy, sector), sentiment analysis, biosphere advisor, and the weekly digest (12:00–14:00)

Weekly (Monday, 06:00–10:00 UTC)

Election pipeline: candidate news scraping, discovery, enrichment, weekly campaign digest, and page data rebuild.

Monthly (1st, 02:00–10:00 UTC)

• 4 live source rescrapes: aircraft register, ship register, companies registry, World Bank
• 4 government imports: health, planning, elections, economy
• 22 AI narratives: every dataset gets a fresh AI-generated analytical narrative using Azure OpenAI
• Species assessment and FOI import

The numbers

• 65+ distinct pipeline runs per cycle
• 22 AI narrative generations monthly (Azure OpenAI)
• 5 weekly AI advisors covering careers, economics, employers, strategy, and sector analysis
• 4 parallel vehicle scrapers running 12 hours nightly
• 30+ datasets kept current across jobs, vehicles, biosphere, economy, infrastructure, and more

All pipelines are idempotent — they can be re-run safely without duplicating data. New occupations, species, or datasets are picked up automatically. Failed runs are logged and don't block subsequent pipelines.

What's Next

• AI-generated fluency matrix content for all ~500 active occupations (currently using intelligent fallback descriptions)
• PM2 cluster mode activation for better CPU utilisation
• Nginx static asset caching headers
• Continued expansion of IoT sensor network