A quick platform note
Smart Island has grown a fair bit since we started. To put numbers on it:
- 30+ open datasets, each with its own pages, charts, and APIs
- ~200 pages generated from those datasets (occupations, courses, jobs, biosphere, vehicles, planning, ships, weather, the lot)
- Several million rows of underlying data sitting in the database — species occurrences alone are 1.5 million, vehicles are another 80k, jobs and their enrichments add another quarter-million, and the rest add up
- ~100 MCP tools exposing all of that to AI assistants
- A codebase that's tripled in size since the first release
That's broadly the level we wanted to reach. It comes with side-effects.
What was happening
Every time we deploy a change — which, depending on the week, can be several times a day — the site has to be rebuilt from scratch. Each of those ~200 pages has to be pre-prepared so it's instant to load when you visit. For most of the project, that worked fine: build runs, every page gets its data, site goes live, traffic flows.
With the data growing into the millions of rows, the build-time data fetches started timing out. Not the live site — the build process specifically. Every page asking the database for its stats at the moment of deploy started hitting limits that didn't exist a year ago. A handful of busy pages were enough to occasionally tip the build over and stall a deploy.
It's a normal growing pain. The same pattern shows up at most data-heavy services as they cross a certain threshold: behaviour that was a small cost becomes a meaningful one.
What we changed
Without going into the deep weeds — we've moved some of the busiest pages off the pre-build path and onto an on-demand path. They used to be prepared once at deploy time and cached; they're now prepared the first time someone visits, then cached briefly.
What you might notice:
- First visit to certain dashboards (homepage, /stats, /ai-observatory, /observatory, advisor pages, etc.) may take a fraction of a second longer than it used to — maybe 100-300 ms.
- Every subsequent visit is the same as before (the cache layer kicks in).
- We can now deploy reliably even as the underlying data keeps growing.
The trade-off felt fine: a tiny bit slower on the first hit, no chance of a deploy stalling because some background query took a moment. We'd rather be able to ship updates than save 200 ms on a first page-load.
And while we were at it
A couple of related housekeeping changes shipped this week too:
- Patched a security advisory in the underlying Next.js framework — CVE-2026-44578, if you're curious about the technical detail. Affected self-hosted instances of the framework; we patched the same day it was announced.
- Tidied the deploy script to be more forgiving of transient hiccups (network blips, brief external service delays). Fewer "please retry" moments.
- Cleaned up some background processes that had drifted out of supervision over time. Same total work, less duplicate noise.
None of these are exciting individually. Together they buy the project more headroom before the next round of growth makes the next set of changes necessary.
What's next
The honest version: we're going to keep growing the data. New datasets are in flight (more on those when they ship), the Biosphere Observatory keeps absorbing GBIF updates, the vehicle and property catalogues refresh on schedule, and the AI advisor pipelines keep humming along.
The build pipeline now has a clearer path through it. The next bottleneck will be something else — that's how this goes. When it shows up we'll deal with it the same way: notice, diagnose, patch, write a short note about it.
Thanks for sticking with us. The site's not going anywhere. The plumbing's just been freshened up.
— If you spotted any wobble in the site mid-week, that was this. Back to normal now.