The numbers didn't add up
Smart Island serves open data for a population of 84,000. We have a healthy audience — researchers, policymakers, journalists, developers pulling data through our MCP tools. But we don't have 39 million DNS requests per month worth of audience.
When we checked our DNS analytics in September 2026, that's what we found. Thirty-nine million requests. For context, if every person on the Isle of Man visited our site 10 times a day, every day, that would be about 25 million monthly requests. We were well above that.
Something was off.
Who was actually visiting
Digging into the request logs told the story:
AI crawlers were hammering our API endpoints. Large language model companies routinely crawl the open web to gather training data. Our site — with 35+ structured datasets, 93 MCP tool endpoints, and hundreds of pages of clean, well-formatted data — was exactly the kind of target their crawlers love. They weren't reading a page and moving on. They were systematically requesting every endpoint, every dataset, every variation.
Bots from everywhere. Not just AI companies. Vulnerability scanners, SEO scrapers, content mirrors, and generic web crawlers were all hitting us constantly. Many weren't respecting robots.txt. Some were rotating through IP addresses to avoid rate limits.
The MCP server was wide open. Our Model Context Protocol server at mcp.smartisland.im exposes 93+ tools for querying IoM data. That's fantastic for legitimate AI assistants — and equally attractive to anything that discovers it and decides to enumerate every tool.
Global distribution made no sense. Requests were coming from every continent. Smart Island is an Isle of Man platform. We expect UK, Ireland, and some international research traffic. We don't expect sustained high-volume traffic from data centres across Asia, South America, and Africa.
What Cloudflare gives us
We migrated DNS and proxying to Cloudflare. The immediate benefits:
Bot management
Cloudflare's bot detection separates legitimate users and known-good crawlers (Googlebot, Bingbot) from the noise. Automated threats get challenged or blocked before they reach our server. We no longer waste Azure compute cycles serving responses to machines that have no business asking.
DDoS protection
An open data platform is an unusual DDoS target, but we'd seen traffic spikes that looked suspiciously coordinated. Cloudflare absorbs volumetric attacks at the edge — they never reach our origin server.
Caching at the edge
Static assets, page data JSONs, and API responses for public endpoints can now be served from Cloudflare's edge network. A researcher in London gets our data from a London PoP, not from our Azure VM. Faster for users, less load on our server.
Analytics that actually help
Cloudflare's analytics distinguish between human visitors, known bots, and automated threats. For the first time, we can see our actual audience separate from the noise. The real traffic is a fraction of the 39 million — and it's much more interesting to understand.
AI crawler controls
Cloudflare provides specific controls for AI training crawlers. We can allow legitimate AI assistants (which use our MCP tools as intended) while blocking bulk scraping by training pipelines. Our data is open and free to use — but there's a difference between an AI assistant querying our API to help a user, and a training crawler downloading our entire database.
The server impact
Before Cloudflare, our Azure VM (2 vCPU, 8GB RAM) was handling every request directly. We'd seen the server peg at 395% CPU after builds — partly because of Next.js ISR revalidation, but bot traffic compounded it. Our Azure SQL database hit 100% DTU (the billing unit for Azure SQL compute) with skills page queries running 14,000+ times per hour.
After Cloudflare: the origin server sees only legitimate traffic that passes through the proxy. Combined with the caching improvements we made to our database queries, the server load has dropped substantially.
Open data, not open season
Smart Island is committed to open data. Everything we publish is free to access, and our MCP server is designed specifically so AI tools can query Isle of Man data programmatically. We're not hiding behind Cloudflare to restrict access.
But there's a difference between serving data to people and tools that use it, and serving as an unwitting training corpus for models that will never attribute or link back to the source. Cloudflare lets us draw that line without affecting legitimate users.
If you're building something with IoM data — whether through our MCP tools, the API, or just browsing the data catalogue — nothing changes for you. The data is still there, still free, still open.
We just stopped subsidising the bot economy with our Azure bill.
Smart Island runs on a single Azure VM serving 140+ pages, 35+ datasets, and 93+ MCP tools for the Isle of Man. Infrastructure details: About.