<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>AI Crawler on miti99</title><link>https://miti99.com/en/tags/ai-crawler/</link><description>Recent content in AI Crawler on miti99</description><generator>Hugo</generator><language>en</language><lastBuildDate>Mon, 16 Nov 2026 17:39:19 +0700</lastBuildDate><atom:link href="https://miti99.com/en/tags/ai-crawler/index.xml" rel="self" type="application/rss+xml"/><item><title>When AI Crawlers Went Strolling Through My Self-Hosted Gitea</title><link>https://miti99.com/en/post/2026/10/29/</link><pubDate>Thu, 29 Oct 2026 00:00:00 +0700</pubDate><guid>https://miti99.com/en/post/2026/10/29/</guid><description>&lt;p&gt;I self-host Gitea on an OCI Always Free server to keep mirrors of my GitHub repositories. I always thought of it as a quiet little corner only I ever visited — until I discovered three regular &amp;ldquo;guests&amp;rdquo; dropping by: ClaudeBot, GPTBot, and Reflectionbot. Starting from a few unusual signs in the logs, I tracked down a private repository that had unintentionally been made public, then figured out how to block the crawlers while keeping all my other apps running smoothly. Here is the full story, from discovery to mitigation and follow-up monitoring.&lt;/p&gt;
&lt;h2 id="discovery"&gt;Discovery&lt;/h2&gt;&lt;p&gt;On the morning of October 29, 2026, I noticed that the server running Gitea in Japan was unusually busy. When I opened Traefik&amp;rsquo;s access log, I realized the logs were rotating so fast that the retained portion only covered about 4.5 hours — from 17:55 to 22:22 UTC on October 28. In just that window, there were &lt;strong&gt;86,765 requests&lt;/strong&gt;.&lt;/p&gt;
&lt;table&gt;
 &lt;thead&gt;
 &lt;tr&gt;
 &lt;th&gt;User agent&lt;/th&gt;
 &lt;th&gt;Requests&lt;/th&gt;
 &lt;/tr&gt;
 &lt;/thead&gt;
 &lt;tbody&gt;
 &lt;tr&gt;
 &lt;td&gt;ClaudeBot (Anthropic)&lt;/td&gt;
 &lt;td&gt;39,860&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;GPTBot (OpenAI)&lt;/td&gt;
 &lt;td&gt;26,901&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;Reflectionbot (Reflection AI)&lt;/td&gt;
 &lt;td&gt;16,412&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;Bun (my own &lt;code&gt;gitea-mirror&lt;/code&gt; syncing)&lt;/td&gt;
 &lt;td&gt;3,587&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
 &lt;td&gt;Others (DuckDuckBot, OAI-SearchBot)&lt;/td&gt;
 &lt;td&gt;5&lt;/td&gt;
 &lt;/tr&gt;
 &lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Around 95% of the HTTP 200 responses came from &lt;code&gt;src&lt;/code&gt;, &lt;code&gt;commit&lt;/code&gt;, and &lt;code&gt;raw&lt;/code&gt; pages across 86 mirrored repositories. The crawlers were walking through &lt;strong&gt;every single commit of every single repo&lt;/strong&gt;, and each commit had multiple pages to fetch. Added together, the request volume snowballed fast.&lt;/p&gt;
&lt;p&gt;Looking back at the full month&amp;rsquo;s metrics, traffic was virtually zero until October 10. Starting October 11, it spiked and stayed consistently high:&lt;/p&gt;
&lt;p&gt;&lt;img src="https://miti99.com/post/2026/10/29/requests-by-day.png" alt="Daily request count from September 30 to October 28"&gt;&lt;/p&gt;
&lt;p&gt;In total, that came out to nearly &lt;strong&gt;13.9 million requests&lt;/strong&gt; and &lt;strong&gt;1.48 TB&lt;/strong&gt; of bandwidth — all to serve a Gitea instance meant for an audience of one:&lt;/p&gt;
&lt;p&gt;&lt;img src="https://miti99.com/post/2026/10/29/traffic-overview.png" alt="Traffic overview: requests, unique visitors, bandwidth, error rate, and P95 latency"&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;If you self-host on a VPS, keep a close eye on bandwidth costs.&lt;/strong&gt; Data downloaded by bots still counts toward your server&amp;rsquo;s outbound transfer quota. Depending on your provider and plan, going over your allowance can mean surprise overage charges or throttled speeds. I got lucky because OCI&amp;rsquo;s outbound bandwidth allowance is very generous, so this incident didn&amp;rsquo;t cost me anything extra. On a VPS with a tighter cap, though, 1.48 TB from crawlers alone could easily turn into an unexpected bill.&lt;/p&gt;
&lt;p&gt;Nearly all of the traffic came from bots, and almost all of it originated in the United States:&lt;/p&gt;
&lt;p&gt;&lt;img src="https://miti99.com/post/2026/10/29/client-types.png" alt="Client type breakdown: almost entirely Bots"&gt;&lt;/p&gt;
&lt;p&gt;&lt;img src="https://miti99.com/post/2026/10/29/traffic-by-country.png" alt="Traffic by country: United States accounts for 13.51 million requests"&gt;&lt;/p&gt;
&lt;p&gt;ClaudeBot alone accounted for 10.82 million requests and 1.4 TB of bandwidth. GPTBot chipped in another 2.52 million requests:&lt;/p&gt;
&lt;p&gt;&lt;img src="https://miti99.com/post/2026/10/29/ai-bots.png" alt="AI agents &amp;amp; bots table: ClaudeBot and GPTBot"&gt;&lt;/p&gt;
&lt;h2 id="figuring-out-the-cause"&gt;Figuring Out the Cause&lt;/h2&gt;&lt;p&gt;My first worry was whether my account had been compromised. Checking the logs, I found:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Every single request was anonymous — none of them were authenticated.&lt;/li&gt;
&lt;li&gt;No external client sent any POST, PATCH, push, or API requests. All write operations came from the server itself via &lt;code&gt;gitea-mirror&lt;/code&gt; running its sync jobs.&lt;/li&gt;
&lt;li&gt;All three crawlers were legitimate: GPTBot&amp;rsquo;s IPs matched OpenAI&amp;rsquo;s published ranges, ClaudeBot came from Anthropic&amp;rsquo;s IP space, and Reflectionbot&amp;rsquo;s reverse DNS resolved back to &lt;code&gt;reflection.ai&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;img src="https://miti99.com/post/2026/10/29/crawler-verification.png" alt="Verifying the three crawlers: Reflectionbot, GPTBot, and ClaudeBot are all genuine bots"&gt;&lt;/p&gt;
&lt;p&gt;Nothing pointed to an account breach. The bots were simply crawling pages that Gitea left publicly accessible to anonymous visitors: they requested &lt;code&gt;/robots.txt&lt;/code&gt; first, got a 404 because none existed, and proceeded to scrape every page returning HTTP 200. Looking back, I had left the front door wide open without realizing it; the primary issue to fix was my own configuration.&lt;/p&gt;
&lt;p&gt;While auditing further, I uncovered something much more concerning: &lt;strong&gt;one private repository had been exposed&lt;/strong&gt;. The source repo on GitHub was private, but its mirror on Gitea was &lt;strong&gt;public&lt;/strong&gt;, and Reflectionbot had already fetched 48 pages from it. Tracing the sync history led me to a stale record that &lt;code&gt;gitea-mirror&lt;/code&gt; had reused:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Originally, that repository name belonged to a public repo. When importing it the first time, &lt;code&gt;gitea-mirror&lt;/code&gt; saved the flag &lt;code&gt;is_private = 0&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Later, that public repo was moved to a different namespace.&lt;/li&gt;
&lt;li&gt;I created a new &lt;strong&gt;private&lt;/strong&gt; repository under the old name. The existing record in &lt;code&gt;gitea-mirror&lt;/code&gt; still pointed to that URL and kept the public flag. Because the tool treated it as the same repository, it never re-checked its visibility.&lt;/li&gt;
&lt;li&gt;Gitea kept pulling updates from that URL using my GitHub token. As a result, the contents of the new private repo were synced straight into the public mirror on Gitea.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Even setting &lt;code&gt;visibility: &amp;quot;private&amp;quot;&lt;/code&gt; in &lt;code&gt;gitea-mirror&lt;/code&gt; didn&amp;rsquo;t prevent this: that setting only applies to organizations created by the tool, whereas individual repository visibility is still taken from the GitHub private flag captured at the initial import.&lt;/p&gt;
&lt;p&gt;I audited everything and found 105 repositories readable without signing in, all under my personal Gitea account. All other organizations were private. Out of those 105 public mirrors, only the one I had just caught corresponded to a private repo on GitHub.&lt;/p&gt;
&lt;p&gt;Why Cloudflare hadn&amp;rsquo;t blocked the bots was even simpler: every DNS record for my apps was an &lt;code&gt;A&lt;/code&gt; record set to &lt;strong&gt;DNS only&lt;/strong&gt; (grey cloud). Bots were hitting the origin server directly, so their requests never passed through Cloudflare to be filtered in the first place.&lt;/p&gt;
&lt;h2 id="mitigation"&gt;Mitigation&lt;/h2&gt;&lt;p&gt;&lt;strong&gt;Step one was stopping any further data exposure.&lt;/strong&gt; I stopped both Gitea and &lt;code&gt;gitea-mirror&lt;/code&gt;, then switched the exposed repository to private.&lt;/p&gt;
&lt;p&gt;Initially, I considered putting all my applications behind Cloudflare Tunnel and Cloudflare Access. However, that approach would have required quite a bit of rework: changing URLs in Coolify to &lt;code&gt;http://&lt;/code&gt;, reconfiguring each app, and sorting out webhooks. I set that aside for later and chose a less invasive first step: &lt;strong&gt;enabling the Cloudflare proxy (orange cloud)&lt;/strong&gt; on my DNS records. They still point to the same IP, Traefik and Coolify keep working the same way, and Cloudflare sits in the middle to filter out bots.&lt;/p&gt;
&lt;p&gt;On the afternoon of October 29, I rolled out the changes step by step:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Switched SSL/TLS from Full to Full (strict).&lt;/strong&gt; Traefik continues using its Let&amp;rsquo;s Encrypt certificates and renews them via HTTP-01 even with the proxy enabled. I avoided Flexible mode because Traefik already redirects HTTP to HTTPS; combining the two would cause an infinite redirect loop.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Tested on a low-risk app first&lt;/strong&gt; — a web-based terminal. I verified the &lt;code&gt;server: cloudflare&lt;/code&gt; response header, page loading, and WebSocket connectivity.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Enabled the bot-blocking layers&lt;/strong&gt; available on the Cloudflare Free plan:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;AI bot policies&lt;/strong&gt;: set Training, Agent, and Search to Block, since I have no reason for these self-hosted apps to show up in AI search results.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Bot Preference Sync&lt;/strong&gt; (Cloudflare-managed &lt;code&gt;robots.txt&lt;/code&gt;): automatically injects &lt;code&gt;Disallow&lt;/code&gt; rules for AI bots into &lt;code&gt;robots.txt&lt;/code&gt;, or serves one if the app doesn&amp;rsquo;t have its own.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;A custom WAF rule&lt;/strong&gt; to catch anything the built-in policies might miss:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;
&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;
&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt;1
&lt;/span&gt;&lt;span class="lnt"&gt;2
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;
&lt;td class="lntd"&gt;
&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;(cf.verified_bot_category in {&amp;#34;AI Crawler&amp;#34; &amp;#34;AI Assistant&amp;#34; &amp;#34;AI Search&amp;#34;})
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;or (http.user_agent contains &amp;#34;Reflectionbot&amp;#34;)
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;
&lt;/div&gt;
&lt;/div&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Enabled the proxy on the wildcard record&lt;/strong&gt; &lt;code&gt;*.miti99.dev&lt;/code&gt;. Flipping that single switch covered nearly 20 apps at once, and any new app using the wildcard record will be proxied by default. If I ever need to roll back, I can just toggle the proxy off.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Enabled the proxy on Coolify last&lt;/strong&gt;, then tested GitHub push deployment webhooks, Sentinel metrics coming from my second server, MCP, and real-time WebSocket connections.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Configured Traefik to trust Cloudflare&amp;rsquo;s IP ranges&lt;/strong&gt; via &lt;code&gt;forwardedHeaders.trustedIPs&lt;/code&gt;, so backend applications see the real visitor IP instead of Cloudflare&amp;rsquo;s edge IP.&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;I also deliberately &lt;strong&gt;left Bot Fight Mode turned off&lt;/strong&gt;. On the Free plan, its challenge mechanism can block legitimate automated traffic, and WAF rules aren&amp;rsquo;t allowed to bypass it. For my setup, turning it on risked breaking GitHub webhooks, Sentinel, MCP, Nextcloud sync, and even Git operations over HTTP.&lt;/p&gt;
&lt;p&gt;Testing with &lt;code&gt;curl&lt;/code&gt;, requests with a normal browser user agent returned HTTP 200, while requests spoofing GPTBot, ClaudeBot, or Reflectionbot user agents received an immediate HTTP 403 at Cloudflare&amp;rsquo;s edge. Once that was confirmed, I started Gitea back up behind the proxy.&lt;/p&gt;
&lt;p&gt;Even so, routing traffic through Cloudflare comes with a few limitations to keep in mind:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Request bodies larger than 100 MB are rejected. Nextcloud uses chunked uploads, so it isn&amp;rsquo;t affected by this limit.&lt;/li&gt;
&lt;li&gt;Requests that take longer than ~100 seconds without sending data can hit an HTTP 524 timeout. Long-running LLM calls that don&amp;rsquo;t stream responses are something to watch out for.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The origin server IP is still reachable directly.&lt;/strong&gt; Any bot that already knows the origin IP can talk straight to Traefik. Proxying filters crawlers that resolve via DNS through Cloudflare, but it doesn&amp;rsquo;t replace proper access control.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="follow-up-monitoring"&gt;Follow-Up Monitoring&lt;/h2&gt;&lt;p&gt;The post-cutover logs illustrated that last point clearly. I enabled the proxy for Gitea at 09:14 UTC (16:14 Vietnam time), then filtered the access log on the origin server to see whether the three bots were still getting through:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;
&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;
&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt;1
&lt;/span&gt;&lt;span class="lnt"&gt;2
&lt;/span&gt;&lt;span class="lnt"&gt;3
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;
&lt;td class="lntd"&gt;
&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-bash" data-lang="bash"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;for&lt;/span&gt; ua in GPTBot ClaudeBot Reflectionbot&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&lt;span class="nv"&gt;$ua&lt;/span&gt;&lt;span class="s2"&gt; &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;grep -c &lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&lt;span class="nv"&gt;$ua&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt; access.log&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;done&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;
&lt;/div&gt;
&lt;/div&gt;&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;ClaudeBot&lt;/strong&gt; stopped completely at 09:14:20 — almost immediately after the DNS change.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Reflectionbot&lt;/strong&gt; trickled in for about 6 more minutes, likely due to DNS caching.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;GPTBot&lt;/strong&gt; kept sending requests directly to the origin IP — another &lt;strong&gt;7,056 requests over the next hour&lt;/strong&gt; — before finally stopping at 10:14. The &lt;code&gt;ClientAddr&lt;/code&gt; field showed OpenAI&amp;rsquo;s IPs rather than Cloudflare&amp;rsquo;s. My guess is that the bot caches DNS responses longer than the record&amp;rsquo;s TTL.&lt;/li&gt;
&lt;li&gt;From that point until the time of writing — about 4 hours — the origin server recorded &lt;strong&gt;0 requests&lt;/strong&gt; from all three bots.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;To make sure this setup stays effective, here&amp;rsquo;s what I&amp;rsquo;ll continue monitoring:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Security → Events&lt;/strong&gt; in Cloudflare: verify that AI bots continue to be blocked and check whether any new user agents need to be added to the WAF rule.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Traefik access logs on the origin server&lt;/strong&gt;: filter using the snippet above. If a crawler reappears with a &lt;code&gt;ClientAddr&lt;/code&gt; outside Cloudflare&amp;rsquo;s IP ranges, it&amp;rsquo;s hitting the origin IP directly. At that point, I&amp;rsquo;ll either lock the firewall down to allow only Cloudflare IPs or revisit the Cloudflare Tunnel approach.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;robots.txt&lt;/code&gt; on each host&lt;/strong&gt;: run &lt;code&gt;curl https://&amp;lt;app&amp;gt;.example.com/robots.txt&lt;/code&gt; to confirm Cloudflare&amp;rsquo;s AI bot &lt;code&gt;Disallow&lt;/code&gt; rules are present.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;New applications&lt;/strong&gt;: Coolify creates new DNS records in DNS-only mode by default. Any host with its own dedicated record needs the proxy toggled on manually; forgetting one leaves a direct path open again.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Cloudflare IP ranges&lt;/strong&gt;: although they rarely change, I still need to keep Traefik&amp;rsquo;s &lt;code&gt;trustedIPs&lt;/code&gt; list updated when they do.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;I still have a few follow-up tasks left: enabling &lt;code&gt;REQUIRE_SIGNIN_VIEW&lt;/code&gt; so Gitea requires authentication before viewing any page, wiping the existing mirrors, and rotating the GitHub token used by &lt;code&gt;gitea-mirror&lt;/code&gt;. Requiring sign-in is what truly stops outsiders from reading repositories; the Cloudflare proxy is just an extra layer of defense in front. I&amp;rsquo;m also considering sending data deletion requests to Reflection AI, OpenAI, and Anthropic.&lt;/p&gt;
&lt;h2 id="takeaways"&gt;Takeaways&lt;/h2&gt;&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Anything public on the internet will eventually be scraped thoroughly.&lt;/strong&gt; AI crawlers don&amp;rsquo;t stop at your homepage — they will walk through every single commit.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Double-check visibility settings when syncing across systems.&lt;/strong&gt; A single stale record under a reused repo name was enough to mirror a private GitHub repository into a public Gitea repo.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;If you want Cloudflare to filter traffic, traffic actually has to go through the proxy.&lt;/strong&gt; For my small self-hosted setup, flipping on the proxy was a practical first step before moving to tunnels.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Audit your automated integrations before enabling Bot Fight Mode.&lt;/strong&gt; When your server also receives webhooks and serves non-browser clients, bot protection has to account for those workflows too.&lt;/li&gt;
&lt;/ul&gt;</description></item></channel></rss>