For years a website had two kinds of visitors: people and search engine crawlers. Now there is a third — an AI agent acting on behalf of a user, comparing prices, reading delivery terms, filling in an enquiry form or trying to book a time. Most small-business websites in Estonia are neither prepared for this nor have consciously decided whether they want that traffic. The decision gets made for them, usually by some default setting.
There is no neutral option
If you don't decide, your robots.txt, your hosting firewall or a security service default decides for you. The outcome can go either way: either an uncontrolled amount of automation roams your site, or your price list is effectively invisible to agents. Neither extreme is a management decision — it's an accident.
The practical frame is simple. If your business runs on comparison and availability — online shop, accommodation, service booking, spare parts, quote requests — then an agent is a potential customer and worth serving. If your content itself is the product — a paid database, curated material, a painstakingly built listings base — then machine traffic is mostly a cost and a competitor's tool, and limiting it makes sense. Most companies sit somewhere in between: some pages you want read, others you don't.
What an agent actually needs from your site
An agent doesn't need beautiful design. It needs facts in machine-readable form. In practice that means a few boring things you can fix in a single working day: price, availability, delivery time, service area and contacts as text rather than inside an image or PDF; structured data on product and service pages; forms that work even when every button isn't a bespoke JavaScript widget; stable URLs that don't change with every campaign.
We also recommend building one plain "facts page": company basics, list of services, pricing logic, opening hours and terms in clear text. It helps the agent, the search engine and a new employee alike. We build such pages for clients increasingly often, precisely because the same content serves three audiences at once.
Where the line is: reading is not the same as transacting
Let's be honest here. Letting an agent read your pages is low risk. Letting it carry out transactions in your self-service portal is something else entirely. When someone hands a third-party tool their account credentials, you have no idea who actually placed the order — and in a dispute, that becomes your problem.
A sensible minimum: a rate limit on requests, a confirmation email or message for every booking and order, a clear log of which channel a transaction came through, and human confirmation for larger or unusual orders. And one honest caveat — this topic isn't relevant to everyone. If you sell complex B2B services where every deal starts with a meeting and ends with negotiation, agent traffic brings you nothing right now. In that case the right decision is to put the subject on a watch list and spend your time on something more profitable.
Start with logs, not opinions
Before rebuilding anything, spend one hour with your website logs. What share of requests comes from machines? Are they old crawlers, scrapers, or newer agent-type clients? Which pages do they visit? Do they get stuck anywhere? That one hour teaches you more than ten opinion pieces, because every site looks different.
Then write a one-page decision: which pages are open, which are closed, what the limits are, and who reviews this once a quarter. The document matters more than the technology — an IT partner can change the technical configuration in an hour, but without an agreed rule everyone changes it in a different direction. In our experience this small written decision prevents most of the later surprises: both the traffic that suddenly disappears and the server load that suddenly appears.