Two small files. One big competitive advantage. Why we’re updating every client site right now — and why you should too.

Something significant has changed in how people search for businesses, and most websites are not ready for it.

When someone opens ChatGPT and asks “Who provides free senior care placement in Sacramento?” or “What’s the best electrical contractor for high-voltage work in Northern California?” — they are not getting a Google results page. They are getting a direct answer, generated by an AI that has read and indexed websites across the internet.

That AI either knows about your business or it doesn’t. And right now, for most local businesses, the answer is: it doesn’t. Not because your website is bad. Because it was never set up to talk to AI search engines.

That’s exactly what we’re fixing. And it starts with two small files: a robots.txt update and a new file called llms.txt.

The Search Landscape Has Fundamentally Changed

Google is still the dominant search engine — that hasn’t changed. But a new category of search has arrived and is growing fast.

ChatGPT now handles over 100 million queries per week. Perplexity AI has become the go-to research tool for millions of professionals. Google itself has launched AI Overviews that answer questions directly at the top of the results page. Apple Intelligence is rolling out across every iPhone. And Claude (the AI that powers this very agency’s internal tools) is increasingly being used to research and recommend service providers.

“AI search isn’t coming. It’s already here. The businesses that get set up correctly today are the ones showing up in those results over the next 12–24 months.”

The fundamental difference between Google search and AI search is this: Google returns a list of links. AI search returns a direct answer — and usually recommends a specific business.

If your business is the one being recommended, you win. If an AI doesn’t know you exist, or doesn’t have the right information about you, someone else wins that customer.

Problem #1: Your Website May Be Blocking AI Crawlers

What is robots.txt?

Every website has a file called robots.txt sitting in its root directory. It’s a plain-text file that tells search engine crawlers which pages they are allowed to visit and which they should skip. Think of it as the bouncer at the door of your website.

The standard WordPress robots.txt looks something like this:

User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://yoursite.com/sitemap_index.xml

That “User-agent: *” line means “these rules apply to all crawlers.” It was written when “all crawlers” meant Google, Bing, and a handful of others. It worked fine for years.

But AI search engines have their own crawlers, and many security tools and server configurations — including Cloudflare’s Bot Fight Mode, WordPress security plugins like Wordfence, and hosting-level firewalls — are blocking those crawlers by default. They see an automated request that doesn’t look like a human browser, and they slam the door.

The result: ChatGPT, Perplexity, and Claude literally cannot read your website. Not a single page. Including your homepage, your service pages, and your contact information. The llms.txt file you worked hard to create is invisible to them.

The AI Crawlers You Need to Explicitly Allow

Bot Name	AI Platform
GPTBot	ChatGPT (OpenAI)
OAI-SearchBot	ChatGPT Search Results (OpenAI)
ClaudeBot	Claude AI (Anthropic)
anthropic-ai	Anthropic AI Training
PerplexityBot	Perplexity AI
Google-Extended	Google AI (Gemini, AI Overviews)
Applebot-Extended	Apple Intelligence

The fix is adding explicit permission for each of these crawlers to your robots.txt. It takes about two minutes in an FTP client. Here’s exactly what to add:

User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Allow: / User-agent: anthropic-ai Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: /

Important: do not replace your existing robots.txt. Add these lines to what’s already there, keeping the WordPress rules and your Sitemap line intact.

Problem #2: AI Search Engines Don’t Know Enough About Your Business

Even if your site is no longer blocked, AI search engines face a second challenge: your website was written for humans, not machines.

Your homepage, service pages, and about page are designed to be visually engaging and easy to navigate. They work great for a visitor who is clicking around. But when an AI crawler is trying to quickly extract the essential facts about your business — what you do, who you serve, where you operate, what makes you different, and what questions you answer — it has to work much harder to piece that together from design-heavy pages built in WordPress with Elementor.

Sometimes it gets it right. Often it gets it partially right. And occasionally it gets it wrong in ways that could steer potential customers elsewhere.

AI systems work best when they have a single, clear, authoritative source of truth for a business. That’s exactly what the llms.txt file provides.

The Solution: llms.txt — Your AI Business Card

What is llms.txt?

llms.txt is a new, emerging standard for websites — a plain-text markdown file that lives in the root directory of your website (the same folder as your homepage) and is specifically designed to be read by AI systems.

It is accessed at yourwebsite.com/llms.txt and tells any AI crawler exactly what it needs to know about your business in a clean, structured format it can read and reference with confidence.

Think of it this way: your website is your showroom. Your llms.txt is your press kit — a concise, authoritative summary that gives AI systems everything they need to accurately represent you in search results and recommendations.

What Goes Into an llms.txt File?

A well-built llms.txt file includes:

• A clear business description and unique value proposition
• Every service you offer, described in plain language
• Your complete geographic service area — cities, regions, and coverage details
• Your contact information, hours, and key website links
• Frequently asked questions written in natural language — the same way people actually ask AI assistants
• Trust signals: credentials, licenses, years of experience, notable clients
• A list of the specific search queries you want AI systems to surface your business for
• A usage policy that explicitly authorizes AI systems to cite and recommend you

The more specific and complete the file, the more confidently an AI can recommend your business when the right question is asked.

A Real Example

Here is a condensed excerpt from an llms.txt file we built for one of our clients:

# First Place Plus – Senior Home Care Placement Services > First Place Plus is a doctor-led, free concierge service that matches > families and seniors with thoroughly vetted, experienced in-home > caregivers throughout California — with active service coverage across > the Greater Sacramento region and the San Francisco Bay Area, > including San Jose, Santa Clara, Sunnyvale, and Palo Alto. ## Frequently Asked Questions Q: How much does First Place Plus charge families? A: Nothing. The service is completely free for families. Q: How is First Place Plus different from Care.com? A: A physician personally vets every caregiver before a family    ever sees them. Care.com shows unfiltered listings.

When a family in San Jose asks ChatGPT “Who provides free senior care placement near me?” — that file is why our client gets recommended.

Why This Is a Significant Opportunity for Local Businesses Right Now

Here is the honest truth about where we are: most local businesses have not done this yet. The llms.txt standard is less than two years old. The robots.txt updates for AI crawlers are something most web developers and marketing agencies have not gotten around to.

That creates a meaningful window.

If you are a plumber in Sacramento, a jeweler in Elk Grove, or a high-voltage electrical contractor serving Northern California — and you get your llms.txt in place before your competitors do — you become the default recommendation for AI-powered searches in your category. Not because you paid for it. Because you were there first with the right information.

• AI search favors businesses with clear, structured, machine-readable information
• Most of your competitors have not set this up
• The window to establish early authority is open right now
• This is a one-time setup that compounds over time

Getting into AI search is closer to planting a tree than running an ad. The best time to start was last year. The second best time is today.

What We’re Doing for Our Clients

We have begun rolling out AI search readiness updates across all of our active client websites. For each site, the process includes:

• A full AI SEO audit to assess whether the site is accessible to AI crawlers
• Identification and resolution of any server-level or plugin-level bot blocking (including Cloudflare Bot Fight Mode and Wordfence rate limiting)
• An updated robots.txt file with explicit permissions for all major AI crawlers
• A comprehensive, custom-built llms.txt file researched and written specifically for that business — including services, geography, FAQs, trust signals, and targeted search queries
• Ongoing updates to the llms.txt as the business evolves, adds services, or expands to new markets

Every client’s llms.txt is built from scratch — not a template. A senior care placement service needs a completely different file than a high-voltage electrical contractor, and a jewelry store needs something different again. The value is in the specificity.

What You Should Do Next

If you’re not an HM Marketing client and you want to know whether your site is set up for AI search, here are three things to check today:

• Test your robots.txt: Go to yourwebsite.com/robots.txt in a browser. If you get an error instead of a text file, or if you don’t see any AI crawler entries, you have a gap.
• Check for an llms.txt: Go to yourwebsite.com/llms.txt. If it returns a 404 error, the file doesn’t exist yet.
• Ask an AI about your business: Open ChatGPT or Perplexity and ask: “Tell me about [your business name] in [your city].” What comes back will tell you a lot about what AI systems currently know — and don’t know — about you.

If any of those three checks reveal a gap, we can help. We offer an AI SEO audit that covers exactly these areas, along with a full llms.txt build and implementation.

Ready to show up in AI search?

Reach out for a free consultation. We’ll review your current AI search readiness and walk you through exactly what it would take to get your business in front of the right AI-powered queries in your market.

📞  (916) 992-4744     ✉️  hello@helpingmerchants.com     🌐  helpingmerchants.com/contact-us/

Published by HM Marketing — Sacramento’s full-service digital marketing agency. Serving local businesses since 2015.