The Database of Human Intent: Using Google to Optimize for AI Search

AI technology has reached a level of human reasoning that will have sweeping effects on both society and search.

Take a look at the Google Deepmind team’s roadmap for AI:

There are specific risks associated with each level of AI that the Deepmind team has outlined. It gets interesting at level 3: “rapid societal change” could be positive, negative, or everything in between. At level 4, you may begin to feel concerned: “societal-scale ennui, mass labor displacement, decline of human exceptionalism.” There’s no doubt about it: those are serious risks.

In mid-September 2024, OpenAI previewed their new o1 model — and it performs at level 4, exceeding PhD-level humans with its capabilities.

There’s no doubt that AI technology is transforming everything it touches, and that includes the search landscape. What does this mean for brands already wondering how to compete in an AI-powered world of search?

It can feel alarming at first, but here at Botify we’ve been keeping a finger on the pulse of the industry for over a decade. This year, the theme of our annual Botfiy Connect conference was Leading in the Search Revolution — almost every session focused on how brands can come out ahead. In the end, it all came down to one vital message: 

Although AI tech in search is a new frontier, it actually brings with it an abundance of opportunity — and you already have everything you need to succeed.

Organic search data is your superpower in the age of AI

For over twenty years, we’ve been typing every question that comes to mind into a search box.

Best breakfast near me
Where to buy running shoes
Who wrote war and peace

The idea of Google as a database of intentions was pioneered by Wired co-founder John Battelle. Questions like these, and the human behavior and metrics that accompanied them — CTR, bounce rate, time on page, pogo sticking, and more — helped shape the search engine indexes that have cataloged the world’s content. Prioritized and ranked by more variables than you can imagine, the database that stores all human knowledge was the perfect building block for the next step in technology: generative AI.

We can assume that the top AI models — OpenAI, Anthropic, Google, Meta, Amazon, and so on — have already crawled 99% of the internet’s valuable resources. All the data that powers the AI technology driving the most change in search today is completely accessible to you — it lives within Google Search Console.

The search interface has changed, but consumer intent hasn’t

First, let’s cover what’s actually changed in search, and what hasn’t. We’ll start at the beginning.

Years ago, before Google or Bing, we had the Directory Age. This was when people would head to libraries and search through a card catalog for the information they needed. When it moved to the internet, we had databases like Lycos and Yahoo that were a long list of categories you could click into to discover websites.

Next came the Search Age. This is the era of the search box, a place where anyone can type a question or need and receive an answer almost instantly. It represented a huge leap in efficiency: the effort of finding the right category and best answer was done for you, provided by the search engine that had cataloged relevant website content into an index.

Now, we’re entering the Intelligence Age. In this age, consumers can have ongoing, interactive conversations with AI assistants that learn their preferences, personalize their responses, and make search even more efficient and easy. LLMs contain all the knowledge of the search index they’ve trained upon, and combine that static knowledge with live retrieval, or referencing a search engine index (like the Bing API) to find, understand, and then summarize recent content.

Across these ages, it’s clear that what has changed is the interface powering search. What hasn’t changed is the consumer intent behind the searches.

Consumers from 2000 needed to know where to buy a running shoe nearby, and they consulted their local Yellow Pages directory. In 2012, they were able to consult Google for results based on their location. Now, in 2024, consumers still need running shoes, and they have more options than ever — and with a quick request to an AI assistant, they’ll receive a summary of their known preferred brands, where to find them, and why they’re the best, without having to manually explore a list of links on a search engine results page.

You already know every question your customers will ask AI

Within Google Search Console is all the information you need to satisfy consumers via AI-powered search. Your organic search data can anticipate future trends.

Whether people are asking a friend, a search engine, or an AI assistant, their core questions and needs don’t change. As long as you know your consumer’s intent, their desired outcome, and the journey they take to get there, you have the ability to craft a satisfying, ideal customer experience.

Control in a fracturing search ecosystem

We’ve all seen the Gartner headline predicting a 25% drop in search engine volume due to the rise of AI assistants and chatbots. It was an anxiety-provoking prediction, leading many brands to ask about the value of organic search when traffic is expected to drop.

The key to remember is that consumers won’t stop searching — they’ll just be searching across diversified platforms.

Rather than the manual action of querying Google and evaluating a list of results, clicking onto pages, seeing if the content is right, clicking back to the SERP, and trying again, consumers will be able to query AI assistants with efficiency and speed. The AI platform will do the search work for them, return a response, and allow the consumer to complete most of their customer journey — from awareness to consideration — directly in conversation, with no need to visit a website until they’re ready to purchase.

That means that while traffic may drop, the traffic received will be more qualified. Consumers will be closer to purchase by the time they hit your website, and this self-qualifying behavior will result in higher quality website traffic and better conversions overall.

"What we are seeing is that the AI experiences [on Bing] often offer more click transfer value over traditional search because we are really better able to understand the user and we are better able to understand the intent … so while clicks may go down, the revenue per click will go up."
Fabrice Canel, Principal Product Manager, Bing, at Botify Connect 2024

As a brand, you have the ability to control the narrative consumers encounter on AI-powered platforms. Here’s how.

1. Make sure your website is indexed across all major search engines

We know that AI platforms rely on search engine indexes to provide accurate, fresh content to consumers. If your content isn’t discoverable there, AI models won’t be able to learn from your content or cite it as an answer within conversations.

It’s important to make sure your freshest and most valuable content (such as product inventory, updated prices, news content, and more) is indexed as quickly as possible. Waiting for search engine bots to find and crawl new content can be inefficient and they may miss important updates. There are some actions you can take to get your content indexed more quickly:

• Generate optimized sitemaps: Using Botify Activation, you can remove complexity from your engineering team to generate optimized sitemaps that document every important page on your website (and only the important pages.) This reduces errors and omissions, and better indicates to search engines what they should prioritize.
• Push to Bing: The Bing API powers the majority of AI platforms. That means that while Google is still important, if your content isn’t visible in Bing, it’s not reaching every consumer it needs to. Take advantage of the Push to Bing feature in SpeedWorkers to proactively submit your content directly to Bing’s index, to guarantee visibility across Bing and all AI engines powered by Bing’s index.‍
• IndexNow: Use the IndexNow protocol to submit URLs that have changed to a number of top search engines to automatically request a crawl of those pages.

2. Understand how search engine bots and AI bots are behaving on your site

While data on AI in search is still largely a black box, you can get insight into how your content is being found and explored by AI platforms by analyzing their bot behavior via log files.

Log file analysis can tell you which bots are accessing your site, what content they’re finding (and not), if they’re encountering errors, and more. It’s one of the most valuable tools at our disposal for reliable and actionable data, and it can be analyzed automatically and at scale via AI Bot Tracking Reports in Botify Intelligence.

Learn more about log file analysis and AI bots here:

• What Are AI Crawler Bots?
• Understanding Web Crawlers: Strategies for Traditional and AI Search Bots
• Tracking AI Bots on Your Site with Log File Analysis

3. Use your existing keyword data to predict consumer questions to AI platforms

The keyword data you use today is your strongest indicator of what consumers will ask on AI platforms. Here are three ways to use that data to predict and prepare for appearing in queries across AI-powered search engines:

Analyze question patterns

The patterns behind consumer search behavior can help you understand your customer’s intent, give you insight into their journey, and help you plan a content strategy that satisfies their needs at every step on the path to purchase.

• First, identify question-based queries. These answer the “five Ws (and one H)”: who, what, where, when, why, and how. 
• Group similar questions together — this can reveal common consumer pain points and needs at different stages of the funnel. 
• Next, compare and contrast the language consumers are using to search for your products versus how you describe them on your website. Understanding how the consumer describes their need can help you better tailor your content for them; “noise-blocking headphones” tells a more vivid story about the outcome your customer seeks versus a term like “active noise cancellation.” 
• Finally, pay special attention to queries that indicate conversational search, such as “what’s the best [product] for [purpose]?” These longer-rail questions using natural language often lead to follow-up questions in discussions with AI assistants and chatbots.

Highlight content opportunities

Satisfying consumer needs in both AI-powered and traditional search requires a comprehensive content strategy that speaks to a customer’s intent, their desired outcome, and their journey to purchase from start to finish. 

Begin by identifying questions with greater search volume but low click-through rates. What’s driving impressions but not clicks? The answer could be a need to update your content to serve your consumer’s needs. It could also indicate that your customer is getting an answer directly on the SERP via a featured snippet or AI Overview, with no need to click on a website link for more information. Either way, it’s an opportunity to understand how your content is fulfilling their needs and driving them through the funnel.

Keep an eye out for seasonal trends that may be affecting how consumers research and find your products. Investigating what’s driving that seasonality helps you understand and optimize your content strategy for your customer journey.

Optimize your content strategy

Take everything you’ve learned about your consumer’s intent, desired outcome, and detailed customer journey and incorporate it into your content strategy for AI search. Directly answer the top questions you identified, create a content experience that’s structured to match the natural flow of a customer’s questions and needs, provide comprehensive details that anticipates follow-up queries, and overall address the full spectrum questions your consumer needs answered. 

The better your website provides in-depth, helpful answers, the better your opportunity to be referenced in conversational AI-powered search and traditional search alike.

4. Focus on ranking highly to appear in AI summaries

When it comes to live retrieval — the process of tapping into a search engine index and referencing fresh content — AI assistants only have so much time and effort. To provide fast, high-quality responses in conversation with a consumer, they will only pay attention to the first ten results or so, summarizing the content they find. 

To give your brand the best chance of appearing in an AI assistant’s citations and responses, you need to focus on getting your most valuable, important content ranked in the top ten results on traditional search engines. And because so many AI platforms rely on Bing, that also means ensuring you’re focusing on optimizing for engines beyond Google.

You have what you need to succeed in the Intelligence Age of search

Search has changed tremendously in the last year, and that doesn’t promise to stop anytime soon. Technology is advancing quickly, and those advances are entwined with search on a foundational level. The most important thing to keep in mind is that you have everything you need to succeed as a brand in this new era: you know your consumers, you have robust data to predict their behavior, and you have the technical expertise on your SEO team to strategically optimize your website to be found wherever consumers are searching.

Remember: the future is not something that just happens to us; it's something you can control, and it’s something we create together.