Mapping the Expanded Search Landscape

The past year has introduced a lot of new words into our SEO vocabulary: 

ChatGPT Search, Google AI Overviews, Microsoft Copilot, Perplexity.ai, Mistral le Chat. 

LLMs, RAG, answer engines, discovery engines, generative responses, AI-powered search. 

They mark a change in organic search: it’s not just being disrupted — it’s being transformed. The search landscape is expanding to include new technologies and platforms beyond Google, and the brands that invest in AI search today will be the ones consumers rely on tomorrow.

What does that investment look like?

• Understanding how AI technology is changing organic search
• Knowing which platforms consumers search on and why
• Skilling up your teams responsible for organic search
• Identifying the challenges and opportunities with AI search
• Finding solutions that optimize for both Google and AI search at the same time

So much change may feel overwhelming, but it’s more manageable than you think. Let’s cover everything you need to know about the expanded search landscape, and wrap up with actions you can take now without adding strain to your resources.

What is decentralized search?

One way we refer to the expanded search landscape is with the idea of decentralized search. “Decentralized search” is the expansion of organic search beyond Google.

For the past twenty or so years, people have defaulted to Google as their search engine of choice. It has enjoyed the majority of search engine market share, so well-used that its brand name became a verb. 

Suddenly, with the rise of AI technology, consumers are heading to new search destinations, drawn by exciting new experiences and ways to get information.

The timeline of AI-powered search expansion has been fast-paced:

• November 2022 - ChatGPT launches
• December 2022 - Perplexity AI debuts an AI-powered answer engine
• February 2023 - Microsoft debuts GPT-powered Bing Chat, later renamed Microsoft Copilot
• May 2023 - Google announces Search Generative Experience (SGE) in Google Labs
• May 2024 - OpenAI search engine rumors lead to a GPT-4o launch
• May 2024 - Google announces US rollout of AI Overviews (formerly SGE)
• June 2024 - Apple announces that ChatGPT will be incorporated into an upcoming version of iOS, called Apple Intelligence
• October 2024 - OpenAI launches ChatGPT Search, the much-anticipated search engine
• October 2024 - Apple Intelligence releases on iOS 18
• November 2024 - Perplexity Shopping is released
• November 2024: Mistral le Chat is announced
• December 2024 - ChatGPT announces 300 million weekly active users

There’s a growing opportunity for brands on non-Google platforms. Which ones are valuable, and how does SEO fit in?

AI-powered search engines

To optimize for your consumer’s journey in the expanded search landscape, you have to know the different platforms, what they’re used for, and how they work.

Here are the most popular AI-powered search engines to date:

• Google AI Overviews: You’ll find AI Overviews in the traditional Google results as an AI-generated summary answering the consumer’s question right under the search box.
• Google Gemini: Google’s GenAI chatbot, offering a conversational medium for search, as an AI assistant on Android phones, and across other generative AI applications
• Microsoft Copilot: Microsoft’s AI assistant, available across multiple Microsoft properties and directly within Bing’s search results (similar to Google AI Overviews.)
• ChatGPT Search: OpenAI’s conversational GenAI tool focused on providing “fast, timely answers with links to relevant web sources,” powered by data from partners and Bing’s search index.
• Perplexity: A self-described “answer engine,” Perplexity offers a conversational AI chatbot experience and has even expanded into the world of e-commerce with Perplexity Shopping.
• Apple Intelligence: An AI-powered experience built into iPhones, Macs, and other Apple products, with ChatGPT integration.
• Meta AI: Meta’s AI assistant available across various properties like Facebook, Instagram, WhatsApp, and Messenger.
• Mistral le Chat: A conversational AI work assistant using Mistral’s models and providing web search capabilities.
• TikTok: TikTok is now considered the go-to search engine for Gen Z and Gen Alpha, incorporating some Google search results into its responses.

Knowing which platforms are the most valuable for your brand relies on your consumer and their customer journey. Regardless, the single best thing about optimizing for AI search is this:

You don’t have to work harder to appear in AI search.

The work you already do to rank organically directly impacts your visibility in AI search. You just have to optimize for the engines you already focus on — Google and Bing — and make sure the tools you’re using support you with automation, data, and insights to benchmark your progress.

That’s where the role of technical SEO comes in. It’s more important than ever, with a potential impact that spans online consumer destinations beyond Google.

The role of technical SEO in AI search

The role of the technical SEO isn’t being displaced. In fact, this skillset is more important than ever. Technical search experts are the heroes at the forefront of change, the only people who can bridge the gap between a website’s technical health and the needs of consumers. 

We’ve seen sea change in search before — mobile-first indexing, featured snippets, the menagerie of algorithm updates over the years. It just hasn’t happened on such a scale or with the speed of AI technology. 

There are skills that technical SEOs will need to maintain, such as:

• Controlling the crawl/render/index/serve pipeline
• Automating manual and repetitive tasks
• Split testing optimizations, learning, and iterating
• Log file analysis
• Implementing structured data
• Incorporating strong content SEO into their overall strategy
• Optimizing site speed
• Translating technical progress into business metrics for leadership

And those they’ll need to gain or hone:

• Becoming familiar with different crawler bots and their purposes
• Controlling bot behavior on their site
• Understanding what search data feeds different AI platforms
• Expertise with AI technology — how it works, its limitations, and its evolution
• Leveraging and expanding automation & AI-powered solutions to save resources and scale

The secret to any brand’s success in AI search will be the expertise, creativity, and resilience of their technical team. The very people who have spent the last two decades demystifying Google algorithm updates and reverse-engineering how search works are exactly the right ones to lead the decentralized search revolution.

How it works: traditional vs AI search

How traditional search works

Traditional search engines, like Google and Bing, share a list of ranked website links pulled from their search index databases:

1. Crawler bots traverse the internet, following links to discover new content across websites
2. The pages they find are rendered, where the bot “sees” all the findable content on the page
3. Pages found and rendered are then cataloged in their website index
4. A series of algorithms ranks those websites according to relevance to a given search query, and it’s provided to consumers within that list of links

At this point, the consumer has a job to do: look through the links, read the title tags and meta descriptions, consume any helpful SERP features like images, Knowledge Graph elements, or People Also Ask questions, and choose a link to visit, where they’ll continue their customer journey and convert.

How AI search works

AI-powered platforms use the power of generative AI to make searching as easy for consumers as having a conversation:

1. Crawler bots traverse publicly available content (like the internet) to train and refine large language models (LLMs)
2. The LLM interprets the meaning of a search query
3. If applicable, the LLM consults the top-ranking results from a traditional search index (like Google’s or Bing’s) using retrieval augmented generation (RAG) to find and provide up-to-date website citations and links
4. The LLM generates a contextually relevant, often conversational response to the query
5. The consumer receives a complete answer with no need to visit a website or research on their own, unless preferred

This process, at once so different from traditional search and still dependent on its indexing power, results in a satisfying, simple experience for consumers. There’s no need to compare websites and find the best answer themselves — it’s summarized directly in the chat. If they’re using an AI assistant or chatbot, the consumer can even further refine their question to get more information or accuracy. The customer journey can be completed all in one place (the AI search interface), with no need to visit any website until a conversion action is required.

Why optimize for AI search?

If it implies fewer site visitors and less traffic, what’s the point? Why even optimize for AI search?

There are a few reasons:

Search technology will change, whether you optimize for it or not

First and foremost, it’s important to understand that AI-powered search is the present and future of organic search. Technology isn’t going backwards, and consumers are already adapting their behaviors — 300 million weekly active users on ChatGPT can vouch for that.

If you stay ahead of the changes now, you’ll be ready to capitalize on opportunities within AI search as they appear. If you take a wait-and-see approach, you’ll be playing catch-up as your competitors enjoy the results of investing early in optimizing their AI search presence.

Organic traffic will become more qualified and conversion rates will increase

There’s an upside to fewer website visits, believe it or not. When your site receives traffic from AI search engines, those consumers will arrive having done their research already. They’ll have made comparisons, explored the best recommendations, researched prices, and completed much of the customer journey in conversation with an AI assistant. By the time they click a link, they’re that much closer to conversion, making your traffic more valuable than ever (and easier to convert.)

Consumers can already move from awareness to conversion without ever visiting a website

Perplexity Shopping uses strategic partnerships with big brands and services like Shopify to provide a comprehensive experience for consumers, from discovery to purchase. Today, there are already options for consumers to research, find, and purchase products without ever having to visit a website — it all happens in the AI search platform. Any brand that doesn’t appear there can’t compete.

Optimizing for AI search doesn’t require much extra effort

It may seem overwhelming, but in reality, optimizing for AI search doesn’t require significant resources or multiple strategies for each platform. The beauty of the expanding search landscape is that most of these technologies and platforms require the same things to find, rank, and serve websites to consumers:

1. The website needs to be findable by crawler bots (both traditional and AI-powered), and its important content must be indexed
2. The content should align with your customer journey and be the most satisfying answer for your consumer’s need
3. SEO fundamentals must be strong to earn the high organic rankings that inform both traditional and AI search links and mentions

Optimizing for visibility across AI platforms really comes down to two things: efficiency and scale. Your efforts to rank across Google and Bing can and will inform your presence on AI search platforms — and to take advantage of that opportunity, you must find ways to automate as many manual tasks as possible while scaling your success across the entire site. Let’s explore how.

How to optimize your website for the expanded AI search landscape

What it means to rank is evolving before our eyes. As mentioned in our AI Overviews study, “ranking” is becoming something more than a position in the ten blue links. Generative responses in the SERPs, like AI Overviews, can take over up to 76% of the results page. In some cases, organic Google rankings aren’t even visible above the fold. Instead, brands are appearing in the list of cited and linked websites alongside the AI Overview itself — a much more visible position, in many cases.

It’s a similar story with conversational AI platforms like ChatGPT and Perplexity. They can share URLs and recent content with consumers, but within a highly personalized, conversational response — the consumer has no responsibility to scroll through results and choose the best one. It’s chosen for them.

How can you be sure that your site is referenced with a link in AI platforms? How can you make sure your brand is mentioned at all?

Like everything in search, it starts with being found.

1. Make sure your site is indexed

Before you can ever appear in an answer on Perplexity, ChatGPT, Microsoft Copilot, or Google’s AI Overviews, your site needs to be found by the crawlers that catalog the internet.

Organic search indexes inform website links and mentions in AI platforms. Some reference Google results, while many use Bing’s open API access to reference website rankings and links. That means that everything you’re already doing to rank your top content in Google and Bing will see results in AI search.

Get proactive with your website indexing:

1. Push your content directly to Bing, skipping the need to be crawled and indexed and instead providing your content updates directly to Bing’s search index.
2. Use a protocol like IndexNow to alert a coalition of search engines that your content needs an updated crawl
3. Generate optimized sitemaps that signal to Google how it should prioritize crawling your site content

Additionally, make sure your site is accessible to the AI crawlers powering any platforms you want to appear in. ChatGPT and Meta AI are in the process of building their own search indexes to reduce dependence on Google and Bing; if your site content isn’t findable by their crawler bots, you’ll never be referenced in their upcoming indexes.

2. Develop a strategic bot governance plan

Right now, bot traffic comprises almost 50% of all website traffic. That’s only going to increase as AI search expands even more. Bot traffic isn’t all bad, but unfettered access to all could strain your site or create security concerns.

Managing access to your website will be integral to your AI search strategy. You’ll need to understand the different platforms and bots, what they do and why, whether it’s helpful or harmful to allow them to access your site and content, and make a plan accordingly via technical solutions like robots.txt.

3. Automate and scale your optimizations

Keeping up with AI tech demands efficiency. Brands that rely on manual optimization processes, friction with prioritization and implementation, and competing engineering resources will lag behind those who find resource-saving solutions.

Here are 4 things you can automate for success across both traditional and AI search:

1. Indexation: Botify automates indexation and optimized sitemap generation for large websites, helping your content get into top search engine indexes like Google’s and Bing’s (which informs AI engines like ChatGPT and Meta AI.) XML sitemap generation cues Google that your highest-priority content needs a recrawl automatically.
2. Page rendering: SpeedWorkers delivers pre-rendered site content to search engine bots and AI search bots automatically, reducing the amount of effort they need to crawl your pages, maximizing crawl budget, and getting more of your top pages found.
3. Optimizations at scale: PageWorkers automates and scales your on-page optimizations across thousands of pages. It can be deployed and rolled back instantly, allowing you to accomplish previously tedious tasks like title tag updates immediately, at scale, and automatically.
4. Testing & iterating: SplitTesting allows you to test optimizations against a control set of pages, allowing you data-rich insights into what works and letting you roll back or deploy tests immediately.

4. Use organic keyword data to understand and optimize your content for AI customer journeys

You have everything you need today to understand consumers using AI search; after all, they’ll ask the same questions — just on a different search interface. Here are a few ways to use your existing keyword data to predict their questions in AI search:

• Analyze question patterns to optimize your content. Identify question-based queries. Group similar questions to understand pain points and milestones. Align the language your site uses with your consumers’, and analyze long-tail queries to understand how conversations with AI evolve.
• Identify content opportunities hidden in queries with high search volume but low CTR (is your consumer getting a complete answer in the SERPs?) and those with seasonality that you can investigate to better answer your consumer’s needs.
• Satisfy the full customer journey with your website content using the insights you uncover; even if consumers are learning about your brand, solutions, and products via AI platforms, those conversations will be informed by the content you create and provide. You control the narrative across both traditional and AI search — use that power to your advantage!

5. Maintain and grow your SEO fundamentals to rank highly in traditional search

Because AI search engines rely on traditional search indexes to provide links and citations to content, it’s vital that you continue to focus on all the things that contribute to high rankings in organic search.

The goal of an AI search platform is to provide a fast, satisfying response, so their crawlers don’t have infinite time to find resources to cite via live retrieval. Realistically, they’ll only tap into the top-ranking sites they find on the indexes they reference, so you end up with double the reward when focusing on your main SEO strategy for Google and Bing.

6. Understand and monitor AI crawler bot behavior on your website

Here at Botify, we’re constantly adding functionality to help you understand the impact of AI search on your visibility:

• LogAnalyzer allows you to automate log file analysis and examine AI crawler bot activity on your site, helping you see your site through an AI search engine’s eyes. How much of your content can they access? What are they missing? Which bots are hitting your site, and how often?
• Our new AI Bots Reporting Template in CustomReports automates bot behavior analysis and reporting, providing clear visuals and data to inform your strategy and your teams on the impact of AI search for your brand.
• Using AI platform filters in RealKeywords allow you to see which keywords are driving traffic from top AI search engines like ChatGPT, Perplexity, and more.

Be an early adopter of decentralized search strategy

It’s been a long time since brands have had to look beyond Google to reach consumers organically. So much sudden change can cause anxiety in even the strongest brands. There are clear actions to take today to prepare for tomorrow, though, and having the right tools and strategies on your side can make all the difference.

At Botify, we’ve been innovating alongside changes in search from day one. We’ve got the solutions brands need to hit the ground running in the decentralized search race: analytics to show you how AI crawlers access your site, tools to push your top content to the most exciting generative AI platforms, and the automation to keep sites technically sound at scale so crawlers can index and render them with ease.

The expansion of the organic search landscape is a good thing. The rise of decentralized search is a breath of fresh air, an opportunity to shift your perspective on organic visibility, to reassess how you reach consumers and where, and to stand out from your competitors by becoming an early adopter of decentralized search strategies. As a bonus, by embracing a search strategy that goes beyond Google alone, you’ll be safeguarding your future organic traffic and revenue by optimizing for today’s search reality. 

If you want to learn more about the future of search and how Botify can help, reach out — our team would love to help.