The Future of SEO Is Here: AI Overview, GPT-4o, Plus GPTBot Data Now in Botify

AI-powered answers are the future of search.

It's been a big week for SEO news. First OpenAI announced GPT-4o, a new version of their generative AI model with improvements across text, vision, and voice. Then Google followed up with a string of announcements about search at Google I/O, rolling out AI Overview (previously known as Search Generative Experience [SGE]) to all US searchers this week.

Traditional search exists not parallel to, but integrated with AI technology. It's expanding beyond the bounds of the browser: circle to search, searching via video and image, and combinations of visual and audio requests will change how consumers look for the information that leads them to your website.

Perplexity, Microsoft Copilot, and others have been innovating in the AI-powered answer space as well. There's a real opportunity for new players to stand out by providing unique experiences that satisfy consumer intent.

SEO professionals are the right experts to meet this moment. Ours is an industry of testing and learning, of finding what works and relentlessly chasing success. These changes create huge potential for brands: new places for your site to appear, new ways to reach your consumers, and new competitive advantages to uncover.

It's not a new search engine, but it's still a disruption for search. A space that was once dominated by a single major company is opening up to competitors. It's a burgeoning industry with the potential to reach consumers in completely new ways. And most excitingly, it's the perfect opportunity for SEOs to do what they do best: study, experiment, learn, and optimize.

What does this mean for SEO?

The way queries are asked and answered is changing before our eyes. How does this affect the search landscape as we know it?

The top innovators in search today are hyper-focused on connecting people with the information they seek in user-friendly ways. The leadership at OpenAI has embraced the idea of finding a "much better way to help people find and act on and synthesize information," emphasizing that getting the information you need in the most appropriate format is better than a ranked list of links. It echoes Google's original mission, which they reiterated at Google I/O: "To organize the world's information and make it universally accessible and useful."

No matter where you show up, satisfying the person behind the search is your north star. Search and answer engines are prioritizing:

• Natural, conversational queries
• Creative, human-like ways of addressing questions and problem-solving
• User experience
• Speed of responses
• Intent optimization

The implications for how we do SEO will be a mix of the familiar and the unfamiliar. Strong technical foundations will become even more important, while the nuances of ranking -- from keywords to measurability and beyond -- will change alongside shifts in search behavior.

The future of organic search is intertwined with AI-powered answers. Understanding both will be the secret to staying ahead of changes to come.

How search and AI interact

SEOs know how Google works. Even with the technology behind AI Overview layered on top, the methods Google uses to find, understand, and share information remain the same. And though there's less information on how new AI search technologies work, the facts we have are strong:

• All AI models need content to train on. ChatGPT uses a model trained on "publicly available information that is freely and openly available on the Internet," in addition to information licensed from third parties and user-generated information.
• Search engine results can inform answer engine responses. ChatGPT Browse searches Bing for certain queries to generate answers that require recent information.
• Crawling the web matters. GPTBot is one of the most active search bots online according to Cloudflare. It's currently more active than crawlers like YandexBot, Applebot, the Google Images bot, and more.

Both AI-powered answer engines and search engines rely on crawlers to find the content that powers their responses, whether they use proprietary bots or tap into an existing search engine index.

Organic conversion begins with crawling

The idea of the SEO funnel has been foundational to Botify, guiding our perspective on search to this day. It consists of a five-step process: crawl, render, index, rank, and convert.

Ranking and conversion are the steps many brands prioritize with regards to SEO, and there's an entire world of strategy and tactics therein. Plus, generative answers are formulated using results and ranking signals from traditional search engines. Ranking is an essential step in the SEO funnel for both search and answer engines.

But without a strong technical foundation at the crawling, rendering, and indexing stages, everything you do to rank and convert is moot. Your site needs to be found and understood by crawlers before anything else.

It is useless to try to optimize pages that are not crawled by Google, as they will not be indexed or active.

Adrien Menard, Botify Co-Founder & CEO, 12/12/2012

GenAI models need to find and crawl content, too

Generative AI models are limited by what they know. The data they've trained on informs everything they reference and all the answers they can provide.

That's why until recently, ChatGPT's responses were trapped in time, limited by data sets that ended in September 2021. Integrating Bing's real-time Internet browsing has allowed the AI engine to tap into Bing's robust, fresh website index for answers to questions that require recency, although it's restricted to paid ChatGPT users.

This is interesting to SEOs for several reasons:

1. It highlights the relationship between traditional search and AI-powered search

Answer engines need a website index to reference, meaning that the work you do to rank within Google and Bing can also influence how your content is served via AI engines. While it remains to be seen how answer engines cite their sources, appearing in generative answers can benefit brand recognition and for queries with specific intent, even traffic to your pages.

2. It shows the importance of robust technical SEO

The optimizations you're already making for your site's technical health make crawling easier for any bot. We know that AI-powered engines provide responses sourced by the same kinds of web crawlers that power traditional search. The difference isn't in how the information is discovered and parsed, but in how it's used.

3. It affirms that your SEO work informs generative answers

Generative responses use traditional ranking signals to summarize top content and formulate answers. Often, these answers summarize the highly useful content hiding behind the "boring" ten blue links. Your efforts to rank in traditional search have a ripple effect that influences how and where your content is used across AI-powered engines.

The impact of AI-powered SEO: What may change

Complex as it seems, we anticipate a gradual evolution rather than a 180-degree change. To provide the most satisfying answer to a query, you'll need to know your consumer inside and out: what they want and need, where they search, how they like to receive information, and what their customer journey looks like.

Visibility on non-Google engines becomes more important

Most SEOs are used to their work being all Google, all the time. That dynamic is changing. Fewer monopolies and more competition will result in a better search experience for consumers. There is the issue of data, though: what can you use to guide your understanding of these new technologies? There's currently no Google Search Console analog for OpenAI. Even GSC plans to combine AI and traditional search data, making it difficult to uncover insights.

NEW: GPTBot now available in Botify Analytics

Log file analysis will be key to understanding how web crawlers access your site, including AI crawlers like OpenAI's GPTBot. To keep our customers ahead of evolving search trends and innovations, we've integrated tracking for OpenAI's GPTBot via LogAnalyzer within Botify Analytics. With this new integration, Botify customers can now gain clarity on whether their most valuable site content is being accessed by ChatGPT.

Submit your site directly to Bing with Botify Activation

For Botify customers using SpeedWorkers within our Botify Activation suite, our partnership with Bing means that you can submit your site directly to Bing's index, delivering fully rendered HTML 24/7 and ensuring your freshest and most important content is found. As ChatGPT continues to use the Bing index for fresh answers, it becomes even more important to optimize beyond Google alone.

ChatGPT has over 100 million users, while Perplexity has 10 million active monthly users and Bing's market share has been growing steadily. And with almost six million searches performed every minute, optimizing your site for Google search traffic remains vital. It's undeniable that consumers are looking for different ways to get the answers they need, and your future search strategy should take that into account.

Look for the opportunity within the noise

A decentralized search landscape may sound daunting. Keep in mind that many of your optimization efforts will still be effective across different search and answer engines, especially if your technical SEO foundations are sound. Technology is advancing rapidly, but factors like search behavior will be slower to change.

Keyword research and optimization shifts to intent

In a world where everything is more conversational, specific keywords will become less relevant. Consumers will use longer, more natural language to search. And in some cases, from what we've seen via AI Overview and GPT-4o, there may be no language involved at all.

Optimizing for keyword phrases will fade in favor of optimizing for intent (and in the case of GPT-4o, even emotion), especially as non-text-based search increases. Rather than targeting hyper-specific terms, looking at groups of keyword phrases by intent and working to understand what your consumer truly needs will be paramount. This makes the volume and quality of your data more important than ever: Botify Analytics tracks an average of 400,000 keywords per website, with up to tens of millions for our largest customers.

Traditional measures of success are replaced

With so much in flux, there's no way traditional measures of success can stay the same. Not even our most well-loved SEO metrics are immune. And that's a good thing -- if searcher behavior is expected to change, the ways we measure our reactions to it should evolve as well.

Keyword rankings

Measuring the ranking position of keywords will shift as we begin to optimize for intent instead of specific phrases. Looking at how topically segmented groups of pages are performing will be a strong alternative.

Organic traffic

Gartner's prediction that search engines will lose 25% of their volume to AI competitors by 2026 has been made and also challenged. Either way, reporting on organic traffic may have less of an impact when more consumers get comprehensive answers directly within a search interface.

One of the drawbacks of ChatGPT has been its inability to cite sources and verify generative responses; although it can now provide some reference links for paid users, a rise in the prominence and popularity of AI-generated responses could mean that fewer websites are linked in responses, resulting in less organic traffic overall.

This may sound scary, but perspective is everything. If more top-of-funnel intent is being satisfied directly in search, the traffic your site does receive may be more qualified, resulting in a better conversion rate. Understanding how to measure SEO's ROI will be critical.

The impact of AI-powered SEO: What will stay the same

It's easier to define what won't change in the age of AI-powered search. Despite all the unknowns, there are also foundational aspects of the industry that will remain the same.

The big players will still matter

Google and Bing aren't going anywhere fast. They're backed by some of the most powerful companies in the world with a huge influence over the US economy, making up 11.21% of the S&P 500. They have a huge amount of infrastructure invested in search and many partnerships that rely on their data to function.

Building and maintaining a proprietary website index is no simple task. As of now, there isn't a good alternative to initially finding and discovering a website without having first crawled and rendered it -- even for AI-powered answer engines, which still reference Google and Bing indexes to inform their responses. We can rely on the fact that Google Analytics, Google Search Console, and Bing Webmaster Tools will continue to provide value even as third-party data from answer engines could be hard to get. Botify customers can access and cross-reference GA and GSC data in Botify via our unified data platform.

Websites still need to be crawled, rendered, indexed, and provided as an answer

The way websites are discovered and understood by site crawlers isn't going to change anytime soon. Sites are built with code, and that code needs to be translated by a machine into something useful for humans.

No matter where your site is being referenced as an answer, it first needs to be found and understood. Even if "ranking" becomes less like ten blue links and more like which website is referenced first in a carousel at the end of a generative response, every search and answer engine needs your site content to fulfill consumer needs.

Content still needs to be valuable and satisfying to your consumer

More than ever, you need to deeply understand your consumer's intent and speak to it with your content. Content strategies may need to be adjusted, but quality will be table stakes alongside factors like E-E-A-T.

How to stay ahead of advances in search

SEO professionals are both resilient and optimistic. They know that with challenges in search also come opportunities: the chance to test, learn, and forge a new way of doing things. SEOs are also community-minded; once they know a tactic works, they shout it from the rooftops. There are few industries with such a focus on education and mutual success.

At Botify, we've always envisioned a world where every brand has the power to be found. Even as AI changes search in rapid and unexpected ways, we're constantly thinking about how SEOs and brands can come out on top.

Be found

Before you can worry about how well you perform on AI-powered engines, you need to first know if they're finding your site. Take another look at the SEO funnel: there's no path to conversion if you aren't found and crawled.

Keeping your technical site health strong via technical SEO fundamentals like de-crufting, optimized internal linking, structured data, fast rendering, and log file analysis will hone your competitive edge as search evolves.

Use log file analysis to understand the crawlers visiting your site

As new crawlers enter the scene without insight from supporting data tools, it's hard to know whether you're being found. That's where the power of log file analysis comes in.

Your log files store valuable information about how your site is accessed. While you can review them manually, Botify's LogAnalyzer automates much of the process and pulls the information into a unified interface.

This opens up a world of possibility when it comes to making data-driven decisions for an emerging search landscape. With GPTBot data in Botify, you'll get insights into crawl activity and organic visit data such as:

• Whether AI-powered engines are crawling your site
• How deep their crawl goes
• Whether they're finding your freshest, most important pages
• How often they crawl your site

You'll understand which content is seen as quality, satisfying, and useful from an answer engine's perspective, vital data you can use to shape your content strategy going forward.

Serve your site to crawlers faster

Crawlers have limited resources to explore a website. The faster they can move through your site and understand the content, the more of it they'll find and be able to share with consumers. Botify's SpeedWorkers solution serves fully rendered pages to crawlers to maximize their crawl resources on finding and understanding more of your content.

Activate your curiosity

Lean into your naturally curious SEO mindset. The people who become search professionals have one thing in common: they are relentless problem solvers, obsessed with piecing together puzzles and earning success. AI-powered search and answer engines are a different version of the same kind of change you've always met in the past, from algorithm updates and mobile-first indexing to Core Web Vitals, featured snippets, and more. People will always seek answers, and any intermediary service will always need websites to supply the content for those answers. Your job is to piece together the puzzle that gets the best answers to consumers, satisfying both humans and site crawlers along the way.

For over a decade, we've been focused on getting websites found. That doesn't stop with AI-powered search -- it accelerates. In a world where the news comes fast and the data is scarce, the key is to hold fast to what you know and leverage the information you have. Understanding how you're found in the new world of search isn't impossible, especially with insights into the top AI crawlers guiding your decisions.

If you want to learn more, we'd love to show you -- reach out to see how we can help!