The State of Modern SEO: Trends in the AI Era

Over a decade ago, Google shook up the search industry by launching the Hummingbird algorithm update, beginning the search engine’s shift toward natural language processing (NLP) in an effort to serve up content more relevant to consumers. 

Today, NLP remains the basis for AI-driven search. But search strategies are no longer only about ranking high on Google. A more diverse, conversational, and AI-augmented ecosystem is unfurling. Large language models that power answer engines, with information collected from indexes like Bing’s, are already reshaping how consumers find information, and brands must adapt to a rapidly approaching future where search goes far beyond traditional query-based systems. 

At Botify Connect 2024, we welcomed guest speaker Nikhil Lai, senior analyst at Forrester, to join Botify expert Frank Vitovitch on-stage to explore Traditional Vs GenAI Search: Everything You Need to Know. Their experiences discussing the state of search with top businesses reveal an industry undergoing a revolution, where changing priorities, technologies, and consumer behavior are redefining what it means to be found.

Historically, SEO has been a powerful, if often undervalued, tool in digital marketing strategies, consistently delivering a high ROI when done right. To continue growing in the modern search era, brands need to rethink their investments, develop holistic search strategies, and prepare for AI-augmented customer interactions that go beyond the click. 

The state of organic search today

What brands are seeing

Underinvestment in SEO vs. paid search

Despite the long-standing evidence of SEO’s high return on investment, many brands continue to underinvest in organic search compared to paid search. It's not unusual to see brands allocating $500K a year for SEO while dedicating $100M to pay-per-click. This means that SEO’s potential remains largely untapped. While paid search offers immediate visibility and quick results, it’s in fact organic search that delivers long-term, sustainable traffic.

Organic search has a marketing problem, which lies in the fact that it’s too often undervalued and under-prioritized because its results are slower, even when it yields better ROI over time. Many brands still focus on short-term gains through paid search, while neglecting the growth opportunities of organic search.

Changing consumer behavior

Innovations in search also means changes to how consumers interact with search engines. All signs point to a shift toward zero-click searches, where consumers can get direct answers without having to visit a webpage. Coupled with a growing use of ad-blockers and cookie clearing, search experiences are becoming increasingly frictionless. This trend represents both a challenge and an opportunity for brands looking to capture consumer attention in an era of instant, AI-driven results.

What brands are concerned about

Along with these new trends come new concerns, as brands consider how to adapt their organic search strategies and what to prioritize. Among the many relevant questions, several commonalities among brands that Botify works with have emerged: 

1. How to prioritize paid and organic: With AI-powered search engines often providing more immediate, conversational responses to queries, traditional methods of ranking and traffic measurement, both paid and organic, may no longer be as effective. Brands are asking themselves how to prioritize these two forms of search to drive results, and whether their existing approach to SEO and SEM needs to evolve.
2. Preparing for the rise of AI agents and the decline of traditional web-based experiences: As AI bots become more integrated into search, businesses are left wondering how (and if) to adjust their strategies to increase visibility. In addition, confusion is arising around how to track first-party vs. third-party data. Brands are grappling with how to ensure they can still measure and understand customer behavior accurately in a world where AI bots are increasingly driving traffic and to queries happening outside search engines. 
3. What metrics to use for AI-driven search: Traditional metrics for measuring the success of search efforts, such as clicks and conversions, may no longer fully capture the impact of AI-integrated search. Brands must consider new ways to assess the effectiveness of their search strategies in a world where AI is curating and serving content in increasingly personalized ways. This includes addressing vertical-specific challenges. For publishers, AI bots that generate direct answers to search queries can reduce clicks to their sites, which is a main revenue stream for many ad-based models. In addition, e-commerce businesses must consider how AI-driven search surfaces product pages that are regularly updated, and how it may increase the cannibalization of paid vs. organic traffic.
4. The structure and governance of data: Brands must now ensure that their data is structured so that humans and AI bots can both easily understand it. This raises not only the question of how to organize their data to ensure it's accessible across different platforms, but also whether or not they even want it discovered by AI bots. 
5. Addressing infrastructure concerns: Brands need to understand how AI bots interact with their sites, and how best to structure their sitemaps and other technical elements to ensure they’re crawled effectively. This may involve revisiting technical SEO strategies and priorities. 
6. Capitalizing on conversational search: The rise of AI-powered assistants and chatbots has led to an increasing demand for voice and conversational search optimization. These shifts demand that brands expand their SEO horizons beyond Google, and consider how their content can be optimized for new forms of interaction.
7. Maintaining information integrity: Finally, brands must consider the very real problem of maintaining information and brand integrity in a world where AI bots are curating and presenting responses based on a wide variety of sources. Ensuring that content remains accurate, reliable, and trustworthy will require careful oversight. 

The actual impact of AI on search (so far) 

Fewer clicks

One impact SEOs are already seeing from AI-augmented search is declining click value. With more and more search results providing immediate answers via AI-powered systems, consumers are engaging with search results in a way that reduces the need for clicks. This makes it harder for brands to capture traffic from traditional search results.

On the other hand, conversions are becoming more efficient. As AI allows for more personalized, relevant search experiences, consumers are finding what they need more quickly, leading to higher-quality traffic and conversions.

The rise of live retrieval

Large language models (LLMs) don’t need to rely solely on their training data any longer. While their training data can be stale, with retrieval augmented generation (RAG), LLMs have the ability to pull real-time data from websites to provide instant answers to consumer queries, without needing retraining. If showing up in answer engines is part of a brand’s strategy, they need to ensure that their content is structured in a way that allows AI models to retrieve it effectively.

E-commerce brands are especially vulnerable to negative impacts as a result of LLM live-retrieval. Inaccessible or outdated product data can lead to missed opportunities for visibility in AI-driven search results.

New technical SEO challenges

SEOs also face the challenge of optimizing for multiple AI bots and ensuring their websites are ready for these advanced crawlers. Site structure and other technical elements still play an important role in feeding information to both traditional search and AI bots, so maintaining a clean and accessible site structure remains a priority. 

How brands should take action

To stay ahead of the curve and adjust to the ever-evolving search landscape, brands must begin to adjust several tried-and-true optimization strategies. 

Understand how bots are interacting with your site

Log file analysis remains a strong strategy for understanding how both traditional and AI bots are interacting with your site. Leverage log data to analyze that traffic, identify potential sticking points, and start to determine the potential rewards for allowing AI bots to crawl your website.

Develop governance strategies for AI bots

Once brands have data on their AI bot traffic, they should evaluate the benefits and risks of AI bot traffic to determine which, if any, to allow. Ask: how is priority content being used in AI-driven responses, and does it help or harm the brand’s visibility? Ensure governance plans and strategies are in place for both bots that crawl to index training data and for live retrieval. 

Holistic search marketing: Aligning paid and organic search

The days of treating paid and organic search as entirely separate strategies are over. Brands need to move beyond the dichotomy of paid vs. organic search. Instead, they should focus on creating synergy between paid ads and organic content, aligning both around a unified question-and-answer approach. This will help avoid wasteful bidding on keywords that could be captured organically, and ensure paid search supports SEO initiatives.

Prepare for decentralized search

The future of search is decentralized, with Bing integrating AI and giants like Amazon improving conversational search elements. People will continue searching and shopping online, but those activities will be more widely distributed. Brands need to ensure they’re indexed across multiple platforms — not just Google — by optimizing for Bing, exploring new tools like IndexNow, and pushing their content to various other platforms depending on their consumer base. 

Listen to the voice of the customer

Understanding the question behind the question — what consumers really want — should be at the core of any SEO strategy, and that won’t change in modern search. AI’s conversational capabilities, however, make understanding consumer intent even more important. Search queries are evolving into more complex, nuanced interactions. Understand where and how your consumers are searching, and diversify strategies accordingly. 

Optimize for natural language queries

There’s an often-shared statistic that 15% of the search queries every day on Google are new, and many are complex, or in the form of questions. In the age of AI search, that percentage will only increase. Content optimized for natural language patterns will be the key to fulfilling search intent, and should remain a priority for brands.

The state of modern SEO is adaptability

Search has always been an industry subject to changing rules and strategies, but recent advancements in AI have catalyzed major shifts. Brands need to act now to invest in AI-aware SEO strategies, continue listening to their customers on all the platforms they use, and create synergies between paid and organic search efforts. 

In the age of AI, search marketing is no longer just about ranking on Google — it's about understanding the questions consumers are asking and providing the most relevant, accessible answers across a wide variety of platforms. The future of search lies in adaptability, and those who embrace these changes now will be best positioned to lead in the evolving search ecosystem.