Botify Supports IndexNow’s Move Toward a “More Efficient Internet”

Indexing the internet is resource-intensive work. With more than half a million new websites being created globally every day, search engines are in constant pursuit of more efficient crawling and indexing methods. (That's how the sitemap was born almost 20 years ago.) With so much content to crawl, it can take days or even weeks for search engines to pick up on new, updated, or deleted pages, costing businesses untold sums in lost revenue.

Thankfully, these kinds of delays may soon be behind us. In October 2021, Microsoft Bing and Yandex announced IndexNow, an open protocol that allows websites to actively notify participating search engines of changes to their content. Instead of crawling entire websites to test for changes, search engines will know exactly which pages to prioritize in crawl.

"Traditional crawling no longer makes sense for the modern web."

Stan Chauvin, Botify Co-founder and Co-Chief Innovation Officer

Microsoft calls IndexNow "an initiative for a more efficient Internet." Not only does it promise a major reduction in energy costs, but it also lightens the workload for site owners and ensures the timely delivery of their content. As part of its ongoing commitment to this efficiency initiative, Botify's enterprise organic search platform now supports IndexNow integration.

Botify's ongoing partnership with Bing

Botify's partnership with Bing dates back to 2019, when Bing was piloting its Content Submission API. Unlike IndexNow, the Content Submission API takes Bingbot out of the indexing equation entirely. Instead of notifying search engines of pages to prioritize in crawl, it pushes those changes directly to the index.

Site owners were eager to take advantage of this new tech, but managing the API calls proved complex. That's why Botify streamlined the process for SEOs. Using SpeedWorkers, you can submit directly to Bing's index. Sitemap generation is available via Botify's FastIndex offering, keeping customer sitemaps meticulously updated for easy indexing by search engines.

But Microsoft is changing the game yet again by opening up the IndexNow protocol to all search engines. Google announced that it would begin testing the protocol. Should it choose to adopt IndexNow, it could have profound implications for SEO globally.

Making search more sustainable

IndexNow is ultimately about conserving resources as the Internet grows ever more massive and complex. Search engines already struggle with resource scarcity, which leaves them no choice but to impose limitations like crawl budget on site owners.

Botify Co-Chief Innovation Officer and Co-founder Stan Chauvin sees innovation toward search efficiency as inevitable.

"Traditional crawling no longer makes sense for the modern web," he says. "It's way too expensive. That's why all our products function as interfaces between websites and search bots: to streamline the transfer of information."

SpeedWorkers customers can notify all participating search engines of updates, deletions, and additions to customer sites, as well as leverage the existing Bing content submission API.

Several Botify partners - including Cloudflare, Akamai, Yext, and WordPress, among others - have already joined the IndexNow initiative, accelerating the transition toward a more efficient Internet.

To learn more about Botify, get in touch, or subscribe to our blog to receive monthly updates in your inbox!