International Search Engines: SEO For Baidu, Yandex, & Naver

Google. It's the most widely used search engine, and for many, it's the first thing people type into their web browsers every day. However, it's not the only search engine people use. According to Alexa, Baidu is the fourth most popular website (with Google ranking #1). In fact, China has the third largest internet user population (731 million back in 2016), so it's no surprise that Baidu ranks so high.

It's not uncommon for websites (extensive enterprise websites) to have an international audience base, so it's critical for anyone thinking about SEO to be aware of all the major search engines.

With varying features and capabilities, each search engine crawler is configured to work at an entirely different pace and aptitude, oftentimes with a completely different end target (mobile-friendliness, website authority, and so on).

So, when you're optimizing for an audience in Russia, for example, you've got to think like the Russian search engines do (in this case, Yandex!).

A big-picture look at search engines across the globe

Google: the search engine with majority market share

Larry Page and Sergey Brin founded Google in 1998, and it is by far the most widely-used search engine with about 90% market share. Google uses the largest number of data points to index and rank pages on the web, making its algorithms some of the most complexes of any search engine. Some of the major factors taken into account include the technical health of your website, external authority (backlinks), mobile friendliness, and, most importantly, the quality of the content. Google's crawler is very advanced as well – able to render JavaScript and crawl billions of pages every day.

The graph below shows the URLs crawled by Google on a particular website. To illustrate the differences between each search engine and its capabilities, all of the graphs represent the same website over the same period of time, with a day over day view.

In the example, Google crawls close to 1M or 2M per day, 100K-500K of new URLs. Its crawl volume exceeds the number of crawled URLs, meaning that your pages can be crawled multiple times in a day (Woohoo!). You can check how often Google is crawling your pages in Google Search Console or directly in the Botify platform.

How do you ensure Google crawls and indexes your new pages?

There are a few ways that you can go about increasing your crawl rate to ensure Google crawls and indexes your new pages. For example, you can improve your site's load time (and, as a result, enhance your UX -- a win, win!) and submit an updated sitemap making sure to list your most important, new URLs first.

After you've taken steps to direct Google to crawl your new URLs, you can use a tool like the Botify Log Analyzer to see if that's actually happening (e.g., Is Google really crawling my URLs? How often?).

Additionally, Botify's new-and-improved JavaScript Crawler will allow you to see exactly how search engines render JavaScript on your website. They may be crawling your new URLs, but are they seeing all of your content?

These aspects and more will help you identify if and when your new pages will be crawled and help steer bots in the right direction.

Bing: Microsoft's search engine

Microsoft launched Bing in 2009, and as of 2017, claimed it enjoyed about 9% global market share and 33% in the U.S.

While similar to Google, Bing has key differences in the way it crawls, indexes, and ranks web pages.

Key differences between Google and Bing

• Ranking: Bing's ranking algorithms use some of the same factors as Google, like backlinks and mobile-friendliness, but weights them differently than Google -- backlinks appear to be much more important for ranking in Bing than Google, while mobile-friendliness is more important in Google than Bing.
• JavaScript: Bing says it renders JavaScript, but in our experience, we've seen few examples of that. Fortunately, Microsoft recently announced that Bingbot is going evergreen. Bing is adopting Microsoft Edge to run JavaScript and render pages, and Microsoft Edge will soon be adopting Chromium (the open-source project that Google Chrome is based on). This change will allow Bingbot to access Chromium's features and continue to tack on the new features available with each update. In addition to being a win for SEOs working with JavaScript, this will also make it easier for websites to optimize for Bing and Google together without needing to troubleshoot for one over the other. Check out our article about Googlebot updating its user agent string to learn more about what an evergreen bot can change for SEO.
• Crawl Prioritization: Bingbot prioritizes URLs by looking at internal linking and sitemaps, while Googlebot uses a multitude of other factors. Additionally, Bingbot sometimes falls into "crawl traps" like faceted navigation and internal search URLs.
• Crawl Volume: As depicted below, Bing's crawl volume nearly matches the amount of crawled URLs (meaning that, instead of crawling URLs multiple times, it crawls them just once, or much less frequently). Meanwhile, the number of new URLs crawled remains very low, making it difficult to get new, important pages indexed fast.

But we've got big news!

At our recent Crawl2Convert conference in New York, we were thrilled to have Fabrice Canel, Bing's Principal Program Manager, present the new Bing API that will help you get your newest pages indexed instantaneously. Even better, the API will be directly integrated into the Botify platform! This integration will launch in correspondence with FastIndex, a component of Botify Activation that will allow Botify customers to create fully optimized sitemaps. You'll be able to increase your daily submission limits and get your pages indexed fast – through FastIndex and the Bing API.

Baidu: China's main search engine

Baidu is the dominant search engine used in China. It is similar in size and scale to Google in the U.S. and resembles Google in its investments. For example, just like Google, Baidu has invested in machine learning to improve its ranking algorithms.

In contrast, though, Baidu has a huge capacity for crawling. Rather than relying on signals like canonicals to prioritize crawling and indexing, Baidu typically can crawl everything and decide on its own what should be indexed.

Additionally, China heavily prioritizes mobile, so Baidu mobile search results can vary drastically from Baidu desktop search results. Mobile is such a priority in China that Baidu even creates mobile versions of desktop-only websites on the fly.

Baidu also has an initiative similar to Google's AMP. Baidu's AMP-like initiative began simply as MIP (Mobile Instant Pages), followed by the introduction of a new, more utilitarian platform called Bear Paw (named in honor of the Baidu logo). In 2018, Baidu announced it would send 50% of its mobile traffic to BP pages.

Baidu's crawl volume is nearly equivalent to the amount of crawled URLs, so the odds of having your pages crawled more than once a week are slim. However, it crawls new URLs at a high rate, which is great for getting your new URLs indexed fast – rather than having to wait days or weeks for indexing. Additionally, Baidu's crawler has the ability to render JavaScript, albeit selectively.

P.S. Although Google China does exist, it was blocked from mainland China in 2010 and is only available in Hong Kong. So, if your website has a large visitor base from China, they're likely searching and finding you via Baidu!

Yandex: Russia's main search engine

Yandex is the search engine of choice in Russia, and it's also the country's largest technology company. Of the search engines in Asia, Yandex is the most comparable to Bing. A lot of the traditional SEO tactics, like building backlinks and article spinning, still work well, but Yandex doesn't get near the scale of other crawlers. The crawler focuses on new and extremely important URLs, rather than frequently crawling an entire website. To note, the crawler does not crawl and render JavaScript, making it very difficult for JS-based websites to rank in Yandex.

Like Baidu, Yandex generally does not crawl URLs more than once in a given timeframe. However, it crawls a fair volume of new URLs (at the rate of nearly 50K per day!), which is optimistic for websites with new web pages that require indexation.

Naver: South Korea's main search engine

Naver is the most popular search engine in South Korea. This search engine behaves more like an early 2000s Yahoo! – acting as a "front page of the internet." While they do index content from websites, a lot of their content and pages are curated search pages rather than standard search results. Naver connects users based on shared interests and uses this as a means of funnelling relevant information to them. Naver also launched Knowledge-IN as a public Q&A portal for users to get their questions answered and for Naver to collect data in order to prioritize topics according to popularity.

In some ways, Naver is more publisher than search engine. Ranking depends heavily on their determination of how authoritative a website is, using both human feedback and a ranking algorithm. Social media presence also plays a role in the algorithm.

Naver crawls websites infrequently, at the rate of just 2K to 6K per day in the example below, with between 500-2K of those pages being new URLs. Additionally, Naver does not render JavaScript.

Whew, that was a lot of information! Can you summarize how each search engine handles JavaScript, canonicals, and more?

We sure can!

So, while you might use a single search engine (likely Google), remember that there are millions of others around the world who use different engines to search the web. Different search engines mean different experiences and different things to keep in mind for SEO. For example, Naver search results are fueled by communities and interests, Baidu prioritizes optimized mobile experiences.

If you're running an enterprise website with a global audience, identify those audiences and the search engines they're using so that you can better optimize your website to reach that audience. For example, are you trying to reach a Russian audience with a JavaScript-heavy website? Or a Chinese audience without a strong mobile web experience?

Knowing what search engines your audience uses is also important for staying up-to-date on the latest news from that search engine. When the next BERT comes along, for example, you'll want to know about it so you can keep pace and avoid falling behind.

While SEO may not be the same across the globe, it's equally important everywhere. So, put your website to the test and make sure you do all that you can to reach the people looking for you, no matter where they are!