Top 10 Takeaways from BotifyCONNECT June 2019

About once a quarter, the Botify team hosts an event where we get together for good food, good drinks, and a lively discussion about hot topics in our industry -- it's called BotifyCONNECT, and if you've never been, we hope you can join us for one in the future!

This time around, the topic was JavaScript SEO. Our very own Frank Vitovitch kicked us off with a presentation, followed by a panel discussion with our New York Search Data Strategists Chris Bryant, Kyle Blanchette, and Will Lau.

Here are the top 10 takeaways!

1. Log files shed light on Google's second phase of indexing

At Google I/O 2018, Google revealed that it defers rendering JavaScript until Googlebot has the available resources. Googlebot will crawl a page, fetch the server-side rendered content, and run initial indexing on that document. Later, once more resources are available, it will go back and render, leading to a "second wave of indexing."

This has led SEOs to wonder, exactly how long does it take Google to come back to our pages after the initial indexing to execute the JavaScript and re-index?

We simply don't know unless we look at our log files. Log file analysis for SEO can allow us to see how often Google is coming back to execute our scripts. Not only will this help you understand when you can expect Google to index your full page content, but it can also reveal where Google might be wasting time on scripts that aren't even critical to changing the page's content.

2. Cache certain scripts to save crawl budget

If you notice during your log file analysis that Google is spending time on scripts that aren't critical to the page content, try caching those resources so Google doesn't try to crawl them again.

Remember, log files are the only way to understand what Google is spending time on, so if you're dealing with an issue where Google isn't indexing all of your important content, try tactics like caching to direct Google's time away from unimportant scripts and toward the most important elements of your pages.

Read up on caching here.

3. Optimizing for render budget presents huge opportunity

Because Google doesn't have infinite resources to parse all the JavaScript on the internet, it uses budgets to determine how much it can and will execute on your site before moving on. We're calling this render budget, and focusing on it can present a huge competitive advantage.

Optimizing your code for search is table stakes, but when your competition isn't focusing on JavaScript optimization (and many aren't), it can lead to huge wins -- more of your important content found, indexed, and earning traffic and revenue for your business.

Optimizing for render budget is all about helping Google out. Focus on making your sites faster and as minimally JS-dependent as possible when it comes to your content and links.

4. Just because something works doesn't mean search engines can find it

We're still seeing many instances where, because something works for the user, people assume it works for search engines as well. This isn't the case.

For example:

• Don't hide links behind an action that requires an API call to make the links visible (those URLs won't get crawled)
• Make your infinite scroll search engine friendly
• Make sure Google can see lazy-loaded content

Developers have a lot of options to achieve a certain user experience, but not all of those options are equally accessible to search engines.

5. Build relationships for the most effective SEO

Because there's been such a heavy emphasis on JavaScript in the SEO community lately, many SEOs may feel discouraged that they don't know JavaScript or overwhelmed at the thought of learning a programming language. Truthfully, the skill you need most to be effective at SEO isn't JavaScript, but rather communication.

SEOs don't necessarily need to be able to program in JavaScript, but they do need to be able to understand when JavaScript might be causing issues and communicate those issues to engineers and product managers.

This is entirely dependent on your organization, but your product team probably isn't thinking (at least primarily) about how their work is affecting SEO. This can cause issues unless you unsilo, embed yourself in the organization, and make good relationships with your developers and other product people.

6. Don't be afraid to question third party tech

Third-party technologies that we use on our websites may use JavaScript to load things like related products or reviews. It's easy to trust that these technologies are doing everything the right way, but it's best to assume a stance of healthy skepticism.

Some tools block Googlebot from crawling their API because it's a lot of additional resources, but you need to make sure anything you want indexed is accessible to Googlebot. It's very possible that a developer blocked something because they assumed Google would never need to see it, but as SEOs, we need to do our due diligence and make sure that's actually true.

Some people also opt to pull from third parties using JavaScript because it's cheaper than coding everything by hand and takes up less of their own resources -- it was more of a self-serving choice than the objective "right" way to code something, resulting in Google having a harder time accessing all your content and putting more strain on the end-user to load all the resources.

7. Prerender is not a silver bullet solution

If Google has a hard time rendering JavaScript, why don't we all just prerender everything?

When we polled the audience, some people were using a prerender solution, some weren't. This underscores our stance on prerendering as well -- it may make sense on some sites, but not others.

Frank polling the audience about prerendering. First of all, prerender is expensive, so it likely won't be a viable solution for many companies that are operating on a tight budget.

Second, prerender makes the most sense for websites with pages that don't change often or are completely static. So if you're an online marketplace like eBay or a publisher like NPR, prerendering wouldn't really make sense. For example, we worked with an e-commerce business whose prerender solution was causing product prices to get out of sync -- Google was seeing one price while users were seeing another.

If you're using a prerender solution, make sure you audit it to see what a user is experiencing versus what Google is seeing.

8. Google sometimes caches expired resources

In response to the question, "Have you seen instances where Google caches JavaScript resources with an expired API key?" our team not only answered that they had seen this before, but we found out that one of them was actually working through this very issue with a customer currently!

The issue is that Google is caching a version of their API with a key that lasts only five minutes. Essentially, if Google doesn't come back within five minutes, the key changes and it no longer can be accessed. If you do cache a resource, Google will typically come back to it. When Google encounters an expired resource, they'll usually make an attempt to crawl and render it again.

9. Don't force your visitors to load everything

Hidden under all the technical jargon surrounding JavaScript is a truth that's easy to forget: the more JavaScript you use, the more strain you put on your visitor's device.

We'll keep this takeaway short and sweet -- don't force your visitors to load everything! They're going to get your content later than you want them to see it and waste a ton of their RAM in the process.

10. SEOs don't need to be developers, but there is room to grow when it comes to understanding how websites work

We've already said it, but it's worth repeating: SEOs don't need to be able to program in JavaScript. However, we need to be careful that we don't overcorrect either.

While you don't need to understand JavaScript at the level of a developer, there is incredible value in learning more about how the internet works. It's the difference between knowing how to code and knowing how Google is interacting with that code. When SEOs commit to deepening their understanding of the web, everyone benefits.

Want to join us for the next BotifyCONNECT? We'd love to have you! Sign up for updates so we can notify you when the next event is going to be and how you can register.