Top 10 Takeaways From Crawl2Convert: Botify's Inaugural Customer Summit

On Monday, October 21st, we hosted our inaugural conference, Crawl2Convert, which brought together SEOs from near and far for a day of looking into the past, present, and future of SEO.For those who missed it, here are our top ten takeaways.

1. SEO needs a 100% investment

Botify CEO & Co-Founder, Adrien Menard opened up the day with a deep dive into how SEO has changed over the years and why it's becoming increasingly important for large, enterprise companies to invest in SEO.Did you know that 77% of the pages on the average enterprise website are inactive, as in they're not getting any organic search traffic? Major search engines have limited capacity to crawl websites, so it's critical to identify what's important on your website (a.k.a. what needs to be crawled) and what should not be crawled or indexed.Not only should you think about SEO on a technical level but also at a higher level within your organization's team structure. In the past, SEO may have seemed like more of a siloed effort fit for an individual tech expert, but it's actually vital for people to think critically about SEO across departments - from UX and IT to marketing.In a study conducted by Forrester Consulting, commissioned by Botify, researchers found that out of 250 surveyed executives only 14% were considered to be "Optimizers" (in comparison to Learners or Maintainers), meaning that they had the most mature SEO organizations (50% of Optimizers organized their SEO teams across marketing and product), were investing more in SEO, and had a company-wide SEO strategy. Of that 14%, 75% believed SEO to be the most effective revenue generator strategy, 74% planned to increase their spending on SEO tools in 2020, and 54% planned to increase their spending on SEO services in 2020.So, if executives who value SEO at a high level are seeing positive results, this poses a big question - one many SEOs are thinking about every day. How do you get the message across to stakeholders in your company that they should invest more budget and resources to SEO? Develop a roadmap with clear objectives, be supportive and understanding to all parties involved, and pull your weight while asking for support from colleagues - and make the plan come to life!If you're interested in learning more about the study, "Realize Greater Revenue by Prioritizing SEO," register for our upcoming webinar.

2. Evaluate your SEO efforts retroactively and proactively

With massive enterprise websites, one of the biggest issues is navigating billions of old pages while paying attention to new pages. But if you're only focusing on the present, you're doing yourself a disservice. There may be easily fixable errors that are getting lost in the mix that could open up huge opportunities for your business.Ryan Ricketts, Technical SEO Manager at REI, shared an impactful story about how his team cut their website down from 34 million URLs to 300,000, before slowly reintroducing significant URLs back into the mix. This deep dive did wonders for his crawl budget, allowing search engines to find the company's most important webpages and ignore the others. Bots tend to waste crawl budget on URLs that are unnecessary, duplicative pages, faceted navigation pages, and more, so it's best to provide clear directions on what should and should not be crawled.For retail and publishing companies, there can certainly be a lot to dig through in order to improve crawl budget. Here are some ways to do that:

• Making sure to expire "Product Unavailable" pages
• Setting up 301 redirects and removing redirect chains
• Cutting down on URLs crawled from faceted navigation (as Ryan's team did above)
• Updating your XML sitemaps and including new, important URLs at the top
• Building web performance smoke tests into the deployment pipeline to anticipate errors before they affect your website
• Checking your site server logs regularly

Botify Log Analyzer allows you to see which pages search engines are crawling, and how often. You can regularly inspect server logs to find blind spots, crawl spikes, and any unusual behavior so you can look critically at what's affecting your SEO - past and present - and develop comprehensive solutions.An additional asset when evaluating your SEO comprehensively is using time over time comparisons. Kenyon Manu, SEO Product Track Lead at Carvana, shared an interesting case study. In 2018, Kenyon noticed an increase of 335% in Carvana's month over month organic keyword count growth from October-December. Sounds like a win, right? But when he looked at the months of December through April of 2019, there was a 24% decrease. So, what went wrong?Because Botify keeps track of changes over time, Carvana was able to pinpoint when the dip occurred so that Kenyon could ask his development team if any changes were deployed at that time. Sure enough, they were. He learned that when pages didn't load fast enough, SSR (server-side rendering) Stability would be turned off. The Carvana team was able to develop a long-term SEO program that focused on performance and server-side rendering. Once the issue was fixed, there was an 82% increase in MoM organic keywords!

3. We've said it once, we'll say it again. Crawl Budget!

We can't say it enough. It's essential to prioritize tasks that improve your crawl budget. At C2C, we hosted a panel discussion with representatives from Conde Nast, REI, and G2 to talk specifically about optimizing your crawl budget, and we also held a Q&A with our Senior Data Strategists at Botify to share their thoughts on the topic. If you're unfamiliar, you can read up on crawl budget in our post on "Google Crawl Budget Optimization."Your crawl budget can be negatively impacted by faceted navigation and session identifiers, on-site duplicate content, soft error pages, hacked pages, infinite space and proxies, bad or spam content, and the list goes on. Any URL that a bot crawls is counted across your crawl budget. Making a site faster improves the user experience and makes crawl budget better. Load times in particular are impactful when encouraging Google and other search engines to crawl more pages. Bots are also interested in fresh content and changing internal links on your site.Here are a few ways that you can improve your crawl budget:

• Clean up your links! Avoid linking to non-200 HTML code pages, avoid URL parameters and ever-changing parameters (like timestamps), etc.
• Block infinite spaces
• Produce quality content
• Submit XML Sitemaps with your newest, important URLs on top
• Make sure to keep your important pages close to the home page. More on this in our article, "Is page depth a ranking factor?"
• Improve page speed
• Maintain a clean, easy to navigate site architecture

So, how do you actually go about prioritizing tasks that improve crawl budget? Just this week, we launched ActionBoard as part of our Botify Intelligence suite. ActionBoard analyzes hundreds of key metrics from Botify Analytics and removes the complexity and time constraints of determining what tasks will drive the biggest impact and why. In summary, Botify organizes the tasks for you, making them easy to manage and check off your list. You can even group similar tasks together in the same sitting. We love efficiency, because it means giving time back to your day!

4. Pay attention to user behavior and intent

User experience and intent go hand in hand. For example, if a user is looking to buy boots, and they're brought to a page with long-form content rather than product pages, they might become frustrated and leave. At the source, you've got to understand the value of content and the value of product pages. There's a time and place for both types of pages, but it depends heavily on the user's intent, established by the keywords they're targeting and how that aligns with what's presented on the page.Additionally, you don't want non-indexable URLs ending up in the SERP. For example, if pages with a non-200 status code, bad content type, or pages that canonical to another URL are reaching end-users, you're providing a bad user experience and impacting conversions. You can easily track all of these instances by tying your conversion metrics to Botify Analytics.A lot of retail businesses in particular are rethinking the way search experience is filtered for their website. Instead of showing best-sellers first, they might show new products based on user behavior and what's most interesting to customers. Googlebot is good at paying attention to these changes on your website and how it affects the end users, which contributes to crawl efficiency.

5. Mobile-First Indexing isn't black and white

Although mobile-first indexing was first announced in 2016, much of the web is living in a desktop-first world. Frank Vitovitch, Vice President of Solutions at Botify, gave C2C attendees a comprehensive look at the factors involved in a website's successful (and in some cases, not so successful) transition over to the MFI. According to our data, about a third of enterprise websites still haven't been moved over to the MFI. If that's the case for you, what can you do to prepare? And what does the impact look like for sites that have been moved over?We studied websites of various configurations (responsive, dynamic, and separate), various sizes (some with over 10 million URLs and some with less than 10k URLs), and across industries to see what factors affected their websites' transitions. We found that, generally, smaller sites are more likely to enter the MFI first, whereas larger websites (>10 million pages) have much more at risk.We looked at real instances where Botify customers were switched over to the MFI and found that their success relied on a combination of factors. Take a look:A positive transition to the MFIFor one publisher, their temporary crawl volume spiked when their website entered the MFI. Thirty days after the transition, 3% more URLs were crawled by Google, their active URLs increased by 6.2%, and they received a bump of 18.1% in organic search traffic. Piece of cake!A negative transition to the MFIFor another publisher, their temporary crawl volume also increased when their website entered the MFI, but their overall crawl rate dropped after the transition. They experienced a 28% decrease in URL crawls, a 19.5% decrease in active URLs, and a 21% decrease in traffic. The cause? 87% of their URLs had content similarity below 60% from mobile to desktop (with, on average, 170 words less on mobile site pages compared to desktop pages).How should you prepare for the MFI (or adjust after the fact)?Audit your site! Crawl the desktop and mobile version of your site and compare your site's architecture, content, and performance to ensure parity (similarity between both versions). Otherwise, you could suffer organic traffic losses after moving over to the MFI. You can use Botify to do all of this and more in one, unified place.

6. JavaScript is our misunderstood friend

You may have heard that Googlebot (and now Bingbot!), has gone evergreen. This means leaps and bounds for rendering JavaScript, but that doesn't mean there won't still be hiccups. It's extremely important to server-side render as much as possible to test and see what works, improve your crawl budget, and provide a positive user experience.Keep in mind that JavaScript is always changing, so Google will always be playing catch-up. Don't chance it. In fact, you can use our new-and-improved JavaScript Crawler to render up to 100 URLs per second (or conduct a 1-million page analysis in just three hours!). The faster you identify issues with JavaScript on your website, the faster you can make fixes to improve your crawl budget, your user experience, and your conversions.

7. Search engines need your help!

We rely heavily on Googlebot, Bingbot, and others to find our important pages, and we can't stress it enough - they need your help! Submitting your XML sitemaps and providing clear directions on what bots should and shouldn't crawl and index can contribute to the frequency and speed at which your pages show up in SERPs. But despite all of the work you do, there's still a waiting period. Wouldn't it be nice if your pages showed up in SERPs instantly upon publishing?Well, we've got good news! At C2C, we were thrilled to have Bing's Principal Program Manager, Fabrice Canel, talk to attendees about our new partnership, which will present all Botify customers with a fully integrated Bing API for submitting your new, important pages for instant indexing.This integration comes along with the launch of our newest product debuting in early 2020, FastIndex by Botify Activation. Through FastIndex, Botify will be able to use the full 1000+ metrics of data retrieved from your log files, combined with the business rules you provide, to create fully optimized sitemaps which can be submitted through our integrated Bing API. Not only will the capabilities be fully integrated, but they'll increase your daily submission limits and will allow for fully rendered HTML to be indexed instantaneously rather than days or weeks after the fact. That means, faster traffic to your website!

8. High quality content is relevant, timely, and answers real searcher questions

Everyone talks about high quality content, but what does that really mean? One thing we know for sure -- it was a common thread throughout multiple C2C sessions.When it comes to evergreen content, we often assume that, because the facts don't change over time, we can take a set-it-and-forget-it approach. Not so! Even evergreen content can lose steam over time. It's a good idea to check your log files to see whether Googlebot is visiting your evergreen content less over time, and how your rankings and organic traffic might be fluctuating with it.User generated content, or "UGC," can also help or hurt your content quality, so it's critical to monitor it. According to Kyle Blanchette, there has been a huge change to how Google handles UGC. For example, sites that allow too much UGC content in forums or Q&A sections can have a negative impact that extends to the rest of the site, but strong editorial guidelines and oversight over user comments can prevent this. When done well, UGC can be a great way to add relevant content to your site that improves the user experience without spending your own resources producing it.Finally, high-quality content is relevant content, and what's more relevant than specific answers to real searcher questions? Chase Lyne introduced the crowd to the benefits of the Google Search Console API via Botify Keywords. By unifying your data and, in this case, adding Botify Keywords to the rest of your Botify data, you're able to find connections you were previously unable to see. This allows you to understand not only how people are searching for your content but also how things like your page depth, content freshness, and crawl frequency may or may not correlate with your rankings.

9. Community is key

There's no denying that SEO is tough, especially at large enterprise organizations with big, complex websites. When the going gets tough, SEOs need to stick together! The theme of community manifested itself in a lot of different ways at C2C.Jennie Baird of News Corp talked about how SEOs at her organization collaborate not only across brands but also across international borders! Through organized councils and continuous, open conversations, the entire News Corp network is able to exchange transferable findings and solutions that prove beneficial to SEOs and their counterparts across the board. There's nothing quite like sharing ideas and working toward a common goal!At Botify, we're committed to supporting the SEO community by not only doing more events like C2C and BotifyCONNECT but by providing educational content that speaks to the challenges and opportunities enterprise SEOs face on a daily basis.That's why we're investing more in SEO support, content, and education through Botify Search Institute, a certification program aimed to help SEOs become proficient in using Botify to tackle their challenges. We're also building a robust ecosystem of partners to build out more capabilities straight from the Botify platform and becoming more prescriptive through machine learning and AI technology in order to clearly tell customers what they should do and when.In essence, we want to make SEO for your company as easy as possible. That means, alleviating your pain points, giving you back valuable time, and sharing our learnings.

10. The future looks bright for Botify and the SEO community

There have been so many changes this year for SEO (mostly for the better!), and we're excited to be part of the innovation ever day.With 2019 coming to an end and 2020 just around the corner, we've got some great initiatives to share with you that will help improve your SEO management and methodology.Botify IntelligenceBotify Intelligence provides prescriptive and prioritized technical SEO tasks based on the unique business requirements of each website. ActionBoard, the first feature under the new Botify Intelligence suite, analyzes key metrics from Botify Analytics and creates an actionable list of tasks, according to level of importance.Also launching as part of Botify Intelligence is AlertPanel, an early warning system that alerts you of critical issues instantly. AlertPanel sees your most important pages at unparalleled speed and scale and is built with the intelligence to know the difference between critical anomalies and seasonal trends. Plus, it can handle the complexities that come along with JavaScript!Botify ActivationThe Botify Activation product suite is designed to streamline and simplify manual SEO tasks. Through FastIndex, you can submit pages for indexing as soon as they're published - and get them indexed FAST.With our inaugural C2C conference proving to be such a success for our customers, partners, and the wider SEO community, we're gearing up to host many more community-connecting events! SEOs are uncovering insight into what works and what doesn't every day on the field and coming up with custom solutions for an ever-changing web. We want to create more opportunities to have open conversations and help one another.We've all heard the saying, "It depends...," when weighty questions fill the air, but without a doubt, it's always the sentences that follow the "..." that ignite spark in our community and inspire us to think bigger.So, let's keep diving in and asking those big questions!