Modern search engines rely heavily on artificial intelligence (AI) to function—and knowing how AI in search works can help you better rank your own website.
AI in search is responsible for everything from what search results you see to what related topics you are encouraged to explore further.
And today, it's impossible for a modern search engine to function without sophisticated AI.
At Marketing AI Institute, we've spent years researching and implementing AI across different marketing functions, and that includes understanding AI for search so we can rank our content better.
In this article, we share our expertise on what you need to know about AI and how search engines use it, so you can use the knowledge to move your business and career forward.
When we talk about AI for search, what does that mean?
AI is an umbrella term that describes a range of smart technologies with the ability to learn and improve on their own.
You may have heard the names of different AI technologies like: machine learning, deep learning, neural networks, voice search, computer vision, image recognition, natural language generation (NLG), and natural language processing (NLP).
These are all considered AI, though these technologies do different things and are at different levels of maturity.
You don't need to know what every type of AI does to understand AI for search. All you need to know is that smart AI machines have the ability to learn and improve without direct human intervention.
Once these machines are trained by humans, they can operate autonomously and learn from their results, getting better and better on their own over time.
That's because AI technologies all excel at finding patterns in data, then using those patterns to make predictions. These predictions can include anything and everything, such as:
As AI systems make predictions, they learn from those predictions to get smarter over time.
You see this in common consumer products like Gmail's Smart Compose feature.
A few years ago, Gmail could predict simple phrases and words that you intended to type next. Today, it has learned so well from billions of emails, it can now finish entire sentences for you. That same type of AI technology is now getting good enough to write entire articles on its own.
This ability to learn makes AI more powerful than the traditional software that came before it. It's also why AI is fundamental to any search engine being used today.
Today's search is way too complex for humans or traditional machines to handle. It's estimated that Google alone processes a whopping 63,000 search queries every second, or north of two trillion searches a year.
Even the largest team of humans couldn't process this volume of search effectively. And traditional software, which has no ability to find patterns in data and make predictions, isn't up to the task either.
It is quite simply impossible to serve up accurate search results in real-time at this speed and scale without AI.
That's why today AI powers almost every part of a search engine, including things like:
In fact, every search result you see is a direct result of decisions made by artificial intelligence.
To get a better sense of how that works, let's deep dive into how search engines use AI.
Almost every aspect of your experience with search engines is directly powered by AI.
The biggest way search engines use AI is to rank webpages, videos, and other content in search results.
Google (and other search engines) rely on complex AI to determine how content gets ranked. The algorithms used by these AI systems have many rules that prioritize different factors, from the types of keywords in your content to your site's user experience.
Together, these factors help Google's AI determine which pages to serve in response to your search query.
While Google gives guidance about what matters to its algorithms, nobody outside the company has full visibility into how these algorithms make decisions.
In fact, much of modern search marketing is focused on learning as much as possible about how this AI works, so companies can rank content in search results.
Search engines are computer applications, but they need to be able to understand human language in order to find users the information they're looking for.
In other words, to serve you the right content, search engines need to understand what you're asking in the first place.
That's a textbook application of natural language processing (NLP), a field of AI dedicated to teaching computers to understand our written language.
Both Google and Microsoft are using NLP to understand their users. A University of Washington study looked at Yandex, the world's fourth largest search engine, and noticed that it also has some advanced applications of NLP and machine learning.
Yandex developers found that they could take all of their users' previous searches and use them to optimize future searches. By creating these personalized search results, they've increased click through rate by about 10%.
Search Engine Journal also points out that Google search uses some of the same practices by optimizing search engine results based on recent previous queries, and allows users to conduct a search with nothing more than a photo (which uses another field of artificial intelligence).
Today, Google has taken understanding search queries a few giant leaps forward. Using its pre-trained language model BERT, the company's search engine now understands complete sentences. That means BERT can understand the context of a search, not just the keywords in it.
Back in the day, certain SEO "specialists" beat the system with shady practices that we've come to know as "black hat SEO techniques." These include aggressive keyword stuffing, cloaking, invisible text the list goes on. No matter the technique, they had one goal: game each search engine results page (SERP) they could.
Of course, this was damaging to search engines because the pages that were at the top of their results weren't necessarily the highest quality content.
Nowadays, companies have updated their algorithms and use AI to separate the high quality content from the low quality spam. Every one of these "algorithm changes" has a potentially massive impact on how companies rank their content and pages in search.
Voice search and image search are newer search capabilities made possible by AI.
AI technologies like NLP have gotten so advanced they can actually understand human voices in real-time, like when you use a Google Assistant. These AI-powered systems can understand your words, then translate those into search results.
The same goes for images. AI technologies like image recognition can determine what is being depicted in an image, then deliver relevant search results around that image.
There are tons of other ways AI influences the search process. But these are the key areas where you're interacting with AI to get the answers you want.
To best understand AI in search, it helps to look at how the world's most popular search engine uses the technology.
When people talk about Google's "algorithm," they're usually referencing how Google delivers search results. Many companies are obsessed with "algorithm changes" that impact their search traffic.
But Google's "algorithm" isn't just one thing. It's a complex web of AI-powered algorithms that determine what search results appear and how they appear.
Nobody on the outside has a complete understanding of how these algorithms work, but often try to learn elements of these algorithms in order to do good SEO.
However, Google frequently gives guidance on how this AI system processes search results. And what's clear is this:
The AI that powers Google search is optimized to provide the very best result for every search. That means the most helpful response to the search and the highest quality result possible, both in content and in on-site experience.
Today, there's no real way to "beat the algorithm" outside of creating extremely high quality content designed for humans.
RankBrain is a core piece of AI at the heart of Google search. This system helps Google search understand which topics are related to your search.
This improves search result accuracy, since Google is often able to determine what you're looking for—and avoids confusing your search terms with similar sounding topics that have totally different meanings.
BERT stands for Bidirectional Encoder Representations from Transformers, and is a type of AI that Google uses to better understand search meaning and intent.
BERT uses AI in the form of natural language processing (NLP), natural language understanding (NLU), and sentiment analysis to process every word in a search query in relation to all the other words in a sentence.
This is different from how Google search used to work. In the past, Google's AI processed words one-by-one in order. The results were accurate, but literal.
For example, Google shares that a search like "2019 brazil traveler to usa need a visa" would have been interpreted in the past as a US traveler wanting a visa to Brazil.
In that scenario, Google didn't account for prepositions that change the context of searches. It wouldn't have taken the word "to" into account, which fundamentally changes the search intent.
BERT is different. BERT takes the whole sentence into account and understands the search is a Brazilian looking for a US visa.
That results in more accurate searches that get at what you're trying to achieve with any given search.
MUM stands for Multitask Unified Model, and is a type of AI that Google uses as a much more powerful version of BERT.
MUM uses more powerful AI techniques to better understand context around searches, search intent, and searches in different languages. In a presentation, Google gave this example of how MUM works:
Say you typed in this search: "I've hiked Mt. Adams and now want to hike Mt. Fuji next fall, what should I do differently to prepare?"
MUM can actually understand the context behind this, which is today difficult for search engines. MUM can return information on how the two mountains are similar and different. It can also surface content for which equipment works best for the new hike.
Here's the big question for businesses: how will AI change the way we we optimize for SEO?
Image Credit: NisonCo PR & SEO
It's pretty simple. Search engines are meant to find users what they're looking for. With machine learning, it will get to the point where they will find exactly that.
That's great for marketers because it means that the best, most relevant content will win.
Choosing the right keywords and following best SEO practices will remain important, but, as search engines become more intelligent, the relevancy and quality of content will be the top ranking factors.
As AI powered search engines collect more data, they become better at delivering results. This doesn't mean you need to game the algorithm. No one will ever know exactly what goes into the AI algorithm that powers any individual search engine.
But it does mean this:
Search engines optimize for something very simple. They want the user to use their search engine the most, and the most frequently.
To do that, they need to serve up the best result. If they don't, users will go elsewhere.
Search engines, like brands and people, have reputations to protect. They don't want to allow poor quality results.
That affects everything from keywords to content quality to page load speed to page experience.
There's no question advanced search engine optimization tactics are important.
But when you think holistically about your SEO strategy in an AI-first world, ask yourself this question:
Is my piece of content something Google would want to serve its users? If you think the answer is yes, then you're already on the right track.