The U.S. District Court for the District of Columbia failed to find evidence that Google harmed rivals by limiting their search visibility.

While Google was cleared in this aspect of a DOJ antitrust lawsuit that began in 2020, Google will still be in court starting Sept. 12. Google will defend claims relating to its Search Ads 360 product, as well as deals the company has struck to ensure it is the default search engine on mobile devices and browsers.

Why we care. It will be worth watching to see whether Google is found guilty of stifling competition in ad buying or in its mobile phone and browser deals – and whether any of this ultimately leads to any changes for search marketers.

Claim: Google weakened Specialized Vertical Providers (SVPs). Google was accused of harming niche companies (e.g., Expedia or TripAdvisor in travel; OpenTable in restaurant reservations; Amazon or eBay in shopping). Specifically by:

• Limiting the visibility of SVPs in Google’s search results.
• Demanding that SVPs make their data available to Google on terms no less favorable than it does to others.

• “With respect to those allegations, Plaintiffs have not demonstrated the requisite anticompetitive effect in the relevant markets to make out a Section 2 prima facie case,” wrote judge Amit P. Mehta.

Claim: Google uses Search Ads 360 to thwart competitors. Google remains accused of “harming competition by delaying the implementation of various SA360 product features for Microsoft Ads that have long been available for Google Ads, thus harming Microsoft’s ability to compete.” Other rival tools mentioned were Skai, Marin and Adobe.

Google launched a new version of SA360 in February 2022, which added four features (call extensions, dynamic search ads, responsive search ads and local inventory ads), and said it was testing a fifth missing feature (auction-time bidding) at issue.

• “…there remains a genuine dispute of material fact with regard to the anticompetitive effect of Google’s disparate development of SA360’s ad-buying features. Summary judgment is therefore denied as to that part of the Attorneys General’s claims,” Mehta wrote.

What Google is saying. Google published the following statement via Court dismisses state AG claims about Google Search:

• “We appreciate the Court’s careful consideration and decision to dismiss claims regarding the design of Google Search. Our engineers build Search to provide the best results and help you quickly find what you’re looking for. People have more ways than ever to access information, and they choose to use Google because it’s helpful. We look forward to showing at trial that promoting and distributing our services is both legal and pro-competitive,” said Kent Walker, Google’s President of Global Affairs & Chief Legal Officer.

Google will be showing fewer rich results in its search results, specifically showing less FAQ rich results across the search result snippets and limiting How-To rich results to desktop devices.

Google said this update will roll out over next week globally.

FAQ changes. Google said the FAQs, FAQPage structured data, rich results will only be shown for “well-known, authoritative government and health websites.”

For all other sites, this rich result will no longer be shown regularly, Google added.

Which sites Google decides to show them for are automated and algorithmic.

Don’t remove structured data. Google said there is no reason to remove structured data from your site. Google said “Structured data that’s not being used does not cause problems for Search, but also has no visible effects in Google Search.”

HowTo changes. Google said the How-To, from HowTo structured data, rich results will only be shown for desktop users, and not for users on mobile devices.

Google added that “with mobile indexing, Google indexes the mobile version of a website as the basis for indexing: to have How-To rich results shown on desktop, the mobile version of your website must include the appropriate markup.”

Why we care. If your site is impacted by this, you may see a decline in clicks from Google Search to your site. Your Google Search Console performance reports would show the decline in clicks and your analytics traffic would show a drop in traffic from Google Search.

Google wrote, “For both of these items, you may also notice this change in the Search Console reporting for your website. In particular, this will be visible in the metrics shown for FAQ and How-To search appearances in the performance report, and in the number of impressions reported in the appropriate enhancement reports. This change does not affect the number of items reported in the enhancement reports. The search appearances, and the reports, will remain in Search Console for the time being.”

Search results are not chat results. 

In search, the input is a query. The output is what we have called a SERP, or search engine results page. 

A SERP is full of answers and ads. It has links to websites, text, images, featured snippets, and videos. “SERP” has been used in search marketing for more than two decades. 

In chat, an input is a prompt. The output is text, ads, images, videos (and sometimes clear links, in the form of small citations) – often trained on or powered by search results.

But what’s that page of results called, exactly? 

We at Search Engine Land believe that this LLM-fueled, generative AI, end user-facing output – whether it’s Google’s Search Generative Experience, the new Bing or another search/AI platform – needs a name. 

“Kein ding sei wo das wort gebricht,” my colleague Kim Davis put it recently. Translation: “No thing can be where the word is lacking.”

So we are introducing a neologism into the search marketing lexicon: CHERP for Chat Experience Result Page.

Search vs. chat experience: The distinction is needed

There’s a fine line between language that clarifies concepts and jargon that unnecessarily clouds issues and creates confusion. 

We are at one such crossroads right now with generative AI. Specifically: the results pages various chatbot interfaces produce. 

We expect Google and Microsoft Bing to continuously update and test the various elements, just as they continue to tinker with the traditional search results pages.

The search marketing industry currently lacks the precise language needed to distinguish between traditional search results pages and the results produced by generative AI. 

Google and Bing have referred to their generative AI offerings as “experiences.” So that’s why we’re suggesting a new acronym for the results by those experiences be dubbed Chat Experience Results Pages or CHERPs. 

What is a SERP?

The term SERP stands for “search engine results page.” In simple terms, it’s a page of search results you see after you enter a query on Google, Microsoft BIng or any other platform.

Origins of SERP

The origin of the term “SERP” can be traced to 2000 in a forum post by Webmaster World founder Brett Tabke. 

The earliest SERPs typically consisted of 10 blue links and endless pagination of search results. PPC ads were also present, usually above organic search results and on the right rail.

Evolution of SERPs

Google began reimagining the SERP with Universal Search in 2007, blending Search with news, video, images, local, maps and more. And as 2010 approached, Google was introducing a variety of instant answers, including weather and sports scores. 

Big Google SERP changes continued in the 2010s, most notably with the integration of the Knowledge Graph into Google Search and seen on the SERPs via knowledge panels. In 2014, Search Engine Land reported on a new type of detailed answer that would later become known as featured snippets.

Now, in 2023, we have seen a massive change to Search: generative AI. Microsoft calls it the new Bing, or Bing Chat. Google calls it a Search Generative Experience. 

As of this writing, both experimental experiences are not fully rolled out. But it’s only a matter of weeks or months before that happens.

What is a CHERP?

The term CHERP stands for “chat experience results page.” In simple terms, it’s the generative AI result you see after you enter a prompt on Google, Microsoft Bing, ChatGPT or any other generative AI platform.  

Let’s illustrate with an example prompt – “what is a SERP” – using Google, Microsoft and ChatGPT.

Google

The chat results page consists of:

• Two paragraphs of text (one dedicated to a Search Engine Results Page; the other to a Supplemental Executive Retirement Plan). This includes two citations (the clickable quotation marks) that, when clicked, result in a dropdown, which lets you click on three sources, which are brand names. 
• Three links (with images) in the Snapshot, which in this case are duplicative.
• Suggestions to Ask a follow up.

New Bing

The chat results page consists of:

• One paragraph of text (only discussing a Search Engine Results Page). This includes three visible numbered citations (plus two that are not visible, you have to click on the +2 more). The sources are domains. 
• No other links or images.
• Two suggested follow-up question bubbles.

ChatGPT

The chat results page consists of:

• Nine paragraphs of text, including an ordered list of six items (only discussing a Search Engine Results Page).
• No citations, links or images. 
• No suggested follow-up questions.

All of these results minimize traditional search and are based on answering questions – and encouraging users to ask more questions. They are results pages unto themselves.

Search engines have SERPs. Answer engines have CHERPs.

Language must change to reflect new realities

Why is coining the term CHERPs necessary? Do search marketers really need another acronym?

Yes. It’s necessary for accurate communication and to provide clarity for clients or stakeholders when explaining whether you have visibility in the search or chat experience. 

Just on Google, since it continues to be, by far, the biggest player:

• Is your ad appearing above, within or beneath SGE?
• Even if you have strong organic visibility in search results, will that matter if your competitors are getting cited (and potentially clicks from) the chat results above the organic search results?

Hopefully, Google and Bing will provide us with the data we need to understand and report on how people are getting to our websites. 

Search continues to evolve. As it does, our language also must evolve.

CHERPs, as a new term, is familiar, while also being different enough to create a clear distinction.

Did you ever used to say or write, “rank on Page 1 of Google”? Well, you can’t do that anymore, thanks to Continuous Scroll. Google evolved. Our language must evolve with the platforms. 

There has also been a push within tech to make terms more inclusive. That may be why Google renamed its Webmaster Guidelines to Search Essentials. The term “webmaster” has become a relic of an earlier era. 

Thankfully, we hear the terms “white hat” and “black hat” a lot less. I’ve always found these to be cartoony (see also: link juice) and undermine all the great and professional work we do that drives billions of dollars in revenue every month for brands and businesses of all sizes.

Words matter. Clarity matters. 

SERPs and CHERPs will co-exist

To be clear, SERPs will continue to exist – as long as Google and Bing serve search results. We’re not suggesting CHERP as a replacement for SERP, like how many have tried to “rebrand” SEO over the years.

No, we think of SERPs and CHERPs as two unique entities that may or may not occupy the same space on a platform that produces content using generative AI.

The purpose of introducing CHERPs as a new term is so that we, as an industry, can clearly distinguish between results pages from search versus chat. 

We think it’s needed. We hope you agree. 

Has your boss or client ever question the value of your SEO results because they can’t see the direct revenue? 

Do you see some stakeholders get wrapped up in vanity metrics that affect your strategic direction?

We’ve all been there.

This article will help you find a way to get away from vanity metrics and determine the SEO metrics that matter.

We know they do because:

• We’ll let our customers define these – so if they matter to our customers, they should matter to us.
• These metrics are leading indicators. They are much more strategic and give us insight into what we can do to affect the big-money metrics and our future results.

To set the metrics that matter, you need to revisit our search strategy by:

• Defining and understanding your audience.
• Channel research.
• Building your customer journey map.
• Socializing the metrics that matter in your business.

Defining and understanding your audience

Every strategy needs to be built with your audience as the foundation. 

As an organization, you should aim for an “outside-in” approach rather than “inside-out.” 

An inside-out business will broadcast what they want to say. 

They’ll use their language, plan their customer journeys and base many decisions on assumptions. 

Outside-in organizations are the complete opposite.

They listen to their audiences, craft their language and base their customer journeys and decision-making on what they learn.

Before you can plan your measurement strategy, you need to know your audience.

• What motivates them?
• What worries them?
• What pushes them forward in a journey?
• What holds them back?
• Who or where do they go for advice? 
• How do we do this? 

Regardless of the method, it should always be bespoke based on the client, access to data, audiences etc.

Below are some ideas.

Surveys

Use surveys with behavioral science baked in. Avoid leading respondents to the answers you want.

Instead, craft the survey so they can tell you exactly how they feel.

These surveys could be sent to various audiences – your current customers or your marketing email database.

You can work with a market research company to find an audience or a mix of the above. 

Social listening

Social listening tools help you find what is being said organically online. 

They help:

• Identify trends.
• Gauge sentiment.
• Reveal audience language. 

They also offer valuable competitor, partner and publication insights. 

Desk research

You likely aren’t the first person to seek audience insights. Existing studies can offer valuable leads to explore.

Client persona workshops

In your organization, you’ll have colleagues who interact directly with customers, like sales or customer service teams. 

Persona workshops leverage their knowledge to validate hypotheses or test findings.

1-2-1 interviews

Like workshops, 1-2-1 conversations unlock potent insights when you find the right person to talk to. 

Remember, it’s a small sample, not your entire strategy. They validate or create hypotheses. 

Align these individuals with your customer-centric strategy for success.

CRM analysis / existing data analysis

Businesses often possess valuable yet untapped data. This research uncovers practical uses, from quantitative data on customer journey timelines to qualitative review trend analysis.

After selecting research methods and conducting them, the data must be analyzed and aligned with customer personas. These personas go beyond demographic details to focus on:

• Motivations.
• Thoughts.
• Feelings.
• Actions.

Channel research

With your audience research nearing completion, you can look at the channels for more learning.

Your research should have surfaced some channels the audience regularly uses, but you may also have other thoughts on channels to validate. 

The aim here is to gather as much quantitative data as possible to understand the size of the audience opportunity on the varying channels. You should be aiming to cover:

Keyword research and search trends

Unearthing how the customer searches (and aligning this to the funnel based on keyword intent) is a big puzzle piece in most projects like this. 

Also, consider the trends:

• Is there seasonality?
• What affects search volume positively and negatively? 

The obvious starting point is Google and Microsoft Advertising. But think wider when it comes to where people go to search (i.e., hashtags on social, YouTube, etc.) 

The data points here will be important for prioritizing your strategy based on impact.

Audience insights and targeting

Using the different search and social channels to understand how you might be able to find your audience based on what you now know about them, can you draw any insights from this? 

What size are the varying audience buckets? How might you be able to target them at different stages in the journey? 

Competitor / partner / retailer reviews

Understanding what your competition is doing and how they perform is always helpful when planning a strategy. 

This is no different when you think about audiences. 

With the deeper understanding of your audiences that you now have, looking to your competitors and your peers to see their approach can be eye-opening. 

Digital PR / social media trends

Looking at the content out there to see what has been popular and how this relates to your audiences is a powerful exercise.

The power of influence, whether from influencers, the collective social audience or publishing businesses, is high.

Having this angle in your research is a must. 

Current performance audit

With all of this research complete, you must consider your performance. 

Without understanding where you are now, it is irresponsible to plan where you are going, as you don’t know the size of the task. 

This performance audit will also help you prioritize actions and manage expectations.

Building your customer journey map

You are now the oracle of information on your customers and many of their key touch points – what now?

You need to turn this into your customer journey map. This is where you will see your KPIs start to appear.

The map can be formed however you wish. Our tool of choice is Miro – for its fluidity in moving information around.

The board needs to be laid out, covering the customer journey. My preference for the stages are:

• Awareness.
• Consideration.
• Purchase/Inquiry.
• Retention and advocacy. 

You may wish to split these differently based on your customer journey, but these are standard. 

The key job here is creating a map for each persona… sorry, I know that is a lot of work, but trust me – it will be worth it!

For each persona, you should aim to map the following for each stage of the journey:

• What they are thinking.
• What they are feeling.
• What they are doing or aiming to do.
• The touch points (your channel opportunities).
• Any behavioral insights e.g., pain points, push points etc.
• KPIs.

This should all come from your research and, in many cases, will have data points to back up your information. 

• Tip: Use key information on the map and link to spreadsheets with deeper dive data and insight. 

From a strategic viewpoint, focus on the opportunity’s size, like understanding search volume if Google is crucial for a specific audience during a particular journey phase. 

Also, assess your performance to prioritize tactical actions. 

To make this clearer, use a map with colored dots (red, amber, green) for performance ratings and stars for major opportunities.

There’s no right or wrong way to present your customer journey. The key here is that it is useful to you and your business. 

Below is an example of a tidy, nicely designed journey. I find a clean and tidy version useful to present to the business. 

But this is not what it looks like when in flow.

My work-in-progress version (covered in colored dots and stars) is the one I work from and keep most up to date. 

Now I know promised you that this article would help you set the measurement framework for your strategy.

If you haven’t spotted it already, we’ve actually just done that. 

After mapping each persona’s journey stages, touchpoints, and their thoughts, feelings, and actions, you gain a clear view of their actions or desired actions. These are the actions you aim to track.

These are leading indicators for your strategy. They gauge audience content engagement and, with time and analysis, aid in predicting money metrics.

At a high level, you should be able to build a map (like the example below) to define the different roles of the channels and their purpose within the strategy. 

Awareness metrics include:

• Engagement
• Newsletter signups
• Downloads
• Video views
• Social follows
• Pinterest / image engagement
• Saves/wish list
• Next page flow
• Scroll depth
• Dwell time

Consideration metrics include:

• Many of the awareness metrics are still valid, plus…
• Live chats
• FAQ interactions
• Review interactions
• Specific, more detailed content views, scroll depth etc.

Purchase/inquiry metrics include:

• Sales and inquiries
• Bookings
• Telephone call tracking
• Add to basket (opportunity for an abandon basket follow-up and remarketing)
• Online to in-store visibility

Retention metrics include:

• Downloads
• Newsletter sign-ups
• Newsletter engagement
• Returning user rate
• Social engagement – likes/shares/comments
• Use of refer a friend discount code

To recap, customer research shapes our strategy. 

Customers inform buying habits, preferences, and journeys for personas based on motivation. 

This reveals key channels and actions to track forming a customer-centric strategy.

The next key step is to ensure you socialize this information within the business and get buy-in – you want everyone aligned on your strategy. 

Socializing the metrics that matter in your business

It’s crucial to guide the business through the journey you’ve worked on for weeks or even months. 

Ensure they understand that customers have shaped this strategy, which holds immense power for future decisions. 

The strategy provides transparency into actions before user conversion. 

When performance falls short, clear data points reveal the why. Additionally, you can test and learn for performance enhancement.

Consider using psychological nudges to make this information resonate effectively.

The customer journey must drive your SEO measurement strategy

Research involving customers and desired audience, even within channels, uncovers customer priorities. 

Remember, what’s crucial for customers matters for you, too. This understanding guides metric selection across the journey and for different audiences.

Getting this right unleashes digital performance. 

Leading indicators improve your strategy throughout the journey, rather than relying solely on big money metrics without insight into hits or misses.

For success, secure business buy-in and ensure everyone understands the essential metrics.

A new metric, called Brand Authority, is now available in the Moz Pro SEO toolset and the Moz API, the company announced today.

What is Brand Authority. Moz’s new Brand Authority metric can be found in Domain Overview. Moz assigns an “online brand strength” score between 1-100. There is also a new brand comparison visualization.

• “At a high level, Brand Authority combines Moz’s knowledge of search intent and search volume to calculate a score that reflects the total strength of a website’s branded search terms,” according to Dr. Peter J. Meyers, Moz marketing scientist.
• “For instance, we know that searches like [iPhone 15] and [iPad Pro] strongly reflect awareness of the Apple brand. Brand Authority combines all this data and scales it against the strongest known brands.”
• “While we can’t tie a specific Brand Authority score to a specific offline factor, it inherently reflects the total (including offline) influence big brands have in the world.”

This metric is based only on U.S. data, but Moz said it plans to add data from more regions in 2024.

The data points used by Brand Authority will be updated at different intervals, but the metric will be refreshed bi-monthly, Meyers told Search Engine Land.

The price of a Moz Pro won’t be increasing with the addition of this metric. However, accessing Brand Authority data in the API will incur additional charges, Moz said.

Why we care. Moz customers may want to check out the metric and see whether it’s useful for SEO, PR or other marketing activities – and get a general sense of how Moz assesses your brand strength compared to your competition. However, I expect many SEOs, especially those who don’t use Moz, to be skeptical of another 1-100 “authority score.”

Why Moz cares. I asked them why how SEOs and marketers can use this score. Meyers said:

• “While the mechanisms of how brands impact search results are complex and not always transparent, we know that understanding and being able to measure brand influence is incredibly important for SEOs, hence the motivation to create a metric like Brand Authority.
• “Because Brand Authority encompasses success signals beyond search, it has many interesting use cases for PR and broader marketers, but it is firmly rooted in our SEO expertise.”

How Moz describes Brand Authority. According to Moz, users can use the metric to:

• “…assess their marketing gaps to maximize their return on investment (ROI), see the true value of sales prospects and potential acquisition targets, and assess the real influence of the media brands that pick up their stories.”

How it’s different from Domain Authority. Moz is the company that created the controversial Domain Authority metric, which predicts “how likely a website is to rank” in search, using a similar 1-100 scoring system.

Domain Authority isn’t going away. Both metrics will co-exist. Meyers said the metrics are “complementary” and there is little overlap between how the two metrics are computed:

• “Domain Authority remains an important indicator of online strength and ranking potential, while Brand Authority captures broader signals of an organization’s influence.
• “Our improved Domain Overview tool now displays a 4-quadrant visualization of Domain Authority vs. Brand Authority to help customers understand their relationship to their competition.”

Brand Authority vs. other brand score metrics. Search marketers have relied on metrics such as NPS (net promotor score), share of voice and sentiment analysis to measure brand strength. But Brand Authority attempts to “capture people’s awareness of a brand even before they begin their search and buyer’s journey,” Meyers said:

• “Brand Authority is a search-centric metric that helps us understand the influence of brands (including ‘offline’ influence, such as traditional advertising and word of mouth) on our online efforts, by analyzing searchers’ awareness of brand terms (including sub-brands, products and services),” Meyers said.

Top 500 U.S. brands. In addition to the new metric, Moz revealed a list of 500 sites with the most Brand Authority. Five brands achieved a perfect score of 100:

• google.com
• youtube.com
• facebook.com
• amazon.com
• walmart.com

Also making the top 10:

• target.com (98)
• yahoo.com (98)
• homedepot.com (96)
• walgreens (94)
• foxnews.com (94)

Google Merchant Center has been issuing a new warning – for “inappropriate titles.”

The message appears to apply to Google Discover and Demand Gen ads within Google Merchant Center.

Product titles that fail to meet Google’s criteria face being penalized with restricted visibility in the U.S., according to a screenshot of the warning:

Why we care. A drop in visibility means your campaign will be seen by fewer potential customers, which could harm your revenue and ROI. In addition, getting Google to lift a penalty can be a lengthy process and, depending on how long it takes, this has the potential to have a long-term impact on your account, negatively affecting your visibility.

Google’s requirements. How can you avoid this kind of Google penalty? Your product title must meet the following requirements:

• Assets must meet the Discovery and Demand Gen product ads quality requirements
• Assets can’t contain obscene or offensive language, including but not limited to profanity and sexual innuendos
• Assets can’t use placeholder text, distasteful formatting such as nonsensical punctuation or capitalization or incomplete sentences.
• Assets can’t be longer than 10 words before a delimiter
• Asserts can’t contain excessive details like product codes, SKUs or part numbers that aren;’t part of the product name.

Reaction. Google Ads ecommerce expert Dennis Moons, flagged the issue on X, prompting several other marketers to report similar experiences.

Some have commented that there may be a “bug” with the warning as it is being issued to product titles that are “harmless.” However, Google is yet to respond.

Deep dive. Read Google’s Discovery and Demand Gen Campaigns guide for more information.

When strategically utilized, ChatGPT can surpass manual human effort in output quality. 

No, the tools won’t write better content.

Instead, I believe a writer armed with this technology can craft optimized content that’s better aligned with Google’s ranking criteria.

By exploring various methods of content scoring and entity extraction, I aim to guide you toward maximizing the tools’ benefits.

“Beyond keywords: How entities impact modern SEO strategies” discussed how and why to include relevant entities across your website (i.e., topical map). 

This article will focus on why and how to use entities to create better-ranking SEO content. 

Before discussing how software optimizes entity use for search results, let’s understand the similarities between entity SEO and OpenAI’s ChatGPT.

Building blocks of language 

At its most basic level, language is built around:

• Subjects: What (or whom) the sentence is about.
• Predicates: Says something about the subject. 

For example, in the sentence “The cat sat on the mat,” “The cat” is the subject and “sat on the mat” is the predicate.

Both Google’s search engine and OpenAI’s ChatGPT are designed to understand the fundamental structure of language. 

Semantic search engines focus on understanding content in a computationally efficient way. 

ChatGPT goes a step further, using far more computation to generate content. 

Semantic search engines 

Google’s search engine identifies entities, which are essentially the subjects of sentences on a webpage. 

It then uses the context around those entities to understand the predicates – or what is being said about those entities. 

This enables Google to understand the page’s content and how it might be relevant to a user’s search query. 

The relationships under consideration are depicted in Google’s Knowledge Graph. 

When Google analyzes an article, it uses its Knowledge Graph to gain deeper insights. 

It identifies relevant entities and predicates in the content, which allows it to discern what keyword searches the piece is most pertinent to.

OpenAI’s ChatGPT

On the other hand, ChatGPT uses its transformer model and embeddings to understand both subjects and predicates. 

Specifically, the model’s attention mechanism allows it to understand the relationships between different words in a sentence, effectively understanding the predicate. 

The embeddings, meanwhile, help the model understand the relationships and meanings of the words themselves, which includes understanding the subjects.

Despite their vast differences, ChatGPT and entity SEO share a common capability: 

Recognizing entities and predicates relevant to a topic. This commonality underscores how vital entities are to our comprehension of language.

Despite the complexities, SEO professionals should focus their efforts on entities, subjects and their predicates. 

So how do we use this new understanding to optimize our content?

Optimizing new content for entities

Google identifies entities and their predicates on a webpage. It also compares them across potentially relevant pages. 

In essence, it’s like a matchmaker, trying to find the best match between a user’s search query and the content available on the web. 

Given that Google’s algorithm is optimized for high-quality results, start your optimization process by examining the top 10 Google results. 

This will give you insights into the attributes that Google favors for a given search term. 

At our agency, we apply a framework to identify potential enhancements that can make our articles 10-20% better, which I’ll share below.

A framework that prioritizes the right aspects can illustrate the difference between your content and the highest-ranking material. 

When creating content, we follow this framework and fulfill these priority items. 

We set ourselves up for immediate success if we meet all these criteria.

Diving into the entity portion of the checklist 

Think of it like this: 

Imagine Google keeps track of how often certain entities and their predicates appear together.

It’s figured out which combinations are most important to users searching for specific topics. 

As an SEO expert, your goal should be to include these key entities in your content, which you can identify by reverse engineering the top results that Google is showing you it already likes. 

If your webpage includes the entities and predicates Google expects for a given user search, your content will earn a higher score. 

We’ll touch on the exception of new entity relationships in a future discussion.

This is where tools that strategically utilize ChatGPT and NLP techniques come into play to help analyze the top 10 results. 

Attempting this manually can be time-consuming and difficult because of the scale of data you’d have to consume. 

Step 1: Extracting entities

To do this analysis, you’ll need to mimic Google’s native entity and predicate extraction processes and then turn your findings into a workable action plan/writer’s guide. 

In technical jargon, this exercise is known as named entity recognition, and various NLP libraries have their own unique approaches. 

Luckily, many content writing tools are available on the market that automate these steps. 

However, before you blindly follow the recommendations of an SEO tool, it is helpful to understand what it will and won’t do well. 

Named entity recognition (NER)

Think of NER as a two-step process: spotting and categorizing.

Spotting

• The first step is like a game of “I Spy.” The algorithm reads through the text word by word, looking for words or phrases that could be entities. It’s like someone reading a book and highlighting the names of people, places, or dates.

Categorizing

• Once the algorithm has spotted potential entities, the next step is to figure out what type of entity each one is. This is like sorting the highlighted words into different buckets: one for People, one for Locations, one for Dates, and so on.

Let’s consider an example. If we have the sentence: “Elon Musk was born in Pretoria in 1971.”

In the spotting step, the algorithm might identify “Elon Musk”, “Pretoria”, and “1971” as potential entities.

In the categorizing step, it would then classify “Elon Musk” as a Person, “Pretoria” as a Location, and “1971” as a Date.

The algorithm uses a combination of rules and machine learning models trained on large amounts of text. 

These models have learned from examples what different types of entities look like, so they can make educated guesses when encountering new text.

Relation extraction (RE) 

After NER identifies the entities in a text, the next step is to understand the relationships between these entities. 

This is done through a process called relation extraction (RE). These relationships essentially act as the predicates that connect the entities.

In the context of NLP, these connections are often represented as triples, which are sets of three items: 

• A subject.
• A predicate.
• An object. 

The subject and object are typically the entities identified through NER, and the predicate is the relationship between them, identified through RE.

The concept of using triples to decipher and comprehend relationships is beautifully simplistic. We can grasp the core ideas presented with minimal computation, time, or memory. 

It’s a testament to the nature of language that we get a good sense of what’s being said by zeroing in on just the entities and their predicates. 

Remove all the extra words, and what you’re left with are the key components – a snapshot, if you will, of the relationships that the author is weaving.

Extracting relationships and representing them as triples is a crucial step in NLP. 

It allows computers to understand the text’s narrative and the context around the identified entities, enabling more nuanced understanding and generation of human language. 

Remember that Google is still a machine, and its understanding of language is different from human understanding. 

Also, Google doesn’t have to write content but must balance computational demands. It can instead extract the minimal amount of information that achieves the goal of linking content to search query. 

Step 2: Building a writer’s guide

We must mimic Google’s process of extracting entities and their relationships to generate a useful analysis and roadmap. 

We must understand and employ these two key ideas in the top 10 search results. Fortunately, there are multiple ways to approach the roadmap building. 

• We can rely on entity extraction
• We can extract keyword phrases. 

The entity route

One route that can be tested is a methodology similar to tools like InLinks. 

These platforms employ entity extraction on the top 10 results, likely utilizing Google Cloud’s NER API. 

Next, they determine the minimum and maximum frequencies of the extracted entities within the content. 

Based on your usage of these entities, they grade your content. 

To determine successful entity usage within your material, these platforms often devise their own entity recognition algorithms. 

Pros and cons

This method is effective and can help you create more authoritative content. However, it overlooks a key aspect: relation extraction. 

While we can match the usage of entities with the top-ranking articles, it’s challenging to verify if our content includes all the relevant predicates or relationships between these entities. (Note: Google Cloud doesn’t publicly share their relation extraction API.)

Another potential pitfall of this strategy is that it promotes the inclusion of every entity found in the top 10 articles. 

Ideally, you’d want to encompass everything, but the reality is that some entities carry more weight than others. 

Further complicating matters, search results often contain mixed intents, meaning some entities are only pertinent for articles catering to specific search intents. 

For instance, the entity makeup of a product listing page will differ significantly from a blog post. 

It can also be challenging for a writer to convert single-word entities into relevant topics for their content. Turning certain competitors on and off can help remedy these issues. 

Don’t get me wrong, I’m a fan of these tools and use them as part of my analysis. 

Every approach I’ll share here has its own advantages and drawbacks, all of which can enhance your content to some degree. 

However, my aim is to present the diverse ways you can use technology and ChatGPT to optimize entities.

The keyword phrase route 

Another strategy we’ve adopted in our tools involves extracting the most crucial keyword phrases from the top 10 competitors. 

The beauty of keyword phrases lies in their transparency, making it easier for the end user to understand what they represent.

Plus, they typically capture the subject and predicate of key topics instead of just the subjects or entities. 

However, one downside is that users often struggle to seamlessly incorporate these keywords into their content. 

Instead, they tend to shoehorn in keywords, missing the essence of what the keyword phrase embodies. 

Unfortunately, from a dev standpoint, measuring and scoring a writer based on their ability to capture a keyword phrase essence is difficult. 

Therefore, developers must score based on the exact usage of a keyword phrase, which discourages the true intended behavior. 

Another significant advantage of the keyword phrase approach is that keywords often serve as signposts for AI tools like ChatGPT, ensuring that the generative text model captures the key entities and their predicates (i.e., triples).

Lastly, consider the difference between being given a lengthy list of nouns versus a list of keyword phrases. 

You might find it perplexing to weave a coherent narrative from a disconnected list of nouns as a writer. 

But when you’re presented with keyword phrases, it’s much easier to discern how they might naturally interconnect within a paragraph, contributing to a more coherent and meaningful narrative.

What are the different approaches to extracting keyword phrases?

We’ve established that keyword phrases can effectively guide what topics you need to write about.

Still, it’s important to note that different tools in the market have varying approaches to extracting these crucial phrases.

Keyword extraction is a fundamental task in NLP that involves identifying important words or phrases that can summarize the content of a text. 

There are several popular keyword extraction algorithms, each with its own strengths and weaknesses when capturing the entities on a page. 

TF-IDF (Term frequency-inverse document frequency)

Although TF-IDF has been a popular discussion point among SEOs, it’s often misunderstood, and its insights are not always applied correctly. 

Blindly adhering to its scoring can, surprisingly, detract from content quality. 

TF-IDF weights each word in a document based on its frequency within the document and its rarity across all documents. 

While it’s a simple and swift method, it doesn’t consider words’ context or semantic meaning.

What value can it provide

High-scoring words represent terms that are frequent on individual pages and infrequent across the entire collection of top-ranking pages. 

On the one hand, these terms can be seen as markers of unique, distinguishing content. 

They might reveal specific aspects or subtopics within your target keyword theme that are not thoroughly covered by competitors, allowing you to provide unique value. 

However, the high-scoring terms can also be misleading. 

TF-IDF can reveal a high score on terms uniquely important to specific ranking articles but doesn’t represent terms or topics generally important for ranking. 

A basic example of this could be a company’s brand name. It could be used repeatedly in a single document or article but never in other ranking articles. 

Including it in your content would make zero sense. 

On the other hand, if you find terms with lower TF-IDF scores that appear consistently across high-ranking pages, these could indicate crucial “baseline” content that your page should contain. 

They might not be unique, but they could be necessary for relevance to the given keyword or topic.

Note: TF-IDF represents many strategies, but additional mathematics can be applied in variations. These include algorithms like BM25 to introduce saturation points or calculations of diminishing returns. 

Additionally, TF-IDF can be vastly improved, and often is, by retroactively showing for each term the percentage of top 10 pages that include the word. Here, the algorithm helps you identify noteworthy terms but then helps you better understand the “baseline” terms by showing the extent to which the top 10 ranking terms share the terms. 

RAKE (Rapid automatic keyword extraction)

RAKE considers all phrases as potential keywords, which can be useful for capturing multi-word entities. 

However, it doesn’t consider the order of words, which can lead to nonsensical phrases.

Applying the RAKE algorithm to each of the top 10 pages separately will produce a list of key phrases for each page. 

The next step is to look for overlap – key phrases that appear on multiple top-ranking pages. 

These common phrases may indicate topics of particular importance that search engines expect to see in relation to your target keyword. 

By integrating these phrases into your own content (in a meaningful and natural way), you could potentially increase your page’s relevance and, thereby, its ranking for the targeted keyword.

However, it’s important to note that not all shared phrases are necessarily beneficial. Some may be common because they are generic or broadly associated with the topic. 

The goal is to find those shared phrases that carry significant meaning and context related to your specific keyword.

All keyword extraction techniques can be improved by allowing you to use your brain to turn competitors or keywords on or off. 

The ability to turn competitors and specific keywords on and off will help remedy the aforementioned problems. 

Competitors

Keywords 

This approach essentially provides a way to combine the strengths of both RAKE (identifying key phrases within individual documents) and a more TF-IDF-like strategy (considering the importance of terms across a collection of documents). 

By doing so, you can harness a more holistic understanding of the content landscape for your target keyword, guiding you to create unique and relevant content.

YAKE (Yet another keyword extractor)

Lastly, YAKE considers the frequency of words and their position in the text. 

This can help identify important entities that appear at the beginning or end of a document. 

However, it may miss important entities that appear in the middle.

Each algorithm scans the text and identifies potential keywords based on various criteria (e.g., frequency, position, semantic similarity). 

They then assign a score to each potential keyword; the highest-scoring keywords are selected as the final.

These algorithms can effectively capture entities, but there are limitations. 

For example, they may miss rare entities or don’t appear as keywords in the text. They may also struggle with entities with multiple names or that are referred to in different ways.

In summary, keywords provide a couple of enhancements over straight NER. 

• They are easier for a writer to understand. 
• They capture both the predicates and the entities. 
• As we will see in the next section, they operate as better guideposts for AI to write entity-optimized content. 

OpenAI 

ChatGPT and OpenAI are truly game-changers in SEO. 

To unlock its full potential, it needs a well-informed SEO expert to steer it along the right path and a meticulously constructed entity map to guide it on relevant topics to write about.

Consider a scenario: 

You might have realized you can head to ChatGPT and ask it to write an article about almost any subject, and it will readily comply. 

However, the question is, will the resulting article be optimized to rank for a keyword?

We must draw a clear distinction between general content and search-optimized content. 

When AI is left to its own devices to write your content, it tends to generate an article that appeals to a regular reader. 

However, content optimized for SEO dances to a different tune. 

Google tends to favor content that’s scannable, includes definitions and necessary background knowledge, and fundamentally offers plenty of hooks for readers to find answers to their search queries.

ChatGPT, being powered by transformer architecture, tends to produce content based on observed frequency and patterns in the data it was trained on. A small fraction of this data consists of top-ranking Google articles. 

In contrast, as time passes, Google adapts its search results to their effectiveness for a user – essentially survival of the fittest content pieces. 

The entities found in these enduring articles are vital to emulate as foundational content, which tends to diverge significantly from what ChatGPT produces right out of the box.

The key takeaway is that there’s a difference between content that’s a winner from a readability standpoint and content that’s a winner in a Google environment. In the world of web content, utility trumps all. 

As shown long ago by Nielsen, scannability reigns supreme.

User’s prefer scanning web content over reading from top to bottom. This behavior usually follows an F-shaped pattern. Writing content that does well in search should focus on being easily scannable vs. purely written to be read from top to bottom. 

ChatGPT out of the box

Let’s observe how ChatGPT performs right out of the box, using Noble and Inlinks for scoring. 

Even with a meticulously crafted prompt, without the context of what’s working on the first page of Google, ChatGPT often misses the mark, producing content unlikely to compete.

I prompted ChatGPT to write an article on “How much do travel nurses make per hour.”

When paired with SEO analysis

However, ChatGPT can exhibit its true power when combined with SERP analysis and keywords crucial for ranking. 

By asking ChatGPT to include these terms, the AI is guided toward generating topically relevant content.

Here are a few important points to remember

While ChatGPT will incorporate many key entities relevant to a topic, using tools that analyze SERP results can significantly enhance the mix of entities in your content.

Also, these differences can be more pronounced depending on the subject matter, but if you run this experiment more times, you’ll find this is a consistent trend. 

Approaches based on keywords fulfill two requirements simultaneously: 

• Ensure the inclusion of the most critical entities.
• Provide a more rigorous grading system since they encompass both predicates and entities.

Additional insights

ChatGPT might have difficulties achieving the necessary content length on its own.

The further the page’s intent deviates from blog-style posts, the more noticeable the performance gap becomes between ChatGPT and SEO tools that use ChatGPT separately.

Despite the AI’s capabilities, it’s essential to remember the human factor. Not all pages should be analyzed due to mixed search results. 

Additionally, keyword extraction techniques aren’t foolproof, and edge cases can yield irrelevant proper nouns that might still make it through the scoring system. 

Therefore, the optimal balance between human intervention and AI involves manually disabling any competing site with a different intent and combing your keyword list to prune any glaringly wrong keywords.

Last steps: Taking it one step further

The methods we’ve discussed are a starting point, allowing you to create content that covers a broader range of entities and their predicates than any of your competitors. 

By following this approach, you’re writing content that mirrors the characteristics of pages that Google already favors. 

But remember, this is just a jumping-off point. These competing pages have likely been around for some time and may have accrued more backlinks and user metrics. 

If your goal is to outperform them, you’ll need to make your content stand out even more.

As the web becomes increasingly saturated with AI-generated content, it’s reasonable to speculate that Google might start favoring websites it trusts to establish new entity relationships. This will likely shift how content is evaluated, emphasizing original thought and innovation more. 

As a writer, this means going beyond merely incorporating the subjects covered by the top 10 results. Instead, ask yourself: what unique perspective can you offer missing from the current top 10?

It’s not just about the tools. It’s about us, the strategists, the thinkers, the creators. 

It’s about how we wield these tools and how we balance the computational prowess of software with the creative spark of the human mind. 

Just like in the world of chess, it’s the combination of machine precision and human ingenuity that truly makes a difference. 

So, let’s embrace this new era of SEO, where we’re creating content and crafting experiences that resonate with our audience and stand out in the vast digital landscape.

Better keyword rankings. More traffic. Extra conversions from organic search. These are the KPIs used to measure SEO performance.

But beyond growth metrics, there’s a key element that some consultants or agencies overlook when managing a client’s SEO campaigns:

Preventing confidential client content from appearing in Google search results.

When neglected, this might result in a breach of trust or expensive litigation that can ultimately end a client relationship.

All this need not happen if you know how easily client data can enter Google’s index and how to avoid it. 

Uncover the critical search indexing issue many SEOs miss, the accidental exposure of client data on Google, and ways to deindex such content. 

How I found sensitive data

I’m a full-time independent SEO consultant who has partnered with varied midsize companies since 2018, having improved organic search results for over 10 years.

When doing a technical SEO audit, I use a site search operator (entering site:domain.com) on Google to check out results. Here, I can quickly see how site names, titles, URLs and snippets look across different page categories.

I also notice patterns of what’s getting indexed, perhaps appending keywords to the operator to get more specific when needed.

For most clients, I’ll sometimes notice dev/testing/staging sites getting indexed, thin content diluting link equity or harming crawl efficacy (or leading to keyword cannibalization) and paid landing pages that aren’t meant to rank.

I’ve started to detect, though, with alarming frequency, something unique to SaaS clients: 

Pages typically under subdomains that no one ever thinks about – in either marketing or product teams – being indexed.

The most innocuous are customer subdomains that customize their login experience (e.g., client.example.com).

Even here, a client may not wish to have their name in search results. Depending on your product, this could reveal a differentiator or vulnerability to competitors.

In far more serious cases, web-based forms with collected data (from specific people) could be found. 

In the worst cases (and with the right search query), even form fields could be accessed and changed due to a lack of password protection.

While not related to growing through organic search, I’m quick to point these out. It seemed obvious to me that much could be at stake here. 

In at least several cases, this became an “all-hands-on-deck” problem in that I was asked to get this data out of search results faster than ASAP.

One CEO mentioned that his security consultants never mentioned this possibility. This was quickly found through a basic step that most SEOs would do in an audit. 

To be fair, it nearly always requires an unusual search to find these sorts of pages.

Yet consider the odd searches that clients, maybe even your leadership team, would enter – not to mention rivals. (Never forget the enduring statistic that 15% of search queries on Google are unique!)

Even if not a legal issue, sensitive data in search results found by clients first could still harm your relationship.

Why is this data even on Google?

A single, inconspicuous link to a page from any resource accessed by search engines, anywhere on the web, is all it takes:

• Is the page listed in your XML sitemap, even if it’s not linked on your site?
• Could there have been a reference on your site in the past or something that goes unnoticed in JavaScript?
• More often than not, the client links to the page – but it’s only intended for specific people to see, like survey participants, not the general public.

Thankfully, awareness is more than half the battle here. Once you know the pages to be removed from search, you can quickly jumpstart the correction process, starting with Google.

How to quickly deindex content in Google

Find a pattern for URLs with sensitive data shown in Google search results

For example, it’s common to have a subdomain titled data.example.com that houses the web-based version of your SaaS product. You can use the site search operator to scan results pages.

Use the Page Indexing report in Google Search Console (GSC) to view all indexed URLs 

This may not show everything. Contacting your product team about this could help, as they may be able to provide everything you need more quickly and accurately.

Double-check your URLs

Confirm by using the URL Inspection tool for every URL, if possible or at least a sample, in GSC in case the links you found are no longer at those locations.

To find the offending pages, consider all URL versions that may canonicalize toward what you see in search results. 

With the canonical URL removed, the alternate versions may get indexed.

Apply the pattern (the second radio button under New Request), likely a subdomain, or list every URL by making a new request in the GSC Removals tool. 

For a limited set of pages, using the URL Inspection tool once this step is applied may speed up removal and can also confirm the latest status. This must be done one at a time. (Though not the giant that Google is, at least today, you should also do this in Microsoft Bing’s Block URL tool.)

By taking these steps, removal from Google’s index will only last six months. 

It will not prevent the issue forever or from occurring on other search engines, so you’ll need to take a final step below.

How to remove content from Google permanently

Two methods can work here:

1. Use a noindex meta robots tag in the head section of those pages

You should have your web developers add this to the page template to replicate it across all pages.

• For PDFs, images and other non-HTML content, you can add an X-Robots-Tag HTTP header with a value of either noindex/none. This is also valid for regular HTML pages but not as fast to implement.

Note: Don’t use robots.txt disallow rules (exception for images), which only work if there’s no problem in the first place. A disallow blocks crawling but not indexing.

2. Gate the content

Password-protecting your webpages or files will ensure that only authorized users can access them. This is also another way to block your content from appearing on Google.

Preventing sensitive content from appearing in search results

After taking one of these steps, you can rest assured that pages with sensitive client data will be removed and not re-enter Google’s index, with pages removed within a day, in most cases.

In good faith, you should tell your clients exactly what happened. Just remember that nothing ever goes away completely on the web.

OpenAI has published information about GPTBot, its new web crawler.

What is GPTBot. GPTBot is OpenAI’s web crawler. OpenAI uses it to crawl the web, consume knowledge for its AI features (e.g., ChatGPT) and provide AI-generated answers to questions (or prompts).

Useragent. GPTBot’s User agent token is “GPTBot”. Its full user-agent string is: “Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)”.

Robots.txt. You can use robots.txt to block GPTBot from accessing your website, or parts of it. To disallow GPTBot to access your site you can add GPTBot to your site’s robots.txt:

User-agent: GPTBot
Disallow: /

To allow GPTBot to access only parts of your site, you can add the GPTBot token to your site’s robots.txt like this:

User-agent: GPTBot
Allow: /directory-1/
Disallow: /directory-2/

GPTBot documentation. You can read the documentation on GPTBot.

GPTBot IP ranges. OpenAI also published the IP ranges that GPTBot uses. It only lists one, but I suspect they will add more over time.

Why we care. You can disallow GPTBot from crawling your site if you don’t want OpenAI using your content in any way. This is the same protocol you would use to block GoogleBot, BingBot or other web crawlers. These companies are also looking for an alternative to robots.txt for these purposes.

Dig deeper. Should you block ChatGPT’s web browser plugin from accessing your website?

Are Google’s AI-generated summaries in their Search Generative Experience (SGE) interface unfairly stealing clicks from the website publishers who originally created those answers?

This is the question everyone in SEO and digital marketing has been asking for a couple of months now. 

With Google’s latest update displaying links to SGE’s sources more prominently, now seems the best time to revisit that question.

Let’s stop talking about AI for a moment

But before we start, please do me a favor. for the next couple of sections, let’s set aside how the snippets in SGE are generated.

Why? Because I’ve noticed that too many discussions about SGE devolve into arguments about generative AI and large language models. 

It’s a worthwhile discussion to have, and we should be talking about it. But we’re missing the point by reducing our thinking about SGE to the particular algorithm that’s currently powering it.

Fixating on this particular iteration of an algorithm that Google uses to generate previews of search results distracts us from asking much more interesting questions. 

As SEOs, we should think about features in terms of the search experience itself and ask:

• What is the search page displaying?
• In what ways is the search page presenting that information?
• How do the structure and content of the search page impact those who create the content being displayed (website publishers) and those seeking answers to their search queries (users)?
• How do those impacts compare against alternative implementations of a search results page?
• Which of those implementations makes for the best search experience (whatever that might mean)?

We can’t ask any of those questions when we are getting stuck in the complexities of how Google’s generative AI model works. 

So, for a few minutes, let’s pretend it doesn’t matter how Google generates the answers displayed in SGE. 

Our discussion won’t depend on whether those answers are crafted by generative AI, a different algorithm, submitted by website owners, hand-written by a Googler stuck in some basement office, or magically cast into existence by a team of little green fairies. 

Can you do that for me? Great!

Now, let’s talk about SGE.

SGE’s new link attribution

On Tuesday, Google introduced a new design to SGE that made links more visually prominent. 

Previously, the default SGE interface didn’t show sources in an obvious way. Instead, users had to click on a button to see links explicitly within the text of that snippet:

As of Aug. 1, there is only one mode, which shows a clickable chevron at the end of each paragraph:

When you click on that chevron, you’ll see a dropdown with clickable links for webpages where SGE sourced its summary:

As you can probably tell, this new UI looks very similar to the expanded mode on SGE’s old interface. 

However, I suspect this will be a major improvement for most users. I’m probably in the minority for having regularly clicked on that tiny expansion button.

Why do we care?

Even small design tweaks to how Google displays attribution are important to anybody creating and publishing content on the web.

After all, Google’s previous iteration of SGE received a lot of public criticism. Most notably, Avram Piltch from Tom’s Hardware wrote in early June:

• “Instead of highlighting links to content from expert humans, the ‘Search Generative Experience’ (SGE) uses an AI plagiarism engine that grabs facts and snippets of text from a variety of sites, cobbles them together (often word-for-word) and passes off the work as its creation.”

Many have already praised Google’s new UI as a significant improvement. 

• SGE “is learning to give credit where it’s due,” says Android Police. 
• 9to5Google hopes this design will end up “directing more traffic through to the sources.” 

I even heard good things from one of the more vocal critics of SGE’s initial lack of attribution, Lily Ray. Ray, who serves as the Senior Director, SEO and Head of Organic Research at Amsive Digital, told me:

• “I’ve been very vocal since day one about the importance of Google incorporating links into SGE answers, just like Bing Chat has done from the start. I was excited (and relieved) today to see that Google took our concerns seriously and seems to be testing multiple layouts for incorporating links in SGE answers.” 

Slow down: Is SGE actually a ‘plagiarism engine’?

But wait. What did Tom’s Hardware mean by “plagiarism” in the quote above? 

Piltch’s description is genuinely fascinating.

Read this again, but apply every statement to Google’s pre-existing featured snippets:

• “…engine that grabs facts and snippets of text from a variety of sites, cobbles them together (often word-for-word) and passes off the work as its creation.”

The only real difference between SGE’s summaries and featured snippets is the number of links they include (and a lack of pretty colors):

You could say that the example above, like SGE, also “grabs facts and snippets of text” explicitly “word-for-word” from Untold Italy’s website.

So why is one interface treated as searching while the other as stealing?

I believe SGE includes quotes from websites functionally the same as the text displayed in featured snippets or most other rich results within Google’s SERP. 

Danielle Stout Rohe, Sr. Data Program Manager, Content and SEO at Cox Automotive Inc., agrees. As she told me: 

• “When SGE first came out, it felt like an expanded featured snippet (FS) to me. The only major difference was that the default view in SGE didn’t clearly indicate what text was tied to what site, but the other SGE view does.”

SGE feels like a natural development from previous SERP features. 

• In 2007, Google wrote about meta descriptions that they “want snippets to accurately represent the web result.”
• In 2012, they introduced rich snippets guidelines to “provide even better summaries” to users. 
• In 2014, they added “structured snippets” for additional data. 
• In 2018, Danny Sullivan announced a relaunch of featured snippets, saying:

“We display featured snippets in search when we believe this format will help people more easily discover what they’re seeking, both from the description and when they click on the link to read the page itself. It’s especially helpful for those on mobile or searching by voice.”

Most of this language tracks pretty directly across all those developments, from 2007 to the 2023 test with SGE. 

But I wasn’t there in those early days of SEO. Thankfully, I know someone who was. 

So, I asked fellow marketer and my occasional collaborator Rand Fishkin (CEO of SparkToro; formerly Moz), who told me:

• “Even in the pre-2010 days, most SEOs I talked to had the sense that the “10-blue-links” era was fading (especially after the integration of things like Google Maps, weather, sports scores, etc. between 2006-2009). I never believed much in predicting the future, but I don’t think the rise of zero-click searches and instant answers would have been a huge surprise to anyone who watched the SERPs in those years.”

We can speculate how much SGE was influenced by the recent popularity of OpenAI’s ChatGPT. But I don’t think these features are simply Google’s attempt to capture industry hype. 

SGE’s summaries are simply the newest form of “instant answers” in a long line of previous iterations. 

Why were we afraid of featured snippets?

What do we fear when we accuse a search engine like Google of “stealing”? 

One way to approach this problem is to look backward. How did we feel when featured snippets and their lengthy quotes first launched?

Featured snippets also caused a fair amount of anxiety. As Paul Shapiro, Head of Technical SEO at Shopify, explained to me:

• “When Google first introduced featured snippets, I was concerned about the potential rise of ‘zero-click’ searches, fearing that Google might retain all traffic on the SERP or within its own ecosystem.”

Shapiro’s fear makes a lot of sense. When we publish content online, we hope that users will find it. The whole premise of SEO is to help websites get found for relevant queries.

Ranking higher on the SERP is merely a way to increase a site’s exposure to more users, in the hopes that they’ll click through.

Even Google themselves agree. When describing the new link UI in an Aug. 2 update, they wrote:

 “SGE is designed to be a jumping-off point for exploring helpful information on the web, with links to search results included alongside each AI overview.”

It’s completely fair to question whether we want a search engine “stealing” our clicks or reducing our CTRs. But that question is not unique to SGE. Our concern about losing clicks to SERP previews should also apply to other content previews that were introduced in the past. 

When Google first announced rich snippets in a 2009 blog post, they described the feature as a way to increase the amount of clicks that websites would receive:

“It’s a simple change to the display of search results, yet our experiments have shown that users find the new data valuable—if they see useful and relevant information from the page, they are more likely to click through.”

Worried about losing clicks to SGE’s summaries? We also need to consider those worries’ place within the long history of search engines featuring samples of website content in other forms. 

We could argue that a search engine is taking away readers by presenting featured snippets, rich cards, meta descriptions, ranking algorithms, or the very design of the SERP.

After all, webpages ranking higher on search results get more clicks than those ranked below.

Why don’t we wonder how “fair” it is for a search engine to decide which sites are more worthy of being featured?

Do ‘zero-click’ snippets really reduce traffic?

It’s easy to look at a search snippet and think that users won’t click on your site when they can simply get their answers inside the SERP itself.

But user behavior is much more complicated than that, and some users will still look at the usual “blue links” even with SGE. 

For example, Rohe noted:

• “Other times, I still find myself scrolling the results. Old habits die hard? Or I think it is more of the fact that we still want choice if we can have it (at least I do).”

And in the years we’ve witnessed other iterations of Google embedding content directly within the SERP, our websites haven’t exactly collapsed. As Shapiro observed:

• “The impact hasn’t been as severe as I initially anticipated. Featured snippets continue to direct traffic to the originating website. While I acknowledge that Google might continue to cannibalize more traffic over time, I don’t believe that SGE represents a future of SEO that we should worry about, at least not in its current form.”

Yes, Google as a corporation has reason to keep users on their own search results page.

However, if all users abandon their search engine because it’s not helping them find relevant websites, then Google’s revenue would suffer. 

At the end of the day, they have to keep providing a decent enough user experience and provide publishers with enough visitors to make Google worth indexing their sites on.

And, as far as I can tell, the visually distinct top featured snippet actually tends to increase CTR.

Websites ranking in the top spot get 74.5% more clicks than the ones ranking second, according to Backlinko. 

In that case, wouldn’t featuring even more websites in that coveted “spot zero”, as SGE does, be even better for driving visitors to websites?

What makes a good search engine?

At the core of our unease with significant changes to the SERP is a much bigger question than their impact on CTR: what makes a search engine like Google’s “good”?

We might not know it, but we all make certain assumptions about search.

Whenever we discuss updates to SERP features, we’re wrestling with our understanding of what makes a “natural” UX when searching the web.

The main problem that Google is solving is not “how do we steal traffic from websites.”

It’s easy to hate a large corporation, but for those working on Google search, the main question is, “how can we help users determine if certain results are what they are looking for?”

From that perspective, contextual curation is extremely useful. SGE feels like features of certain academic and professional databases, such as EBSCOhost or Westlaw. 

Especially when meta descriptions and titles for multiple sites might sound similar, getting a more detailed content preview can be extremely helpful in figuring out what content to click on. 

However, in its current form, SGE is often tacked on above other rich SERP features:

Ray noticed the same concern in her experience:

• “SGE results are often highly redundant with the information we already had in other SERP features, like featured snippets, Top Stories and Google Maps. SGE often appears above or intertwined with these existing features, which can be a bit overwhelming, especially given that it takes a few seconds to load.”

I believe comparing the quality of results between SERP features is exactly the type of discussion we should have.

Many SEOs never get to it because we get stuck on a vision of scary AI coming to mess up everyone’s search experience.

But… were things even that good before?

Search wasn’t perfect – and neither was SEO

I’ll be honest – I was excited about SGE because I thought it could shake up SEO. And we need that. We need some more variety and randomness in our industry. 

Perhaps we need the field of SEO to fundamentally change. Because as things stand now, many search results have become unusable for average internet users. 

The Washington Post recently argued that Google is losing its relevance as the web’s main search engine, partly because certain website owners have manipulated their rankings too much. 

And even from our perspective, many marketers aren’t actually excited to produce SEO content. 

I’ve heard marketers say that they treat their search content as a drag, an obligation, a chore.

And others have speculated whether some SEOs might be ashamed of their own work.

We’re living with a dark legacy of SEO, having started as a field with a pretty bad reputation. As Fishkin recalled to me:

• “SEO had a terrible reputation in the business, marketing, and technology world for the first decade I practiced it. My goal was always to prove that SEO was a respectable, powerful marketing channel and a world filled with wonderful people practicing a valuable craft. For years, Google hated the SEO community and treated it as a cancer to be eradicated. It took a long time and a lot of relationship building, content marketing, and value contribution before that turned around.”

It’s hard to feel like our work is not valued by the very search engine that we’ve made into one of our key marketing channels. 

So when Google pushes out significant and disruptive changes to the SERP, we feel uncomfortable.

We don’t want the rug pulled out from under us or our clients, all of that hard-earned traffic taken away overnight.

But my question to you is – how valid is SGE, particularly, as the object of that fear? 

Yes, there are plenty of reasons to be unhappy with Google as a corporation or their particular product decisions around search. And I am not going to defend the existence of large tech monopolies.

But I also know that whether we argue about SGE or not, Google will not cease to be a gigantic corporation. Those kinds of changes are up to lawyers and regulators, not SEOs.

We are marketing professionals living in a world where Google is the main search engine – a world where most people still use Google to browse the web.

We can foam at the mouth, arguing that Google might be stealing our traffic with a slightly fancier snippet, or we can ask more interesting questions.

And perhaps some of those questions could even shape the web in a direction we would like, whether it’s Google releasing those features or not.