The Evolution of Search: From Phrase Indexing to Generative Passage Retrieval and how to optimize LLM Readability and Chunk Relevance

The landscape of online search has dramatically evolved beyond simple keyword matching, driven by the increasing need for more direct, relevant, and comprehensive answers to user queries. Traditional search engines, which historically indexed documents based on individual words, often failed to capture the nuanced meaning conveyed by phrases or the broader conceptual relationships between terms, leading to less precise search results.

Users frequently sought specific answers rather than just lists of resources, prompting a significant shift towards sophisticated systems capable of delivering concise and informative “answer passages”.

This article delves into the core innovations underpinning this revolution, including advanced context scoring, generative retrieval, phrase-based indexing, and AI-driven content simplification and organization, all designed to enhance the accuracy, relevance, and user experience of modern search.

The research for this article is based on all Passage based retrieval related patents and research paper in the database of the SEO Research Suite.

The last passages of this article are highlighting optimizing approaches to become more citaion worthy by LLMs. They are only exclusively available for SEO Research Suite members.

A short summary you can hear in the SEO Research Suite Podcast:

The Shifting Landscape of Search

The landscape of search has undergone a significant transformation, moving beyond the simple matching of keywords to a more nuanced understanding of user intent and the provision of direct, comprehensive answers.

Brief Overview of Traditional Keyword Search Limitations

Traditionally, search engines relied on indexing documents based on individual words rather than broader concepts or phrases. This approach had several limitations:

• It often failed to identify relevant documents that did not contain the exact search terms but possessed related content.
• It struggled with the dynamic nature of language, where new phrases constantly emerge, and existing ones evolve or become obsolete.
• Such systems often produced imprecise search results because they could not capture the conceptual relationships between words. For instance, a search for “Australian Shepherds” using traditional methods might return unrelated documents about Australia or general shepherds, rather than relevant content about other herding dogs like Border Collies.
• Traditional systems also typically returned mere snippets of documents, which often failed to provide comprehensive answers to complex user queries, leaving users to sift through entire documents for relevant information.

The Transition Towards Understanding User Intent and Providing Direct, Comprehensive Answers

Recognizing these limitations, search technology began to evolve, aiming to provide users with direct answers to specific questions rather than just a list of resources. This transition involved:

• Classifying queries as “answer-seeking” to determine if they sought specific information, distinguishing them from navigational or transactional queries.
• Developing processes to select and score explanatory answer passages (also known as “answer passages”) to deliver more relevant and helpful long-form answers.
• The goal shifted to providing a presentation that includes high-scoring passages as potential answers, often displayed prominently in formats like “answer boxes”.

Introduction to Phrase-Based Indexing as an Early Advancement

The concept of phrase-based indexing represents a foundational shift in information retrieval, moving beyond the limitations of traditional keyword-based search.

What is Phrase-Based Indexing?

Phrase-based indexing is an information retrieval system that indexes documents using phrases and related phrase information, rather than individual words. This system identifies phrases that predict the presence of other phrases within documents, allowing documents to be indexed based on these identified phrases. The technology analyzes the co-occurrence of words in specified positions within a text corpus to determine phrase coherence and establish phrase boundaries. It also involves using phrases to retrieve, organize, describe documents, and detect duplicates.

For example, the system determines “good phrases” based on their frequency and distinctiveness. These “good phrases” are statistically validated to predict the presence of other phrases, indicating a stronger semantic or topical relevance. This approach aims to move beyond simple collections of words or possible phrases by focusing on those that are meaningful within the document corpus.

Purpose

The primary purpose of phrase-based indexing is to enhance the accuracy and relevance of search results. Traditional search engines often fall short because they index documents based on individual words, which fails to capture the nuanced meaning and conceptual relationships conveyed by phrases. This can lead to imprecise search results; for instance, a search for “Australian Shepherds” might incorrectly return documents about Australia or general shepherds instead of related content on other herding dogs like Border Collies.

Phrase-based indexing addresses several key limitations of older systems:

• Failure to identify relevant documents that do not contain exact search terms but have related content.
• Struggles with the dynamic nature of language, where new phrases emerge, and existing ones evolve or become obsolete.
• Inability to capture conceptual relationships between words, leading to imprecise results.
• Missing the richer context provided by phrases compared to simple co-occurrence patterns of individual words.

By focusing on semantically significant phrases and their interrelationships, the system aims to provide more precise and comprehensive results, improve document classification, and enhance user interaction.

An early significant advancement in information retrieval was the move towards phrase-based indexing. This methodology aimed to overcome the shortcomings of single-word indexing by:

Core Methodology of Phrase-Based Indexing

The process fundamentally relies on two intertwined stages: Phrase Identification and Related Phrase Identification.

Phrase Identification:

The system begins by crawling documents to systematically identify potential phrases.

• Phrase Window Analysis: It uses a “phrase window,” which is a sliding window of words (e.g., 3-5 words), to traverse each document and collect possible phrases. This window helps in recognizing sequences of words that might form meaningful phrases.
• Classification as “Good” or “Bad”: The collected phrases are then classified as “good” or “bad” based on several criteria. This classification process assesses their frequency and co-occurrence statistics within the document corpus.
  • Frequency Thresholds: A phrase is initially deemed “good” if it meets certain frequency thresholds, such as appearing in more than a specified number of documents (e.g., at least 10 documents) and having a certain total number of occurrences (e.g., more than 20 times overall).
  • Distinctiveness/Interesting Instances: Additional weight is given to phrases that appear in distinctive contexts or “interesting instances,” such as in titles, bold text, or within HTML tags. These formatting markers serve as signals of a phrase’s importance.
• “Good Phrases” Characteristics: Fundamentally, “good phrases” are statistically validated to predict the presence of other phrases within documents. This indicates their stronger semantic or topical relevance. They accurately represent the content and context of documents, making them crucial for improving search efficiency and accuracy. The system prunes phrases that do not meet these predictive measures, ensuring the quality and relevance of the indexed phrases.

Related Phrase Identification

Once good phrases are identified, the system focuses on understanding their interrelationships to enhance contextual retrieval.

• Co-occurrence Matrix: The system maintains a co-occurrence matrix to track how frequently pairs of good phrases appear together within a defined contextual window in documents. This matrix is crucial for identifying relationships between phrases and understanding semantic connections.
• Information Gain (IG) Calculation: Information Gain (IG) is calculated as a predictive measure to quantify the predictive power of one phrase for another. This statistical measure assesses how much knowing the presence of one phrase (gj) helps to predict the presence of another phrase (gk), effectively reducing uncertainty about the related phrase’s occurrence.
  • Calculation Method: The IG is computed by comparing the actual co-occurrence rate of two phrases against their expected co-occurrence rate if they were unrelated. The expected rate is derived from the product of their individual occurrence probabilities across the document corpus. The formula for Information Gain is given as: PMI(q, a) = log (CR / GR), where CR is the conditional rate and GR is the global rate, reflecting how often the phrase pair occurs versus the individual occurrences.
  • Threshold for Relatedness: A high IG value (e.g., typically between 1.1 and 1.7, or >1.5) indicates a strong semantic relationship between phrases. Phrases are considered significantly related if their calculated information gain exceeds this predetermined threshold, allowing the system to filter out random co-occurrences or noise.

Role of Information Gain

The Emergence of Passage-Based Retrieval in the Era of Generative AI

In the era of generative AI, search capabilities have further advanced with the emergence of passage-based retrieval and retrieval-augmented generation (RAG). This represents a significant shift from traditional document-level retrieval:

• Passage-based retrieval systems focus on indexing individual document passages, which are then organized with keywords and annotations to enhance the ability to identify text relevant to user queries efficiently.
• These systems score candidate answer passages by determining the hierarchical path from a root heading to a subheading where the passage is located and adjusting its score based on this context.
• Generative Retrieval (GR), a newer paradigm, redefines retrieval as a sequence-to-sequence problem, using Transformer models to directly generate document identifiers (DocIDs) from queries, effectively eliminating the need for traditional external indices.
• Retrieval-Augmented Generation (RAG), on the other hand, combines an external retriever (like BM25 or dense vector search) to fetch relevant documents with a language model for final text generation. This approach often performs better at scaling to very large document collections.
• Systems like GINGER (Grounded Information Nugget-Based Generation of Responses) break down retrieved passages into “atomic information units” called “nuggets,” which are then clustered, ranked, and synthesized into coherent, accurate, and verifiable responses.
• The “thematic search” system, which generates “themes” (short descriptive phrases) from summaries of passages in top documents, further exemplifies this trend by enabling guided, drill-down exploration of search results based on common subtopics, influencing features like AI Overviews.
• Furthermore, LLM-based text simplification techniques are being used to generate minimally lossy simplified versions of complex texts, improving user comprehension and reducing cognitive load, which is critical for making information more accessible in the AI age.

Core Methodology of Passage-Based Retrieval

The process of Passage-Based Retrieval involves several sophisticated steps:

1. Query Reception and Classification
   • The system begins by receiving a user-submitted query that is identified as seeking an answer. This identification can leverage language models, machine learning, or other algorithms to recognize question queries.
   • The query is then classified as an “answer-seeking” query, and associated with a particular question type (e.g., factual, descriptive, procedural). This classification is crucial for retrieving relevant “answer types”. Answer types consist of specific elements (e.g., numerical measurements, entities, verbs, n-grams) that characterize a proper answer.
2. Resource Identification and Indexing
   • Before passages are extracted, the search system identifies and retrieves relevant resources (e.g., web pages, documents) that are responsive to the query. These resources are typically pre-indexed and stored.
   • The underlying indexing mechanism utilizes a phrase-based approach, moving beyond individual words to capture conceptual relationships. This involves:
     • Identifying “good phrases”: Phrases are classified as “good” based on their frequency (appearing in a minimum number of documents and total occurrences) and their ability to predict the presence of other phrases.
     • Determining related phrases: Relationships between phrases are established using a predictive measure like information gain, which quantifies how much knowing one phrase predicts another (e.g., “Bill Clinton” and “Monica Lewinsky” have high information gain).
     • Multi-indexed storage: Documents are stored across a primary index and a secondary index for efficiency. The primary index holds a fixed maximum number of highly relevant documents (e.g., 32,768) per phrase’s posting list, ordered by relevance scores. The secondary index stores additional documents, typically by document number, that exceed the primary’s capacity. Documents are re-ranked and re-partitioned periodically based on their relevance scores.
     • Site quality scores also contribute as an input to the search engine’s ranking system, influencing which resources are considered top-ranked.
3. Gathering Candidate Passages
   • From the identified top-ranked resources, the system extracts candidate answer passages. These are text sections typically found subordinate to headings within a document.
   • Candidate passages can be various content units, including complete sentences, individual fields from structured data sets, or paragraphs. For structured content, specific criteria are applied, such as including all steps in a list or matching key-value pairs.
4. Determining Contextual Structure
   • For each relevant resource, the system establishes a hierarchical structure of headings (e.g., H1, H2, H3).
   • A heading vector is then determined for each candidate answer passage. This vector represents the path from the root heading to the specific heading under which the answer passage is categorized. This multi-level analysis strengthens the understanding of the passage’s comprehensive context.
5. Scoring Candidate Answer Passages
   • Each candidate passage is assigned an initial answer score, which is then adjusted based on a variety of factors:
     • Query Term Match Score: Measures the similarity between terms in the user query and terms within the candidate passage.
     • Answer Term Match Score: Measures the similarity of likely answer terms (identified from top-ranked resources and weighted by frequency/IDF) to the candidate passage. This score may be reduced if the passage lacks entities matching the query’s expected entity type.
     • Query Dependent Score: A combination of the query term match score and the answer term match score.
     • Context Score: Calculated based on the heading vector. This score considers:
       • Heading Depth: Deeper headings (more levels from the root) can indicate more specific information and may be rewarded.
       • Text Similarity: How similar the text in the headings (within the heading vector) is to the user’s query.
       • Passage Coverage Ratio: How well the candidate answer covers the relevant text in its source context.
       • Additional Features: The presence of distinctive text (e.g., bolded), preceding questions, and list formats.
     • Query Independent Score: Evaluates passage quality features not directly tied to the query terms. These include:
       • Resource Scores: Such as the resource’s search ranking, reputation, and site quality score.
       • Language Model Scores: How well the passage conforms to natural language.
       • Position Scores: Based on the passage’s location in the resource (e.g., higher scores for content at the top).
       • Interrogative Score: Penalizes passages containing questions or interrogative terms to ensure declarative answers are prioritized.
       • Discourse Boundary Term Position Score: Penalizes passages starting with terms that might indicate a contrast or continuation.
6. Adjusting and Selecting the Best Passage
   • The candidate answer score is adjusted using the calculated context score.
   • Finally, the system evaluates all adjusted answer scores and selects the passage with the highest score as the ultimate answer to be presented to the user. This chosen passage is then presented, often in a direct “Answer Box” format.

This sophisticated methodology allows search engines to move beyond keyword matching to deliver precise, contextually rich, and highly relevant answers directly to user queries, significantly enhancing the search experience.

Core Mechanisms for Answer Generation

Modern search systems employ sophisticated mechanisms to identify, score, and present the most relevant answers to user queries.

Context Scoring for Answer Passages

Scoring candidate passages is a crucial process in Passage-Based Retrieval, designed to identify and select the most relevant long-form answers to user queries. This system evaluates various factors to determine the quality and appropriateness of a text section as a direct answer, distinguishing it from traditional keyword-based search.

The scoring methodology involves both query-dependent and query-independent factors, and heavily relies on understanding the contextual structure of the content.

Here are the key scoring criteria for candidate answer passages:

• Query Term Match Score This score measures the similarity between the terms in the user’s query and the terms present within the candidate answer passage. It is typically proportional to the number of matches found between the query terms and the passage terms. This ensures that the retrieved passage directly addresses the keywords used in the user’s question.
• Answer Term Match Score This score assesses the similarity of likely answer terms to the candidate passage. These “likely answer terms” are generated from top-ranked resources and are weighted based on their relevance and frequency.The calculation involves:
  1. Creating a list of terms from top-ranked resources that are expected to be part of a good answer.
  2. Assigning a weight to each term based on its occurrence in top-ranked resources and its Inverse Document Frequency (IDF) value, which reflects the term’s uniqueness and importance.
  3. Counting how many times each of these weighted terms appears in the candidate passage.
  4. Multiplying the term’s weight by its occurrence count in the passage.
  5. Combining these individual term scores to get the final answer term match score.

  A crucial aspect is that the score may be reduced if the passage lacks entities matching the query’s expected entity type. This ensures that the answer passage contains the specific types of information (e.g., numerical measurements, entities, verbs, n-grams) that characterize a proper answer for the given question type. For example, if a query asks for nutritional information, the system prioritizes passages that explicitly include relevant attribute values like “calories” or “protein”.

• Query Dependent Score This is a composite score that combines both the Query Term Match Score and the Answer Term Match Score. It provides an overall measure of how well the candidate passage directly responds to the query’s specific terms and anticipated answer elements.
• Context Score (Crucial for Long-Form Answers) The context score is vital for providing relevant long-form answers, as it evaluates the hierarchical and semantic fit of a passage within its source document. It is determined by several factors derived from the document’s structure:
  • Heading Vector: This vector represents the hierarchical path from the root heading to the specific heading under which the answer passage is categorized. It helps the system understand the broader and narrower topics surrounding the passage, thereby strengthening the overall contextual understanding. Distinctive text, like bolded sections, can even be appended to the heading vector to further refine context.
  • Heading Depth: This refers to the number of levels in the heading hierarchy from the root heading to the passage’s respective heading. Deeper headings often indicate more specific information, and thus passages found under deeper, more granular headings may receive a higher context score.
  • Heading Text Similarity to Query: The system measures how similar the text in the headings (within the heading vector) is to the user’s query. Higher similarity scores can significantly boost the context score, indicating that the passage’s organizational context closely aligns with the user’s search intent. This includes evaluating similarity at multiple levels, such as the immediate heading, its parent (penultimate), and all headings in the path. The system may use techniques like neural network distillation for efficient sentence similarity scoring to measure this.
  • Passage Coverage Ratio: This metric indicates how well the candidate answer passage covers the relevant text in its source context. A higher coverage ratio generally suggests a more comprehensive and complete answer, which can lead to improved scores.
  • Additional Features: The context score also adjusts for other features present in the passage, such as:
    • Distinctive text (e.g., bolded or italicized words).
    • The presence of preceding questions immediately before the candidate passage, which can boost its score if the passage directly answers them.
    • List formats (e.g., enumerated or bulleted lists), which can receive additional boosts, especially for “how-to” or step-modal queries, due to their clarity and structured nature.
• Query Independent Score This score evaluates aspects of the candidate passage that are not directly related to the query terms, focusing on the inherent quality and reliability of the passage and its source. These scores are crucial for ensuring the overall quality and trustworthiness of the selected answer.Key factors include:
  • Resource Ranking and Reputation: Passages from resources that rank higher in general search results or have a strong reputation score are given higher query independent scores.
  • Site Quality Score: Passages originating from sites with higher overall quality scores contribute positively to the query independent score. This site quality score can be predicted for new websites based on phrase models, mapping phrase frequencies to average site quality scores, and is used as an input to the search engine’s ranking system.
  • Language Model Score: This assesses how well the passage conforms to natural language and grammatical structures, contributing to readability and coherence.
  • Passage Position: Passages located higher up in the resource (e.g., at the top of a web page) tend to receive higher scores, as they are often more crucial or summary information.
  • Absence of Interrogative/Discourse Terms:
    • Interrogative Score: Passages that contain questions or interrogative terms are penalized. This is important because it helps ensure that the final answer is declarative, informative, and directly responsive to the user’s query, rather than posing further questions or creating ambiguity. Users expect clear and direct answers.
    • Discourse Boundary Term Position Score: Passages starting with discourse boundary terms (e.g., “however,” “conversely”) are penalized. This prevents the selection of passages that might indicate a contrast or continuation of a previous idea, which might not be directly relevant or self-contained as a direct answer.

After all these scores are computed, the system combines the query dependent and independent scores to generate an overall answer score for each candidate passage. The passage with the highest adjusted score is then selected and presented as the most relevant long-form answer, often in a direct “Answer Box” format.

Generating Elements of Answer-Seeking Queries and Answers

Beyond contextual scoring, systems classify queries to understand user intent and select answers based on predefined “answer types.”

• Query Classification as Answer-Seeking: The system receives a user query and classifies it as an answer-seeking query, identifying the specific question type it represents. This involves matching query terms against predefined question types and identifying elements like part-of-speech tags, entity instances, and root words. For example, “How to cook lasagna” would be identified as an answer-seeking query.
• Answer Types and Elements: Associated with each question type are one or more answer types, which consist of elements characterizing what constitutes a proper answer. These answer elements can include:
  • Measurement Elements: Numerical values like dates or physical measurements (e.g., “Feb. 2, 1997”, “12 inches”).
  • N-gram Elements: Sequences of words or tokens (e.g., “fuel efficiency”).
  • Verb/Preposition Elements: Identifying verbs or prepositions to understand actions and relationships.
  • Entity Instance Elements: Specific entities like “Abraham Lincoln”.
  • N-gram/Verb/Preposition Near Entity Elements: Combinations where an n-gram, verb, or preposition occurs near an entity.
  • Verb Class Elements: Instances of particular verb classes (e.g., “verb/blend” for “add,” “blend,” “combine”).
  • Skip Gram Elements: Patterns allowing for intermediary terms between words (e.g., “where * the”).
• Scoring Based on Matching Answer Types: For each candidate passage, a score is calculated based on the count of matching answer types present in the text. The more answer elements that match, the higher the score.
• Role of PMI Score: The Point-wise Mutual Information (PMI) score is used to reflect the predictive quality of a question type/answer type pair based on training data. A higher PMI score indicates a stronger correlation, reflecting how often a question type/answer type pair occurs together compared to their individual occurrences. This score helps in determining if the computed score for a passage meets a certain relevance threshold.

Weighted Answer Terms for Scoring Answer Passages

To further refine answer accuracy, systems identify and weight specific terms within potential answers.

• Identifying and Grouping Question Phrases: The process involves accessing resource data to identify question phrases within documents. These question phrases are then grouped into clusters based on similarity metrics or matching query definitions.
• Generating and Weighing Answer Terms: For each identified question phrase, a section of text immediately following it is selected as a potential answer. Terms from these selected answers are then extracted, and weights are assigned to each answer term based on predefined metrics like frequency, relevance, or inverse document frequency (IDF). These weighted terms are stored and associated with specific query definitions.
• Scoring Candidate Passages with Weighted Terms: When a user submits a question query, candidate answer passages are generated from responsive resources. The query is matched with a query definition, and the corresponding answer term vector (containing terms and their weights) is selected. Each candidate passage is then scored by comparing its terms to the selected answer term vector, using the calculated weights. The passage with the highest score is chosen as the best answer. This methodology ensures that answers are not just relevant but are also prioritized based on the importance of the terms they contain.

The Weighted answer terms for scoring answer passages patent in the SEO Research Suite

Generative Search Engines and Nuanced Retrieval

Generative Search Engines, driven by advanced Large Language Models (LLMs), represent a significant evolution in how information is retrieved and presented to users, moving beyond traditional “blue-link” search results to provide more direct, comprehensive, and contextually rich answers. This new generation of AI-powered search experiences, such as AI Overviews and AI Mode, leverages sophisticated mechanisms like Retrieval-Augmented Generation (RAG), Information Nugget-Based Generation (GINGER), and Generative Retrieval (GR) to fulfill complex user queries.

The Advent of Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG)

LLMs are at the forefront of this transformation, underpinning a new era of search that focuses on providing direct and explanatory answers rather than just lists of resources. This is evident in systems that generate thematic clusters of information for AI Overviews and AI Mode, allowing users to navigate subtopics without repeatedly refining their queries.

Retrieval-Augmented Generation (RAG) models are a crucial component, designed to enhance the capabilities of LLMs. RAG systems address some of the critical challenges in conversational AI, such as factual correctness, source attribution, and comprehensive information coverage. They operate by employing an external retriever (which can be a BM25 system, dense vector search, or dual encoders) to fetch the most relevant documents or passages. These retrieved sources are then passed to a language model for final text generation, effectively grounding the LLM’s responses in verifiable information. This external retrieval process makes RAG models particularly scalable to very large document collections and allows them to dynamically retrieve new and unseen documents, providing greater interpretability by allowing users to see the source documents that contribute to the generated answer.

Key Mechanisms Supporting Generative Search

Information Nugget-Based Generation (GINGER)

GINGER (Grounded Information Nugget-Based Generation of Responses) is a novel approach specifically designed to generate accurate and verifiable responses in retrieval-augmented generation systems. Its core innovation lies in its meticulous handling of information:

• Breaks Down Retrieved Passages into Atomic “Nuggets”: GINGER achieves high quality by deconstructing complex retrieved passages into minimal, verifiable units of information called “nuggets”. This atomic breakdown makes it significantly easier to trace statements back to their original sources, ensuring precise grounding of information.
• Nuggets are Clustered to Manage Redundancy and Enhance Coverage: Once identified, these atomic nuggets are clustered into logical groups. This clustering serves multiple vital purposes:
  • Redundancy Management: It effectively groups similar or related nuggets, identifying when the same information appears in different forms across various documents. This prevents the repetition of information in the final response.
  • Coverage Enhancement: By organizing information into different facets or aspects, clustering ensures comprehensive coverage of a topic and helps to identify any gaps in the available information. This also increases the information density of the generated output.
  • Structured Organization: The process creates logical groupings of related information, making it easier to rank the importance of different aspects and facilitating the generation of a coherent response.
• Ensures Factual Accuracy, Source Attribution, and Coherent Response Generation: The nugget-based approach significantly bolsters the trustworthiness and quality of responses. It directly helps prevent hallucinations by ensuring that every piece of information in the final response is directly supported by the source material. This facilitates easy verification of factual correctness and robust source attribution. The ultimate benefits for the final response include improved factual accuracy, better source attribution, reduced redundancy, more comprehensive coverage, and better-organized information, all while maintaining response fluency.
• Mitigates the “Lost in the Middle” Problem with Longer Contexts: A common challenge in RAG models is the “lost in the middle” problem, where performance can degrade with very long input contexts. GINGER’s multi-step, nugget-based approach effectively mitigates this issue, enabling the system to maintain effectiveness and utilize additional context without degradation even when processing more passages. In fact, responses generated from more passages (e.g., top-20 vs. top-5) generally result in higher quality, indicating efficient utilization of expanded context.

More about the GINGER Research paper in the SEO Research Suite.

Generative Retrieval (GR)

Generative Retrieval represents a paradigm shift from traditional information retrieval. Instead of relying on separate indexing and retrieval steps that map queries to document embeddings, GR directly transforms queries into document identifiers (DocIDs):

• A Sequence-to-Sequence Model Generates Document Identifiers (DocIDs) Directly from Queries, Eliminating Traditional Indices: In this approach, a single Transformer model is trained as a sequence-to-sequence model to directly generate unique document identifiers (DocIDs) from user queries. This fundamentally eliminates the need for external indices like BM25 or FAISS, streamlining the retrieval process into a single step.
• Synthetic Queries are Crucial for Training GR Models, Expanding Labeled Data, and Improving Recall, Especially at Scale: A key component enabling the scalability and effectiveness of GR models, especially for large corpora, is the use of synthetic queries. These are artificially generated queries that simulate real user search queries. They are created by training a sequence-to-sequence model (like T5 or GPT) on real query-document pairs to learn to generate potential queries a document might answer.
  • Usefulness of Synthetic Queries: They are vital because they expand labeled training data by creating query-document pairs for documents without explicit search logs. They significantly improve recall by teaching retrieval models to associate documents with a broader range of search intents. This bridging of the gap between document indexing (what the model knows about content) and document retrieval (how users actually query) ensures that all documents contribute to training, even those with no associated human-generated queries, leading to enhanced coverage and representation. Research indicates that indexing relying solely on synthetic queries can outperform those using only labeled queries by 2x-3x, highlighting their effectiveness for generating relevant document representations tailored to retrieval.
• DocID Types: The system utilizes various methods for representing DocIDs, each with its own characteristics and trade-offs regarding efficiency and scalability:
  • Naive DocIDs: Standard unique numerical or string-based identifiers.
  • Atomic DocIDs: Single-token, learned identifiers stored directly within the model. These are highly efficient for retrieval as they require only one decoding step but face scalability issues with massive corpora due to memory constraints for storing large embedding tables.
  • Semantic DocIDs (Hierarchical): These are structured identifiers based on semantic similarity, where documents are clustered, and their IDs reflect their position in a conceptual hierarchy. While they improve retrieval accuracy, they are computationally expensive to maintain at scale.
  • Learned Discrete Representations (Codebook-Based): The model learns a compressed representation using a discrete codebook, mapping documents to sequences of tokens. This is efficient as it reduces the DocID space but can have complex training and decoding inefficiency for longer DocIDs.
  • 2D Semantic DocIDs: An enhancement of Semantic DocIDs, incorporating position awareness to retain contextual meaning at each hierarchical level, leading to more robust retrieval but requiring specialized decoders and more complex training.
  • Hybrid DocID Representations: Combine traditional document identifiers with learned embeddings, balancing efficiency and accuracy and allowing for easier integration with existing retrieval pipelines.
• Comparison to RAG: GR and RAG serve different strengths and use cases:
  • GR is generally faster due to its single-step retrieval process, directly generating DocIDs. It works best for closed-domain retrieval and smaller-to-medium corpus sizes (up to 100K documents). However, GR struggles to scale beyond millions of documents due to parameter constraints and offers lower interpretability, making it unclear why specific documents are retrieved. It also requires retraining to include new or unseen documents.
  • RAG, while a two-step process (retrieve then generate) and thus potentially slower, scales significantly better to very large and dynamic document collections (e.g., entire web corpora). RAG can dynamically retrieve new and unseen documents and offers higher interpretability, as it provides transparency into which documents contribute to the generated answer. This makes RAG ideal for open-domain question answering systems and scenarios where knowledge bases are constantly evolving.

Beyond traditional keyword matching, advanced AI models are transforming how information is retrieved and presented to users.

Generative Retrieval Systems

Generative retrieval (GR) represents a paradigm shift, reframing retrieval as a sequence-to-sequence problem that directly maps queries to document identifiers (DocIDs) using a single Transformer model, eliminating the need for external indices.

• Scaling Challenges and Limitations: While GR shows promise on small datasets (~100K documents), scaling to millions of passages remains an open challenge due to computational costs and parameter management.
• The Role of Synthetic Queries: Synthetic queries are artificially generated queries that simulate real user searches, created using language models like docT5query. They are crucial for expanding labeled training data, improving recall by associating documents with a broader range of search intents, and enabling GR scaling by filling training data gaps, especially as corpus size increases.
• Document Identifier (DocID) Representation: DocIDs are unique identifiers assigned to each document. In GR, the model directly generates the DocID, bypassing traditional indexes. Various types of DocIDs are explored:
  • Naive/Atomic DocIDs: Standard unique IDs or single-token learned identifiers, efficient but with scalability and memory limitations for massive corpora.
  • Semantic DocIDs: Hierarchical identifiers encoding document meaning based on semantic clustering.
  • Learned Discrete Representations (Codebook-Based): The model learns a compressed representation using a discrete codebook, mapping documents to sequences of tokens.
  • 2D Semantic DocIDs: Enhances semantic DocIDs with position-awareness for hierarchical dependencies.
  • Hybrid DocID Representations: Combines predefined identifiers with learned embeddings for balanced efficiency and accuracy.
• Generative Retrieval vs. RAG: While GR directly generates DocIDs, Retrieval-Augmented Generation (RAG) uses an external retriever to fetch documents, which are then passed to a language model for text generation. RAG scales better to very large collections and offers more interpretability, while GR is faster for smaller, closed-domain corpora.

LLM-based Text Simplification

LLM-based text simplification is a novel approach designed to enhance user comprehension and reduce cognitive load when reading complex online content, such as biomedical articles, financial reports, or legal documents. A large-scale randomized study, involving over 4,500 participants across six domains, demonstrated a statistically significant improvement in comprehension accuracy by 3.9% overall, with gains up to 14.6% in biomedical texts, and a reduction in perceived task difficulty for simplified texts. This technique utilizes a self-refinement approach with LLMs to generate minimally lossy simplified versions of texts, ensuring accessibility without sacrificing crucial details.

The improvements in comprehension are attributed to specific “micro-edits” performed by the LLM:

• Sentence splitting tends to yield the largest comprehension gain by breaking down long, multi-clause sentences into shorter, more manageable ones, directly reducing working-memory load.
• Vocabulary substitution, which replaces rare or technical terms with common synonyms, provides a moderate boost by lowering average word difficulty.
• Clause reordering, which moves subordinate or parenthetical clauses to more natural positions, offers a smaller but measurable improvement by aligning information in a predictable sequence.

Deploying simplified versions of complex pages has measurable positive impacts on user engagement metrics:

• Pogo-sticking (rapid “back-to-search” bounce) is the most sensitive gauge of content clarity and relevance, with simplified content potentially reducing rates by 20-40%. This indicates users find what they need more quickly and don’t return to search results.
• Dwell time (time on page) tends to rise modestly, often by 5-15%, as users feel more comfortable lingering and exploring easier-to-understand content.
• Scroll depth is a noisier measure and usually shows only slight shifts unless the content is drastically reshaped.

This LLM-based approach can also be applied to generate simplified headings and subheadings. By adapting a minimally-lossy, LLM-driven simplification pipeline, headings can be made ultra-scannable while preserving target keywords. This boosts scanner-reader usability by providing shorter, punchier subheads and reducing jargon, leading to a 0.33-point ease boost on a 5-point scale. For SEO, this means strengthening semantic signals (as clear headings are a “strong signal” for understanding page topics), creating featured-snippet opportunities (especially with question-framed H2s), improving engagement metrics (lower bounce rate, higher time-on-page), and enhancing hierarchical clarity for crawlers.

Knowledge Graph Integration for Query Recommendation

This system enhances user exploration by suggesting contextually relevant queries. Unlike traditional methods that rely solely on the initial user query or historical search patterns, this system generates suggestions based on specific selected passages from search results and entries from a knowledge graph.

The suggestions are presented to the user in a non-intrusive pop-up overlay that appears near the selected passage when a user hovers over the text. This approach refines user exploration by allowing them to dive deeper into topics directly related to the content they are actively viewing.

The methodology involves several key steps:

1. A user submits a query and receives a Search Engine Results Page (SERP).
2. When the user selects a specific passage (e.g., by hovering or clicking), the system identifies the entity referenced within that passage.
3. Relevant data about this entity is retrieved from a knowledge graph, which organizes information into nodes (entities, attributes) and edges (relationships).
4. A generative model, such as a Transformer model, takes the selected passage and its corresponding knowledge graph entry as input to generate a plurality of candidate suggested queries.
5. These candidate queries are then pruned to remove irrelevant suggestions based on relevance criteria, which may involve a machine learning model and the knowledge graph data.
6. The remaining queries are ranked using user interaction data, including historical queries submitted by other users, to prioritize the most relevant suggestions.
7. Finally, the system presents the top-ranked suggested queries in a pop-up next to the original passage. If a user clicks on a suggested query, a new SERP is displayed based on that refined query.

This integration combines knowledge graph mining with generative machine learning, leading to more contextually relevant suggestions without relying on prior user query history, and conserves resources by using consolidated knowledge graph data.

More about the Microsoft patent “Knowledge attributes and passage information based interactive next query recommendation” in the SEO Reserach Suite.

Thematic Search

Thematic Search is a system designed to help users navigate overwhelming search results by automatically identifying and presenting common subtopics or “themes”. In response to a user’s query, the system not only returns a ranked list of web results but also generates concise, descriptive phrases (themes) that represent major topics shared across the top responsive documents. This process describes how AI Overviews and, in part, AI Mode, operate.

The process for generating and presenting these themes is as follows:

1. The search engine first retrieves and ranks a set of top N responsive documents (e.g., 10 to 100 URLs) for the user’s query.
2. Each of these selected documents is then parsed and split into smaller text chunks, referred to as “passages” (typically paragraphs or heading-plus-paragraph chunks).
3. A language model (summary generator) is employed to create a concise “summary description” for each individual passage. This summarization often incorporates contextual information such as the parent document’s title, neighboring passages, and metadata, ensuring the summary accurately reflects the passage’s essence within its broader context.
4. These summary descriptions are then converted into high-dimensional embeddings (vectors), allowing semantically similar passages to be located close together in vector space.
5. A clustering algorithm groups these passage-level embeddings into clusters, with each cluster corresponding to a unique “theme”.
6. To create a human-readable “theme phrase,” the system identifies the representative summary sentence within each cluster, typically the one closest to the cluster’s centroid.

Themes are then ranked and ordered for display based on several signals:

• Prominence: Measured by the number of distinct responsive documents that contributed passages to that theme’s cluster. The more unique pages discuss a theme, the higher its prominence.
• Relevance to Query: How closely the theme (or its supporting passages) aligns with the original search query.
• Aggregate Page Quality Signals: This includes traditional SEO-style ranking features of the underlying documents, such as domain authority, backlink profile, content quality (E-E-A-T measures like freshness, depth, uniqueness), and user signals (CTR, dwell time).
• Freshness/Recency: Newer content supporting a theme can rank higher even if fewer pages mention it.
• Social or Engagement Signals: User engagement measures like clicks or time spent on pages within a theme can also contribute to its ranking.

Users interact with these themes as selectable UI elements (e.g., buttons, cards). Clicking a theme can either instantly display the already-computed subset of search results organized under that theme, or the system can formulate a new, refined search query that combines the original query with the chosen theme phrase (e.g., “moving to Denver” + “neighborhoods”). This refined query then triggers a re-run of the summarization and clustering pipeline, leading to the generation of even narrower “sub-themes” (e.g., specific neighborhoods under “neighborhoods”). This iterative process allows for a layered, drill-down exploration of search results without requiring the user to manually type in successive queries.

More about the Google patent “Thematic Search” in the SEO Research Suite.

Quality and User Experience Signals

Beyond direct content relevance, search engines incorporate signals about overall content quality and user interaction.

• About the Author
• Latest Posts

About Olaf Kopp

Olaf Kopp is Co-Founder, Chief Business Development Officer (CBDO) and Head of SEO & Content at Aufgesang GmbH. He is an internationally recognized industry expert in semantic SEO, E-E-A-T, LLMO & Generative Engine Optimization (GEO), AI- and modern search engine technology, content marketing and customer journey management. As an author, Olaf Kopp writes for national and international magazines such as Search Engine Land, t3n, Website Boosting, Hubspot, Sistrix, Oncrawl, Searchmetrics, Upload … . In 2022 he was Top contributor for Search Engine Land. His blog is one of the most famous online marketing blogs in Germany. In addition, Olaf Kopp is a speaker for SEO and content marketing SMX, SERP Conf., CMCx, OMT, OMX, Campixx...

• What we can learn about Googles AI Search from the official Vertex & Cloud documentaion - 19. September 2025
• What we can learn from DOJ trial and API Leak for SEO? - 6. September 2025
• Top Generative Engine Optimization (GEO) Experts for LLMO - 3. September 2025
• From Query Refinement to Query Fan-Out: Search in times of generative AI and AI Agents - 28. July 2025
• What is MIPS (Maximum inner product search) and its impact on SEO? - 20. July 2025
• From User-First to Agent-First: Rethinking Digital Strategy in the Age of AI Agents - 18. July 2025
• The Evolution of Search: From Phrase Indexing to Generative Passage Retrieval and how to optimize LLM Readability and Chunk Relevance - 7. July 2025
• How to optimize for ChatGPT Shopping? - 1. July 2025
• LLM Readability & Chunk Relevance – The most influential factors to become citation worthy in AIOverviews, ChatGPT and AIMode - 30. June 2025
• Overview: Brand Monitoring Tools for LLMO / Generative Engine Optimization - 16. June 2025