Trey Grainger

Query Understanding is one of the most important, yet usually overlooked, steps in information retrieval. Most search teams focus entirely on “algorithmic ranking”, hoping that the math (BM25, Vector Similarity, Hybrid Fusion) will magically rank the best results without first contextually understanding the queries.

In this talk, we introduce “Learning to Understand” as a corollary to the well known “Learning to Rank” process for building ranking classifiers. BM25, Dense Vector Search, and even Hybrid Search approaches all share one thing in common: they focus primarily on algorithmic ranking of search results, as opposed to query understanding for first identifying and matching the right set of potential query interpretations.

“Trey Grainger and Doug Turnbull, both authors of the recently-published AI-Powered Search book (https://aipoweredsearch.com), take the stage and attempt to answer your biggest questions about search, relevance, and the evolving landscape of AI search.

The field of information retrieval is evolving at a rapid pace, becoming a cornerstone of the generative AI space and absorbing different, yet familiar, techniques and terminology from previously tangential areas of AI world. Search engines are becoming vector databases, and vector databases (and even traditional SQL and NoSQL databases) are becoming search engines.

“Learning user context from behavioral signals is critical for optimizing search relevance, but most search engine and vector database implementations completely ignore personalization today, relying only on keywords and content embeddings.

Trey Grainger, Doug Turnbull, and Max Irwin will all be attending Haystack this year, and we will host a Q&A / Ask Me Anything on AI-powered search. The final chapter of our AI-powered search book (https://www.manning.com/books/ai-powered-search) will be released in ebook form right before Haystack (we’re just polishing off some additional on foundation models / LLMs), and the book will be coming out about a month later in print.

The search relevance landscape is rapidly shifting. Due to the rise of Transformers and Large Language Models (LLMs) and the more recent emergent capabilities shown with Foundation Models, public search engines are now rushing to integrate models like chatGPT directly into the search experience as a form of "generative search".

The first step in returning relevant search results is successfully interpreting the user’s intent. This requires combining a holistic understanding of your content, your users, and your domain.