Vector search in BigQuery
BigQuery vector search is now generally accessible (GA), according to Google Cloud, enabling users to look for vector similarities on BigQuery data. A variety of novel data and AI use cases, such as retrieval-augmented generation (RAG) employing a large language model (LLM), similarity detection, and semantic search, are made possible by this feature, also known as approximate nearest-neighbor search.A serverless and integrated vector-analytics solution for use cases like anomaly detection, multi-modal search, product recommendations, drug discovery, and more is provided by BigQuery vector search, which was first unveiled in February. It accomplishes this by incorporating embedding creation, administration, and search into the data platform.
Moreover, the BigQuery inverted file index (IVF) for vector search is now available to the general public. Utilizing a k-means clustering technique and an inverted row locator, this index generates a two-part index that efficiently looks for comparable embedding representations of your data. The following new advancements in IVF have been made since it was first shown in preview:
Improved scalability: Large-scale applications are now possible due to the ability to index 10 billion embeddings.
Managed index with guaranteed accuracy: If the underlying data changes, vector indexes are automatically updated with the latest version of the k-means model. Based on the most current mutations of the data, vector search consistently produces accurate results even before the system has finished re-indexing the new data.
Stored columns: You can now store frequently used columns in the index to reduce the cost of expensive joins when acquiring more data for the search result. When you require a high top_k or your query data consists of a huge batch of embeddings, these are high result-set cardinality scenarios where this optimization shows the most noticeable speed advantages. given instance, given a table holding one billion 96-dimensional embeddings, using vector indexes with stored columns yields the 1,000 most comparable candidates for an embedding ~4x faster with ~200x less slots than using vector indexes without saved columns.
Pre-filters: Vector search results can be pre-filtered in addition to stored columns by converting the base table statement into a query with filters. While post-filtering adds WHERE clauses after the VECTOR_SEARCH() function, pre-filtering improves search quality, lowers the possibility of missing results, and maximizes query performance.
BigQuery vector search has been utilized by Palo Alto Networks and other clients to find similar, commonly asked questions, which has accelerated the time to insight.
Moreover, prototyping with BigQuery vector search and going to production is simple, even for large-scale workloads like drug discovery, which Vilya has been working on. Furthermore, capacity-based billing models have proven simple to switch to thanks to tools for on-demand pricing and budget review.
Using a model to construct
Suppose you would like to post a question in an internal Q&A forum, but you would also like to check if any questions that are semantically similar to yours have already been answered. For the purpose of illustration, it will presume that it has already generated embeddings for the questions and added them to the database. Once that's done, you can create a vector index and store the commonly used columns (such title, content, and tags) in it for maximum efficiency.RELATING TO
STORING (tags, content, and title)
OPTIONS (index_type='IVF', distance_type='COSINE')
VECTOR_SEARCH() works even in the absence of a vector index, but creating an index typically enhances query performance. When everything is prepared, you may search for topics like "Android app using RSS crashing" among others by using ML.GENERATE_EMBEDDING in conjunction with VECTOR_SEARCH(). To more successfully hone the results, you can apply a pre-filter in the "tags" column to reduce the size of the search space.
SELECT distance, base.content, base.title, and query.query
DIRECTLY FROM VECTOR_SEARCH
(SELECT * FROM THE SEARCH WHERE (tags, "android")),
"embedding"
(
SELECT content AS query, ml_generate_embedding_result AS embedding
ML.GENERATE_EMBEDDING FROM
TYPE,
(SELECT AS content, "Android app using RSS crashing").
)
))
It also recently disclosed a new index type that can improve search performance even more. It is built on Google's ScaNN and is accessible in preview. A multi-modal Retrieval Augmentation Generation (RAG) solution, powered by state-of-the-art Gemini models and built on top of a fully functional BigQuery knowledge base containing multimodal, structured, and unstructured data, is increasingly relying on BigQuery vector search.
0 Comments