Semantic Search Explained: How AI Understands Content
One of the key discoveries in the latest phase of AI is the ability to search for and find documents based on similarity search. Similarity search is an approach that compares information based on its meaning rather than via keywords. Similarity search is also known as semantic search. The word semantic refers to the “meaning or interpretation of words, phrases, or symbols within a specific context.”
With semantic search, a user can ask a question such as “What is the movie where the protagonist climbs through 500 feet of fowl smelling s*&t?” and the AI will respond with “The Shawshank Redemption”. Performing this kind of search is impossible with keyword searching. Semantic search opens up all sorts of possibilities, whether for researchers trying to find specific information out of university collections or giving developers access to precise information when querying API documentation. The genius of semantic search is we can convert entire documents and pages of text into a representation of its meaning. The purpose of this article is to provide the fundamentals of semantic search, and the basic mathematics behind it. With a deep understanding, you can take advantage of this new technology to deliver highly useful tools to users.
The Mathematics of Vectors in AI
The key to this technology is the mathematics of vectors. In physics, a vector is defined as a magnitude plus a direction. For example, a car traveling 50 km/hr north is a vector. In Artificial Intelligence, vectors have a different meaning, and we use them to represent information in a set of data. A vector in AI can be represented graphically as shown below:
In AI, we can create vectors in 2 or 3 dimensions to represent different data points. These vectors help us determine the similarity between vectors, which is crucial for AI similarity search. The concept of vector similarity is essential in various fields like materials science, where stress vectors are compared to understand material behavior under load.
Artificial Intelligence uses arrays to represent different information in a dataset. For example, in Machine Learning to predict housing prices, we can represent each house as an array of data points. By creating vectors or arrays for each house, we can develop algorithms like housing recommendation engines to provide users with relevant information.
Converting Text into Vectors
What makes vectors in AI interesting is that the same mathematics used in Physics vectors applies to arrays of numbers. In AI, we can convert text into vectors to represent the meaning of that information. Specialized AI models can convert words, phrases, or even entire pages of text into high-dimensional vectors that capture semantic meanings and relationships between words based on their usage within the trained data.
By converting text into vectors, AI systems can determine semantic similarity between different pieces of information. This ability forms the basis of similarity and semantic search, allowing AI to provide relevant and accurate search results to users.
Vector Embeddings in AI Models
Vector embeddings are created by specialized AI models like OpenAI's text-embedding-ada-002 model. These models generate high-dimensional vectors that represent the meaning of text, enabling AI systems to perform semantic searches effectively.
OpenAI and Anthrop\c are examples of platforms that provide vector embeddings for text data. By using these embeddings, developers can leverage AI technology to enhance search capabilities and recommendation systems.
Vector Databases for Efficient Search
Specialized vector databases have emerged in recent years to efficiently handle high-dimensional vectors and compute similarity between them. These databases are optimized for similarity search and use mathematical approaches like cosine similarity to determine vector similarity.
Popular database management systems like PostgreSQL, Redis, and MongoDB have released extensions or solutions for vector search capabilities. These vector databases enable AI-powered applications to query and process large datasets effectively for tasks like semantic search, recommendation engines, and more.
Vector databases play a crucial role in enhancing AI applications by providing efficient storage and retrieval of high-dimensional vectors for various use cases.
