Superlinked
Superlinked's vector framework gains a Qdrant connector, with one agreed interface change
Superlinked's open source vector compute framework reached its databases through one shared connector interface. Our engineers worked in its repository and conventions, agreed every Qdrant gap with its engineering lead and architect, and built a connector that matches Redis and MongoDB.
Data & AnalyticsVector database connectorBuilt from zero
- Built
- Qdrant storage and filtered search connector
- Interface
- One agreed addition: vector size
- Parity
- Modeled on the existing MongoDB connector
- Where
- Client repository, Slack and test conventions
The challenge
The framework needed Qdrant as the first of several new vector databases, but Qdrant has no schema, accepts only integer or UUID ids and needs the vector size up front.
What we did
We read the shared interface and the Redis and MongoDB connectors, put every difference to the client as a written question, then built the connector and its tests to the agreed design.
The outcome
Superlinked got a Qdrant connector built to its existing interface, with vector size added by agreement, and the work was delivered and closed with the client's agreement.
The starting point
Superlinked builds an open source vector compute framework for search and recommendations. The framework creates embeddings in its core and hands them to a storage connector, with Redis and MongoDB already supported behind a common interface. It had moved its connector code into a separate repository so outside engineers could contribute, with contributed code merged back into the framework.
What had to change
The framework needed Qdrant as the first of several planned database connectors. Qdrant models data differently from Redis and MongoDB, so a straight port would not fit.
What could not break
- The shared connector interface changes only where the client agrees
- Writing an entity that already exists overwrites it, as in the Redis connector
- Similarity search can be limited to vectors that carry a given field
- Embeddings stay in the framework core, outside the connector
- No work starts on a package before its scope is agreed
The calls we made
We read the client's abstract classes and its Redis and MongoDB connectors first, then turned every way Qdrant differs into a written question for the client's engineering lead before writing code.
- 01
Writes behave as upserts
We asked whether writing an entity should append or replace. The answer matched the Redis connector, where a write overwrites the existing key, so the Qdrant connector upserts too.
Ruled outAppend only writes, raised as the other option in our scoping questions.
- 02
Map the framework's identity to Qdrant's
The framework identifies an entity by schema id plus object id, while Qdrant accepts only integer or UUID point ids. The client confirmed the pair is the unique key, so the connector maps it to an id Qdrant accepts.
Ruled outDropping the schema id from the entity id, which we raised as a design question.
- 03
Keep Qdrant specifics inside the client class
Qdrant has no connection string. When the client approved our proposal, it asked that the extra functions live in the Qdrant client class, which keeps connection parameters as simple as in the other connectors.
Ruled outPutting Qdrant only functions in the connection parameters, as our proposal first had them.
- 04
Answer the architect's risks in the design
At the start of the first stage the client's architect flagged two risks: Qdrant cannot share metadata across vectors in one index, and its ids must be integers or UUIDs. Both went into the design questions we settled with the client.
How the system fits together
What we built
Qdrant connector
A QdrantClient class with its connection parameters, modeled on the MongoDB connector, that upserts entities under Qdrant compatible ids and reads them back by id.
Filtered vector search
A field encoder and comparison operands so nearest neighbor search can be limited to vectors that carry a given field, as the framework requires.
Index and collection management
Each framework index maps to one Qdrant collection, which the connector creates and checks before use.
Tests
Unit and end-to-end tests written with pytest, the test style the client asked for when it approved the proposal.
How it works
The framework builds embeddings
Embeddings are created in the core framework and passed to the storage connector.
The connector writes them
Entities are upserted into the matching Qdrant collection under a Qdrant compatible id.
A query arrives
The framework asks for nearest neighbors, optionally limited to vectors that carry a given field.
Qdrant answers
The connector translates the filter, runs the search and returns results in the framework's own shape.
A closer look
{
"framework_entity": {
"schema_id": "paragraph",
"object_id": "p-1042"
},
"qdrant": {
"collection": "paragraph_index",
"point_id": "3f6c1a2e-7b0d-4c55-9e1a-2d8f0b6a91c4",
"operation": "upsert"
},
"search": {
"type": "knn",
"limit": 10,
"filter": { "field": "category", "has_value": true }
}
}What changed
Superlinked got a Qdrant connector that fits its existing interface, with vector size added by agreement, built to the conventions its own engineers use. The work was delivered and closed with the client's agreement, and its engineering lead said further connector projects would be prioritized after this one.
Under the hood
- Datics role
- Product built by Datics
- Interface
- Implements the framework's common abstract connector interface, the same one the Redis and MongoDB connectors use, with vector size added to it by agreement.
- Data boundary
- The connector stores and searches vectors only; embeddings are generated by the client's core framework.
- Database
- Qdrant through its Python client, one collection per framework index, composite entity ids mapped to point ids.
Data and retrieval
Apps and APIs
Also in the stack
How it shipped
Read the reference code
With access to the client's connector repository, we studied the abstract classes and the Redis and MongoDB connectors.
Scoping questions
We sent the client's engineering lead written questions on writes, reads by id, updates and deletes, connections, collections and testing, and got written answers.
Proposal and design
A proposal and design document went to the client before code; it was approved with two changes, both adopted.
First stage
Design questions on ids, collections, filtering and vector size were answered as the build started, alongside a small proof of concept.
Close out
The connector work was closed out with the agreement of the client's CEO, and the client accepted the close out.
Inside their repository
Our engineers worked in the client's connector repository, built for outside contributions, and followed its structure, abstract classes and testing style.
Questions in their channel
Scoping and design questions went to the client's engineering lead in a shared Slack channel, and risks raised by the client's architect were answered in the design.
One package at a time
The work was estimated and agreed as a package before it started, and no work began without an agreed scope.
What this unlocked
The client planned many more database connectors on the same interface, and this one set the pattern: settle the design in writing, then code.
Related work
Have a product with the same kind of constraints?
Tell us what already exists and what you want it to do. We will tell you what it takes.