Hi, guys, any plans to support vector clustering? It looks like this could be done pretty easy given the index architecture

Hi @Dmitry! Could you please elaborate how do you see clustering in the engine?

maybe, get k cluster centers

perform clustering on request?

yes

but clustering is very time-consuming operation

I believe the index is a tree, so that I would maybe interested to access the tree nodes

hnsw index is a graph, there is no well-defined center. Also index is separated into multiple segments, so each segment would have it's own cluster.

Also which clustering algorithm do you have in mind?

Now I am taking a tiny random subset of data, do pairwise distance and perform a hierarchical/agglomerative clustering

that a take a mean vector for each cluster to find an approximate cluster centers

what qdrant can do is to calculate distance matrix for this subset in one API call, should be pretty fast

you can use recommendation batch request with filters on subset point ids

I am taking random subset from postgres

taking random subset and calculating distances somewhere else might be slower, cause you need to transfer vectors over the network.

is there a way to get random things from qdrant?

I don't think so. What you would need to generate Ids of the subset externally

and query ids as a payload filter?

yes

smart

but how recommendation api would get me a distance matrix?

you wound need to make recommendation batch request

so it would be multiple recommendation requests in one api call

all with same filter

but different positive ID

I am following you

https://qdrant.github.io/qdrant/redoc/index.html#tag/points/operation/recommendbatchpoints

internally, we have an optimization to group requests with the same filter, so filter will be reused between request inside the batch

I will dig into it, thank you