Logo-amall

https://www.pinecone.io/learn/hybrid-search/ any chance we could get smth similar to this in qDrant? Would be hugely valuable for semantic search use cases, they really nailed it

Last active 10 days ago

39 replies

0 views

  • MO

    https://www.pinecone.io/learn/hybrid-search/ any chance we could get smth similar to this in qDrant? Would be hugely valuable for semantic search use cases, they really nailed it

  • AN

    why not just use a specialized solution for both? TBH, I don't see many benefits implementing mediocre full-text search in vector engine

  • MO

    ig a little difficult to combine the two methods no? would likely have to do some sort of filtered search with the full-text and then pass a list of IDs to qDrant for scoring? or do two parallel searches across the two engines and combine the results somehow?

  • AN

    if you only need filtering with full text, we have this. But merging of results from different engines - is not as trivial as just setting up a coefficient, unfortunately

  • MO

    yeah haha for sure

  • MO

    anyway, was just asking to see if it was on the roadmap / suggest it β€” smaller queries (I.e. navigational queries) tend to get much better results using more traditional full-text methods, then as query gets longer, vector search becomes more and more important

  • AN

    Our deverel is actually working on a POC combining Qdrant and Meilisearch for a hybrid solution. It will be published as soon as we have the results.

  • MO

    Ahh nice that’d be cool to see, I wonder if mellisearch supports custom term frequencies / weightings (I.e. SPLADE-based), they kick ass

  • TE

    If I may pitch in, Pinecone above doesn't seem to be implementing full-text search. Instead they seem to allow you to upload sparse vectors in addition to dense vectors and combining scores with a simple linear combination.

  • TE

    @andrey.vasnetsov we had a chat about this before (exposing the underlying inverted index instead of text search functionality) and I think this is a pretty good example of a use case.

  • TE

    In the case of text embeddings, if you naively take an SBERT model and apply it to something other than MSMARCO, you'll often find that vector search will actually underperform keyword search, especially in more obscure domains.

  • TE

    That's why, for semantic search in particular, you get decent results by combining BM25 (or equivalent) and vector search.

  • TE

    Although I haven't used Pinecone's solution, there's a graph on that page that illustrates the difference.

  • MO

    could be a pretty clean impl if y’all decide to go through with it

  • TE

    It roughly matches my own experience, although we use very different models and a different scoring function.

  • TE

    @Morgan My understanding (though I could be very wrong) is that qdrant has already implemented the fundamental building blocks for this because they've built (a form of) text search. Maybe a member of the team can confirm or deny.

  • MO

    Pretty sure the full-text is only for filtering, not included in ranking whatsoever β€” may also be wrong here

  • AN

    yes, currently full-text search is for filtering only

  • TE

    Yes, but presumably this required implementing an inverted index, which seems like the hard part of such a feature.

  • TE

    Though obviously this all is not exposed to the end-user.

  • AN

    the tricky part with sparse vectors is actually how are you going to obtain them. Pinecone proposes to use Spacy to do tokenization and other heavy lifting, which is actually usually done by full-text search engines themselves. On its own it is already quite limiting. If there are other, more advanced ways to get sparse vectors, it might make sense to support them in qdrant as well. But I don't think it is optimal to decouple sparse vector search from the text processing

  • TE

    @Morgan above mentioned SPLADE (I think I sent you the links in the past too) - this is a fairly effective method of creating sparse embeddings that perform reasonably well.

  • MO

    we use splade in prod (example usage: https://github.com/naver/splade/blob/main/inference_splade.ipynb), fixed size vocabulary + term weightings (i.e. hell -> 0.4, qdr -> 2.1)

  • TE

    It's not just about BM25, sparse embeddings in ML is currently a pretty big area of research.

  • TE

    In part because of the limitations of dense embeddings in out-of-domain use cases

  • TE

    Unrelated: interesting work you're doing at Operand @Morgan

  • MO

    model def. not something that should be baked into qdrant itself (i.e. qdrant likely shouldn't be picky when it comes to what model to use for this, splade is the current best one used by all the legit search companies, and new ones come out super often) (also we use GPUs for inference here), but am asking to consider it on the roadmap for qdrant, pinecone did really nail the impl here

  • AN

    we are preparing the next roadmap now, I think we would be able to fit this feature as well

  • MO

    ty!! obv. no pressure, sorry for pushing this a lil hard, would be super valuable for us + hopefully lots of other companies like us πŸ™‚

  • SA

    This is fantastic.

  • SW

    hello, I'm checking qdrant FTS and found this discussion about full-text-search in this channel

    > Pretty sure the full-text is only for filtering, not included in ranking whatsoever β€” may also be wrong here

    Does this mean FTS filter is applied after performing vector search?

  • AN

    No, query planned could decide whether it makes sense to pre-filter vectors with FTS or apply filters during the HNSW graph traversal

  • AN

    we do not do post-filtering as it is very likely to just loose relevant results

  • SW

    that's great, thank you

  • SA

    Any update on this?

  • SA

    It would be really useful.

  • AN

    contributions are welcome 😁

  • SA

    Haha, have to learn Rust properly, first.

  • AN

    FYI. We added hybrid search (via sparse vectors) to the upcoming roadmap.

Last active 10 days ago

39 replies

0 views