This question is not more related to qdrant , but more related to how memmap works and how like if we configure the memmap thwn how its not using the heap memory , but what memory will it use .

This ss is from my qdrant box where i had done the indexing for 700k somthing vector and my buffer/ cache is around 6.3 gb .so wanted to know about how this memmap files are stored at os level . As i would think it may be related with virtual memory or it may be using some disc space and index for the memmap is only stored in heap.
If u guys have some idea pls let me know

Hi, mmap will use the disk space that will be mapped as a virtual memory. Do you experience any troubles with that or just want to understand how things are stored on low-level?

> mapped as a virtual memory.
might be not true

mmap uses disk cache, which is managed by kernel. Disk cache might be shared between processes, so looking at free or htop is not accurate measurement. Internally, we use other technic to determine memoru requirements - we limit the max amount with docker and then let it crash

no just wanting to know more , on how mmap works at low-level ( and what size of box i should go with in prod that was my main motivation as i wanted to know how much heap memory and shared and buff memory can i take in account in my calculation ) , basically after posting this i did one experiment to check so i have 700k something vectors indexed and then size of vector is 384 . so 384 *4 *700*1000 = comes arounf 1 G or something and then i dropped the cache and still my buffer/cache only got cleaned but only till 1 G. and i think as i have vectors in m-map way it must need that cache/buffer.
and in total i would need as 952 M used + this m-map buffer/cache (1.1G) . memory as i am not gonna get back that cache.

cmiiw , as i thought this experiment was interesting . let me know if this thing make sense

i thought it uses shared and i would see shared memory usage increased . but yeah i think it uses the cache/buff component. and it basically i maybe wrong but looks like working as swap memory , so that is why seeing in the htop command after clearing the cache u can see the corret buff/cache size and the virtual memory usage which was around for me was 12 G

this max ammount u are talking about as memory limit right that we can put at docker container level? or is that something wrt to qdrant u are talking about?

with docker container

it also limits the amount of disk cache, so given the cleared cache before the experiment you can completely simulate the low-ram machine with docker

Also just did migration from 0.6.0 -> 0.11.4 standalone with mem-map as 1000kb and on_disk_payload = true
in total we have indexed 689766 (vectors,size=384) with payload .
this was the data,
heap memory : 697.1MiB
and disk space : 11G (collections folder) was this much only way better then 0.6.0 , kudos to the team

also i observed the api response time for my use-case it was okish only so with memmap, i think i can now scale my application to hopefully 10 mil vectors with clusterization , ig. As in 0.6.0 i had to upgrade my box to 32 gb to make sure everything works fine 😓

Also one more question , are u guys thinking to do something with meliesearch or in direction with the keyword based search ? basically in vector search space , what i noticed is usecase to usecase people may want both the keyword based and semantic/vector search . As i have seen both the pipeline will give results better and improved. and if i have semantic enabled and if i can re-use the vectors/payload would be a very good think.

There were discussions here - about sparse vectors and SPLADE. I think we will include it after quantization

but we are not going to implement full support for FTS like language parsing, tokenizers and all this things in the next roadmap

ok got it. will check when it comes out , as right now we have solr which does this FTS. but we were checking the meiliesearch as well , and thought u guys also had something like that , so would bring it up

now we only have full-text filtering, there are no BM25 and very basic tokenization

meiliesearch is cool, I admire their work

yeah i tested a bit looks good.

that is also rust based software, pretty cool

yes;)

as of right now our models which we use are deployed and wrapped with python fastapi apps, was thinking to do some eval as have seen like at inferencing time bert and all are outperforming the python fastapi apps. so pretty interesting space.