Logo-amall

*Description:* Right around 590 upserts (~35 min in) I get a `service unavailable (503)` or a `bad gateway (502)` response. It starts intermittent, then it just becomes persistent. Wondering if I am triggering something in the load balancer. Is there recommended way of upserting? I’m doing it in ~1000 batches for 1.7 million embeddings so ~1700 upserts. Qdrant cloud dashboard says my cluster is `DOWN`. It goes back to healthy once I kill the upset script and give it a break from upserting. *Some specs:* **Embeddings size**: 384 (`all-MiniLM-L12-v2`) **Cloud instance specs**: 4GB RAM, 1vCPU, 16GB Disk **Hosting info**: AWS - US East *Other Info:* I am behind a VPN to access the data I am embedding, but I wouldn’t think that’d be an issue.

Last active 18 days ago

12 replies

0 views

  • NA

    Description:
    Right around 590 upserts (~35 min in) I get a service unavailable (503) or a bad gateway (502) response.
    It starts intermittent, then it just becomes persistent. Wondering if I am triggering something in the load balancer. Is there recommended way of upserting? I’m doing it in ~1000 batches for 1.7 million embeddings so ~1700 upserts. Qdrant cloud dashboard says my cluster is DOWN. It goes back to healthy once I kill the upset script and give it a break from upserting.

    Some specs:
    Embeddings size: 384 (all-MiniLM-L12-v2)
    Cloud instance specs: 4GB RAM, 1vCPU, 16GB Disk
    Hosting info: AWS - US East

    Other Info:
    I am behind a VPN to access the data I am embedding, but I wouldn’t think that’d be an issue.

  • NA

    I havnt figured out a pattern between 502 and 503

  • AN

    Our cloud engineers will take a look tomorrow. However, from my rough calculation you might hitting the memory limit of your setup. for 1.7M vectors of 384dim you will need around 4GB, plus the RAM used for the system. And the behaviour you've described falls in the pattern.

  • NA

    Okay. Sounds good thank you for the advice. I'll try upping the RAM and seeing it it resolves things

  • AN

    Also, you can try to use memmap option, in this case instead of permanent failing the requests will just get slower. Depends on your performance needs of cause.

  • FA

    Hi @nathan.l we will take a look and come back to you with a suggestion. But it looks like an out of memory, your instance restarts a lot, I'd agree with Andre that upping Ram and/or memmap on the collection will do the job. Depending on your use case and data you can further optimise on memory usage by using payload_on_disk: true

  • FA

    I see that you already created a new instance with 8GB of RAM, please let me know if that works now, at least I don't see frequent restarts any more

  • NA

    Increasing the RAM definitely fixed things, thanks so much guys. Out of curiosity… I used the formula:

    memory_size = number_of_vectors * vector_dimension * 4 bytes * 1.5
    

    So, 1.7e6 * 384 * 4e-9 GB * 1.5 ~ 4GB. Is the calculation off? Or its just too rough an estimate?

  • FA

    We usually use * 2 and that is of course also an estimate. There are also other factors that play a role, like payload that you attach

  • FA

    Happy to hear that fixed your issues

  • NA

    Ah, my payloads were large. So that makes sense. Good call. Appreciate all the help

  • FA

    🙂 makes sense, so also payload_on_disk: true will help a lot

Last active 18 days ago

12 replies

0 views