That is going to be extremely time consuming for me to create. So I'm trying to avoid that. I understand that isn't your concern though.

Let's move this discussion to a thread.

@andrey.vasnetsov fyi

Thank you. Sorry for making so much noise.

I'm just really excited about this solution. The results I'm getting back are GREAT!

Just need to figure out this issue with the indexing.

No worries. We would like to help it is just not simple without a complete view.

Completely understand.

does it make any difference if you upload with single vs multi thread?

Missing entries issue

I'm not sure how I would change that.

Right now I'm just doing an upsert.

client.upsert(collection_name=request.collection, points=[PointStruct(id=int(request.meta.get('id')), payload=request.meta, vector=vector.tolist())])

I mean it would be ideal to find a single request which has x points, but only y

also I would double-check that ids are not overlapping

I built a script that goes over all of the records and performs a client.retrieve for the specific ID. If a record is not found then I print a log.

It logs all the missing entries correctly.

If I then attempt to add those they get inserted into the collection correctly.

but the initial update request is different?

Exact same request payload.

The missing entries are different each time, right? Not the same vectors.

I'm not sure on that.

As the results are not consistent.

Sometimes 10k missing records. Sometimes 5k.

I use this exact same script and send more or less the exact same record to a different image match service I have that has an API using Flask and backend of ElasticSearch and it works. So that's the strange part.

I think that I might have identified part of the problem. I switched this over to flask and I'm able to see the threads spinning up in my debug. For some reason it is creating a new thread for each request.

I'm assuming at some point this is just getting overloaded.

flask?

Yes.

do you run it with uvicorn ?

This is just running via the VSCode Debug.

        {"name":"Python: Flask","type":"python","request":"launch","module":"flask","env":{"FLASK_APP":"api.py","FLASK_DEBUG":"1"},"args":["run","--no-debugger","--no-reload"],"jinja":true,"justMyCode":true},

Seems like it is cleaning up the threads though.

uvicorn have setting for required number of workers https://www.uvicorn.org/settings/

Yeah. I've adjusted those with the FastAPI.

If I leave the ID field out of the upsert request will it generate a random ID?

no

if you need random id, I would suggest to use random UUID generator

Hmm.

        id = int(uuid.uuid1())
        client.upsert(collection_name=data.get('collection'), points=[PointStruct(id=id, payload=data.get('meta'), vector=vector.tolist())])

no, that won't work.

str(uuid.uuid4()) - I usually use this

uuid is 128 bit, it won't fit the uint64

Ok… hopefully this is the issue and the ids I was passing maybe were just getting stored wrong.

you mean the request.meta.get('id') is actually a uuid?

Actually…. now that I'm thinking about it. I think this might have been the issue.

I think I was passing in an integer and then converting it to an int

Which I'm assuming could cause issues.

🫢 Well this is embarrassing.

Looking promising. Only 30 records off through 21k records.

Missing entries issue (solved)