Reindex required when es-node is restarted

[chembohuang]

If i didn't understand it wrong, reindex required when es-node is restarted for the the hyperPlanes in AngularLshModel is regenerated.

[alexklibisz]

No, you don't need to re-index. The hyperplanes are defined as a function of a random seed that's stored in the mapping.
Have you experienced a case where you've needed to re-index?

[chembohuang]

You are right!
The hyperplanes is stored in the mapping, where we can always retrieve it from as we restart a node.
Another question is that is it necessary to cache the AngularLshModel with a loadingCache? I find it convenient to retriveve it from the mapping.

[alexklibisz]

The hyperplanes is stored in the mapping, where we can always retrieve it from as we restart a node.

The actual hyperplanes are not stored. We use a fixed seed (0), provide it to a random number generator, and then use the generator to the hyperplanes at runtme.

Another question is that is it necessary to cache the AngularLshModel with a loadingCache

The reason for using the loading cache is to avoid re-generating the hyperplanes on every search request.

I'm guessing you want to retrieve the actual hyperplanes? Right now they're not stored in the mapping. There's one thing you could try: you could use the models library to generate an LSH model. As long as you provide the same seed (0) and the same paramaters (L, k, etc), then the hyperplanes will be the same.

If you can provide some more details on what you're doing, I might be able to offer more ideas.

[chembohuang]

I never thought that the same seed (0) and the same parameters (L, k, etc) will always lead to the same result...
If so, it's no necessary to put the hyperplanes into mapping.

If you can provide some more details on what you're doing, I might be able to offer more ideas.

I am intended to leverage your talented code to speed up our recall of the product search engine.
But we only need the cosine similarity and the technology stack of our team is mainly java, i have to translate some scala code into java, e.g the plugin. The happy thing is i am almost finished here.
Thanks for your code, the recall latency has reduced from 300ms to 50ms with L=99,k=5 in the scene where we have almost 2million vectors in ES7.7. Also, i am going to perform a benchmark to see where we can improve a little bit.

Again, your work is really impressive and thanks :)

[alexklibisz]

But we only need the cosine similarity and the technology stack of our team is mainly java, i have to translate some scala code into java, e.g the plugin. The happy thing is i am almost finished here.

If you are using Elasticsearch, you can just install the plugin. You don't have to copy/rewrite any code. Installation instructions are here: http://localhost:4000/installation/

I'm happy to hear it's helping. I forgot to mention there is more info about how hyperplanes and models in general are stored here: https://elastiknn.com/api/#nice-to-know

[alexklibisz]

I'll go ahead and close this since the original issue/question is resolved. Feel free to open another issue or post on the Gitter if you have more questions: https://gitter.im/elastiknn/community