-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
A lot of noise in semantic search #174
Comments
Hi @ironllamagirl What you describe is the same experience I have with sentence embeddings (I tried all common methods), that you get a lot of noise. You can characterize methods by false positive and false negative rates: TF-IDF / BM-25 has low false positive rate but a high false negative rate, i.e., the result if finds are often relevant, but is sadly misses a lot of relevant pairs. Sentence embeddings have the opposite characteristic: Low false negative rate, i.e, it finds all relevant pairs, but a high false positive rate, i.e., the top results often contain noise with non-relevant matches. Currently I evaluate different approach for question based semantic similarity search. I hope that I can soon (~1 month) share some data + results on this with different experiments. If computationally feasible, I think the best approach is a two step approach: For step 2 you would need some training data. For retrieval with sentence embeddings, it could also make sense to train a model with triplet loss if you have some training data. Best |
Hi @nreimers As an attempt to improve the results, I tried fine-tuning. I fine-tuned the 'bert-base-uncased' model on my custom data, using the triplet loss method. I followed the anchor, positive and negative schema. I am actually not very familiar with the BM25 method for semantic search,. Is its only objective to compute tfidf for documents? There doesn't seem to be a lot of examples online for this method. In step 1, I assume sentence embedding search is what was implemented in the semantic search example, am I right? So you suggest to append the results from both tfidf and sentence embedding search together and then do the filtering? Thanks. |
Hi @ironllamagirl https://arxiv.org/abs/1907.04780 Also a project that might be interesting for you: BM25: BM25 is similar to TF-IDF, but often works much better. It takes different document lengths into the consideration. ElasticSearch (which I can quite recommend) is based on BM25 to index and find documents.. An approach that works really good is the approach that is also implemented in Nboost: Neural re-ranking. The idea is you have two phases: A retrieval phase and a re-ranking phase. In the retrieval phase, you get for example 100 hits. You could split this in getting 50 hits with Elasticsearch (BM25) and 50 hits with semantic search using sentence embeddings, or you could just get 100 hits with BM25 from Elasticsearch. In the second step, you apply a more complex model: The re-ranker. This re-ranker gets as input (query, hit1), (query, hit2), ..., (query, hit100). For each pair, it outputs a value 0...1 about how relevant the pair is. Nboost uses for this BERT which was previously trained on suitable data. It has several pre-trained models, which should generalize quite well to other domains. The final results are then the top-10 pairs that got the highest score from the re-ranker. You can find more details on re-ranking here: Best |
Hi Nils, Thank you very much for these resources. Very helpful. Thank you again! I'm closing the issue for now. |
Just want to say that I've built a semantic search engine using this wonderful package without re-ranking. I'm not able to release the code at the moment, but wanted to share some pointers:
|
Hi Nils, any update on your evaluation/experiments on different approach for question based semantic similarity search? |
so you first use sentence transformers to similarity, then use BM25 for re-ranking? Thanks! |
Hi.
Thank you for this great package!
I am trying to use the semantic search example in order to detect sentences belonging to specific topics. I translated the different topics to query sentences to use with the semantic search.
The problem is that I am getting a lot of noise in the results. Many of the 'matching' sentences to the queries I am using have nothing to do with the query yet they still have a smaller distance than other related sentences with bigger distance. Is there a way to avoid this?
I am trying to solve this, as the use-case I am working on requires me to have as few noise sentences as possible, ideally none. I tried to play with the threshold distance but it is hard to guess which one works best.
The initial idea that I had was to apply a simple keyword search on the results of the semantic search to eliminate noise. However I am afraid that I could be losing many 'good' sentences since the list of keywords can't be completely exhaustive. I am afraid of losing good sentences that are semantically similar to the query but without containing any of the keywords I am choosing.
Another potential way to do this is to train a model so that it can tell if a sentence belongs to a certain topic or not - I haven't tried this yet. Could you please share your opinion/suggestions about this? I would really appreciate it.
Thanks!
The text was updated successfully, but these errors were encountered: