Prediction confidence threshold changes... why? #91
Replies: 1 comment 1 reply
-
It may be helpful to share a little about how these models work. They are trained on 3-second audio segmants containing a call. During training, the audio is presented as a training example multiple times, modified in various ways to simulate how such a call may appear when the model is asked to make a prediction. For example, random examples of noise are mixed in. The audio may be time shifted - in other words, a random period (e.g. 1 second) is snipped from the end and put at the beginning to move the position of the call event within the 3 seconds of the file. Various other methods are used to add variety to the training data. In this way, the model becomes more sensitive to the 'essence' of the call, and not fixed on particular properties of the training data. The confidence figure is really a measure of how closely the example matches a sound (or the inferred essence of that sound) it has learned during training. When you analyse a slightly different selection of audio, you will get different results. This is for three reasons, firstly, the audio is different, secondly when you analyse a selection, no list filters are applied (so you will see - for example - 'vehicle' appear when this would have been suppressd by a list filter) and finally, the results returned represent the maximum confidence for each class for a batch sized series of predictions applied to the selection. If your batch size is 32, the model will perform 32 predictions by extracting 32 3-second audio samples between the start and end of your selection. It will then sort all these results by the maximum confidence for each class known to the model. You will see up to 5 classes depending on how many had a maximum confidence above the current confidence threshold. There is no 'best confidence threshold'. The quantity, quality and diversity of available training data varies for each species, so Chirpity is able to predict some species more robustly and more accurately than others. With time you learn which species are predicted falsely in your location and what sounds tend to cause it confusion. My approach is to use a low threshold initially, and sort the results by decreasing confidence level. Below a certain level for a species, the results tend to be false positives. You can hit delete to remove selected detections one by one, or remove all species' records by right-clicking and selecting 'delete records' in the summary table on the left. I hope that's been helpful! |
Beta Was this translation helpful? Give feedback.
-
Why does Chirpity say, for instance 26% Brant and when I go to this call in the recording and select a few seconds of recording (with the goose call in it ofcourse) the confidence level changes to 32% and 22% vehicle. There is complete silence except the call.
And when I drag nearly exactly what Chirpity did even then the confidence level is completely different.
As a result I get the feeling that the results that Chirpity gives is just one of many results and I can't trust any of them. What do I do to get the best results here in The netherlands?
Beta Was this translation helpful? Give feedback.
All reactions