Darknet can't detect object from train images with threshold = 0.8

[xxradon]

I just noticed that I trained Yolo V2 on VOC2007 dataset,and I tested on the training images,find that yolo v2 can't fit the train images ,in other words,i set threshold = 0.8 no bounding box was detected.
I followed your instructions to train ,and checked it for several times,the result didn't change a lot.
I was wondering whether is the lose function of Yolo causing this error?By the way,I did the same test on mxnet-ssd,faster RCNN on caffe,everything was ok,the bounding boxes were detected with probability 1.0.

[AlexeyAB]

@xxradon

1. The main question is that the Yolo v2 can't increase probability more than 0.8?
2. Are there bounding boxes if you set threashold = 0.2?
3. How many iterations did you train?
4. probability = 1.0 is very similar to over-fitting. When DNN can detect objects on images which it saw with 100% probability, but can't detect the same objects on images which it didn't see. I.e. lost the ability to generalization.

[xxradon]

@AlexeyAB
1.The main question is that the Yolo v2 can't increase probability more than 0.8?
A: Actually my main question is ssd and faster RCNN can fit their training datasets very well and why yolo can not.
2.Are there bounding boxes if you set threashold = 0.2?
A: If I set threashold = 0.2,yes ,bounding boxes would be showed but with bounding location shift,and the probability would about 0.4~0.6.
3.How many iterations did you train?
A:I used the default yolo-voc.cfg ,and the parameters were also defaults,so max_batches = 45000.And I did other test on my own datasets,just 1 class,iterations =40000,it taked nearly a whole day to tarin and the avg loss fuction was less than 0.001,the result was the same ,Yolo can't detect object from train images with a high threshold.If you doubt my results,and I think you should have trained Yolo before, maybe you can test on your own train images.
2.probability = 1.0 is very similar to over-fitting. When DNN can detect objects on images which it saw with 100% probability, but can't detect the same objects on images which it didn't see. I.e. lost the ability to generalization.
A:I trained VOC2007 both on mxnet-ssd and faster RCNN with caffe ,and tested on a lot images which I got by my mobilephone,they worked very well.So I guess my trained ssd and faster RCNN were not be over-fitting.
Thanks for your answers.

[AlexeyAB]

@xxradon

Yes, already trained model shows more than 80% probability on PascalVOC2012 images: http://pjreddie.com/media/files/yolo-voc.weights

• 2007_002445.jpg - cat 97%
• 2007_002597.jpg - dog 87%, cat 86%
• 2007_003580.jpg - person 94%
• 2007_004052.jpg - aeroplane 94%
• 2007_005626.jpg - train 97%, train 74%
  ...

What command do you use to training Yolo v2?

• Do you use pre-trained convolutional weights? darknet.exe detector train data/voc.data yolo-voc.cfg darknet19_448.conv.23 http://pjreddie.com/media/files/darknet19_448.conv.23
• Do you use multi-GPU training? 1000 iteration on 1 GPU, and then on multi-GPU: https://github.com/AlexeyAB/darknet#how-to-train-with-multi-gpu

[matakk]

@AlexeyAB
I have the same problem ,and I can't detect any object use "yolo-voc_2000.weights"," yolo-voc_6000.weights" or "yolo-voc_7000.weights" ,those files is my train.

I follow the pjreddie website to train VOC dataset.
train cmd: darknet detector train cfg/voc.data cfg/yolo-voc.cfg darknet19_448.conv.23
test cmd: darknet_no_gpu.exe detect cfg/yolo-voc.cfg yolo-voc_7000.weights -thresh 0.01
cfg\voc.data:
classes= 20
train = D:/yolo/data/voc/train.txt
valid = D:/yolo/data/voc/2007_test.txt
names = data/voc.names
backup = backup/

gpu is k40.
The train console is show like this :
7678: 0.000112, 0.000137 avg, 0.001000 rate, 9.567000 seconds, 491392 images
Loaded: 0.000000 seconds
Region Avg IOU: 0.982690, Class: 0.999985, Obj: 0.988933, No Obj: 0.002251, Avg Recall: 1.000000, count: 11
Region Avg IOU: 0.979430, Class: 0.999985, Obj: 0.989214, No Obj: 0.002283, Avg Recall: 1.000000, count: 11
Region Avg IOU: 0.982894, Class: 0.999985, Obj: 0.989048, No Obj: 0.002281, Avg Recall: 1.000000, count: 11
Region Avg IOU: 0.984234, Class: 0.999985, Obj: 0.989275, No Obj: 0.002284, Avg Recall: 1.000000, count: 11
Region Avg IOU: 0.980806, Class: 0.999985, Obj: 0.988922, No Obj: 0.002268, Avg Recall: 1.000000, count: 11
Region Avg IOU: 0.990330, Class: 0.999985, Obj: 0.988951, No Obj: 0.002269, Avg Recall: 1.000000, count: 11
Region Avg IOU: 0.989400, Class: 0.999985, Obj: 0.989110, No Obj: 0.002278, Avg Recall: 1.000000, count: 11
Region Avg IOU: 0.990149, Class: 0.999985, Obj: 0.988974, No Obj: 0.002269, Avg Recall: 1.000000, count: 11
7679: 0.000254, 0.000148 avg, 0.001000 rate, 12.231000 seconds, 491456 images
Loaded: 0.000000 seconds
Region Avg IOU: 0.988764, Class: 0.999985, Obj: 0.987742, No Obj: 0.002255, Avg Recall: 1.000000, count: 11
Region Avg IOU: 0.979698, Class: 0.999986, Obj: 0.987552, No Obj: 0.002259, Avg Recall: 1.000000, count: 11
Region Avg IOU: 0.994398, Class: 0.999985, Obj: 0.989050, No Obj: 0.002268, Avg Recall: 1.000000, count: 11

I will retrain voc on linux ,But windows is my main work platform.
Thanks!!

[AlexeyAB]

@xxradon
@matakk it was bug with using rand() for batches of images in training, and now it fixed in last commit.

[xxradon]

@AlexeyAB I had tested your latest code ,the problem was indeed solved.
Your work is excelent,and thank you very much.