Output accuracies per class. #2935
Conversation
| @@ -82,6 +90,11 @@ void AccuracyLayer<Dtype>::Forward_cpu(const vector<Blob<Dtype>*>& bottom, | |||
|
|
|||
| // LOG(INFO) << "Accuracy: " << accuracy; | |||
| top[0]->mutable_cpu_data()[0] = accuracy / count; | |||
| if (top.size() > 1) { | |||
| for (int i = 0; i < dim; ++i) { | |||
| top[1]->mutable_cpu_data()[i] = accuracies[i] / nums[i]; | |||
|
You also need to address |
| int count = 0; | ||
| for (int i = 0; i < outer_num_; ++i) { | ||
| for (int j = 0; j < inner_num_; ++j) { | ||
| const int label_value = | ||
| static_cast<int>(bottom_label[i * inner_num_ + j]); | ||
| ++nums[label_value]; | ||
| if (has_ignore_label_ && label_value == ignore_label_) { | ||
| continue; | ||
| } |
There was a problem hiding this comment.
move ++nums[label_value]; (3 lines above) after this line.
|
@ronghanghu I've not read read this already but has it some overlap point with #759? |
|
@ronghanghu Ok.. Thank you. So I leave it opened if somebody could be still interested. |
| @@ -38,6 +38,10 @@ void AccuracyLayer<Dtype>::Reshape( | |||
| << "with integer values in {0, 1, ..., C-1}."; | |||
| vector<int> top_shape(0); // Accuracy is a scalar; 0 axes. | |||
| top[0]->Reshape(top_shape); | |||
| if (top.size() > 1) { | |||
| int dim = bottom[0]->count() / bottom[0]->num(); | |||
| top[1]->Reshape(dim, 1, 1, 1); | |||
There was a problem hiding this comment.
A note: in order to better align with ND-Array blob and to be compatible with accuracy axis, please change these two lines into
vector<int> top_shape_per_cls(1); // Per-class accuracy is a vector; 1 axes.
top_shape_per_cls[0] = bottom[0]->shape(label_axis_);
top[1]->Reshape(top_shape_per_cls);
And also reshape your buffer blob after here to have top_shape_per_cls. Thanks :)
|
Fixed comments, thanks. |
|
Please run |
|
Fixed. On Sat, Aug 22, 2015 at 11:32 PM Ronghang Hu notifications@github.com
|
| : top[1]->cpu_data()[i] / nums_buffer_.cpu_data()[i]; | ||
| ++output_label; | ||
| } | ||
| } |
There was a problem hiding this comment.
I see what you mean here, but it seems confusing to shift label index in top[1] due to ignore_label.
Better to keep original index in top[1] and simply do
if (top.size() > 1) {
int top_count = top[1]->count();
for (int i = 0; i < top_count; ++i) {
top[1]->mutable_cpu_data()[i] =
nums_buffer_.cpu_data()[i] == 0 ? 0
: top[1]->cpu_data()[i] / nums_buffer_.cpu_data()[i];
}
}
|
Looks good to me now. Please squash into one single commit :) |
Fixed case where number of samples in class can be zero. - Fixed ignore_label case, also added a test. - Two other fixes. Fixed lint errors. Small fix.
|
Squashed :) |
Output accuracies per class.
Fix AccuracyLayerTest for per-class accuracy. Previously in BVLC#2935, it crashes since the test accuracy is nan (0/0) when a class never appear.
Fix AccuracyLayerTest for per-class accuracy. Previously in BVLC#2935, it crashes since the test accuracy is nan (0/0) when a class never appear.
Fix AccuracyLayerTest for per-class accuracy. Previously in BVLC#2935, it crashes since the test accuracy is nan (0/0) when a class never appear.
Fix AccuracyLayerTest for per-class accuracy. Previously in BVLC#2935, it crashes since the test accuracy is nan (0/0) when a class never appear.
Fix AccuracyLayerTest for per-class accuracy. Previously in BVLC#2935, it crashes since the test accuracy is nan (0/0) when a class never appear.
Fix AccuracyLayerTest for per-class accuracy. Previously in BVLC#2935, it crashes since the test accuracy is nan (0/0) when a class never appear.
Fix AccuracyLayerTest for per-class accuracy. Previously in BVLC#2935, it crashes since the test accuracy is nan (0/0) when a class never appear.
|
@rmanor Hi, rmanor! Now I use your code to output the accuracy per class. My train_val.prototxt is: layer { and I use the ImageData layer: layer { I am doing 38 animal classes classification problem. In my log file, I find something strange: The all per-class accuracy(Test net output #(1-38)) is smaller than the accuracy in the all 38 classes test data(Test net output #0: accuracy). I wonder whether there is something wrong. |
|
@chensiqin remember that the final accuracy also depends on how many samples you have from each class, e.g. a class can have zero accuracy, but if it has very little samples then it won't affect much the total accuracy. |
|
@rmanor Sorry to bother you again. My test data has 38 classes, each class has 50 test samples. Look at my log file output, all per-class accuracy is smaller than the final total accuracy, so I am wonder how can the final total accuracy is so high and it should be at least one class accuracy is higher than the final total accuracy. |
|
@chensiqin I guess the discrepancy comes from the behavior of the class-specific accuracy calculation, when there are zero test samples belonging to the particular class in the current batch. In that case, the class-accuracy is explicitly set to zero. Since the reported test accuracy values are averaged across the batches (test_iter), these (somewhat arbitrary) zero values skew the final results. Very misleading, indeed. |
|
How to use this feature in prototxt/python code? |
|
Does anyone know how to use this PR? what should i do after i pull this PR???please help!! |
Fix AccuracyLayerTest for per-class accuracy. Previously in BVLC#2935, it crashes since the test accuracy is nan (0/0) when a class never appear.
|
Hi，i wonder to know which caffe version you used for Output accuracies per class？ I tried to change relevant codes you modified，but failed. could you provide all the caffe codes？ Thank you！ |
No description provided.