-
Notifications
You must be signed in to change notification settings - Fork 18.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Output accuracies per class. #2935
Conversation
@@ -82,6 +90,11 @@ void AccuracyLayer<Dtype>::Forward_cpu(const vector<Blob<Dtype>*>& bottom, | |||
|
|||
// LOG(INFO) << "Accuracy: " << accuracy; | |||
top[0]->mutable_cpu_data()[0] = accuracy / count; | |||
if (top.size() > 1) { | |||
for (int i = 0; i < dim; ++i) { | |||
top[1]->mutable_cpu_data()[i] = accuracies[i] / nums[i]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What if you get 0/0 here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, fixed.
You also need to address |
int count = 0; | ||
for (int i = 0; i < outer_num_; ++i) { | ||
for (int j = 0; j < inner_num_; ++j) { | ||
const int label_value = | ||
static_cast<int>(bottom_label[i * inner_num_ + j]); | ||
++nums[label_value]; | ||
if (has_ignore_label_ && label_value == ignore_label_) { | ||
continue; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
move ++nums[label_value];
(3 lines above) after this line.
@ronghanghu I've not read read this already but has it some overlap point with #759? |
@ronghanghu Ok.. Thank you. So I leave it opened if somebody could be still interested. |
@@ -38,6 +38,10 @@ void AccuracyLayer<Dtype>::Reshape( | |||
<< "with integer values in {0, 1, ..., C-1}."; | |||
vector<int> top_shape(0); // Accuracy is a scalar; 0 axes. | |||
top[0]->Reshape(top_shape); | |||
if (top.size() > 1) { | |||
int dim = bottom[0]->count() / bottom[0]->num(); | |||
top[1]->Reshape(dim, 1, 1, 1); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A note: in order to better align with ND-Array blob and to be compatible with accuracy axis, please change these two lines into
vector<int> top_shape_per_cls(1); // Per-class accuracy is a vector; 1 axes.
top_shape_per_cls[0] = bottom[0]->shape(label_axis_);
top[1]->Reshape(top_shape_per_cls);
And also reshape your buffer blob after here to have top_shape_per_cls
. Thanks :)
Fixed comments, thanks. |
Please run |
Fixed. On Sat, Aug 22, 2015 at 11:32 PM Ronghang Hu notifications@github.com
|
: top[1]->cpu_data()[i] / nums_buffer_.cpu_data()[i]; | ||
++output_label; | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see what you mean here, but it seems confusing to shift label index in top[1]
due to ignore_label.
Better to keep original index in top[1]
and simply do
if (top.size() > 1) {
int top_count = top[1]->count();
for (int i = 0; i < top_count; ++i) {
top[1]->mutable_cpu_data()[i] =
nums_buffer_.cpu_data()[i] == 0 ? 0
: top[1]->cpu_data()[i] / nums_buffer_.cpu_data()[i];
}
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed, thanks.
Looks good to me now. Please squash into one single commit :) |
Fixed case where number of samples in class can be zero. - Fixed ignore_label case, also added a test. - Two other fixes. Fixed lint errors. Small fix.
Squashed :) |
Output accuracies per class.
Fix AccuracyLayerTest for per-class accuracy. Previously in BVLC#2935, it crashes since the test accuracy is nan (0/0) when a class never appear.
Fix AccuracyLayerTest for per-class accuracy. Previously in BVLC#2935, it crashes since the test accuracy is nan (0/0) when a class never appear.
Fix AccuracyLayerTest for per-class accuracy. Previously in BVLC#2935, it crashes since the test accuracy is nan (0/0) when a class never appear.
Fix AccuracyLayerTest for per-class accuracy. Previously in BVLC#2935, it crashes since the test accuracy is nan (0/0) when a class never appear.
Fix AccuracyLayerTest for per-class accuracy. Previously in BVLC#2935, it crashes since the test accuracy is nan (0/0) when a class never appear.
Fix AccuracyLayerTest for per-class accuracy. Previously in BVLC#2935, it crashes since the test accuracy is nan (0/0) when a class never appear.
Fix AccuracyLayerTest for per-class accuracy. Previously in BVLC#2935, it crashes since the test accuracy is nan (0/0) when a class never appear.
@rmanor Hi, rmanor! Now I use your code to output the accuracy per class. My train_val.prototxt is: layer { and I use the ImageData layer: layer { I am doing 38 animal classes classification problem. In my log file, I find something strange: The all per-class accuracy(Test net output #(1-38)) is smaller than the accuracy in the all 38 classes test data(Test net output #0: accuracy). I wonder whether there is something wrong. |
@chensiqin remember that the final accuracy also depends on how many samples you have from each class, e.g. a class can have zero accuracy, but if it has very little samples then it won't affect much the total accuracy. |
@rmanor Sorry to bother you again. My test data has 38 classes, each class has 50 test samples. Look at my log file output, all per-class accuracy is smaller than the final total accuracy, so I am wonder how can the final total accuracy is so high and it should be at least one class accuracy is higher than the final total accuracy. |
@chensiqin I guess the discrepancy comes from the behavior of the class-specific accuracy calculation, when there are zero test samples belonging to the particular class in the current batch. In that case, the class-accuracy is explicitly set to zero. Since the reported test accuracy values are averaged across the batches (test_iter), these (somewhat arbitrary) zero values skew the final results. Very misleading, indeed. |
How to use this feature in prototxt/python code? |
Does anyone know how to use this PR? what should i do after i pull this PR???please help!! |
Fix AccuracyLayerTest for per-class accuracy. Previously in BVLC#2935, it crashes since the test accuracy is nan (0/0) when a class never appear.
Hi,i wonder to know which caffe version you used for Output accuracies per class? I tried to change relevant codes you modified,but failed. could you provide all the caffe codes? Thank you! |
No description provided.