-
Notifications
You must be signed in to change notification settings - Fork 334
[WIP] precision@k recall@k meters (Adapted from Zlapp) #448
Conversation
@iseessel has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
@iseessel has updated the pull request. You must reimport the pull request before landing. |
|
||
|
||
@register_meter("precision_at_k_list_meter") | ||
class PrecisionAtKListMeter(ClassyMeter): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could possibly DRY this for PrecisionAtKListMeter, RecallAtKListMeter, and AccuracyListMeter, but not sure if it's worth it / might make the logic more difficult to follow.
if "accuracy" in meter.name: | ||
for top_n, accuracies in meter.value.items(): | ||
for i, acc in accuracies.items(): | ||
tag_name = f"{phase_type}/Accuracy_" f" {top_n}_Output_{i}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is Accuracy a keyword?
todo: check this and test tensorboard.
d3cde11
to
9d49c40
Compare
@iseessel has updated the pull request. You must reimport the pull request before landing. |
This pull request was exported from Phabricator. Differential Revision: D31548914 |
Summary: Pull Request resolved: facebookresearch#448 Test Plan: 1. Unit Tests 2. Objectnet eval with multiple meters. 3. Objectnet eval with multiple meters + tensorboard. 4. Checking logs are not affected. f302516569 f302378523 ``` INFO 2021-10-11 21:17:38,781 ssl_dataset.py: 224: Loading labels: manifold://ssl_framework/tree/datasets/objectnet/test_labels.npy INFO 2021-10-11 21:17:38,968 ssl_dataset.py: 182: Could not mmap without PathManager. Trying without mmap INFO 2021-10-11 21:59:32,429 trainer_main.py: 204: Meters synced INFO 2021-10-11 21:59:32,430 log_hooks.py: 525: Average test batch time (ms) for 581 batches: 4332 INFO 2021-10-11 21:59:32,430 log_hooks.py: 457: Rank: 0, name: test_accuracy_list_meter, value: {'top_1': {0: 34.3652}, 'top_5': {0: 56.9344}, 'top_10' : {0: 66.52850000000001}, 'top_50': {0: 90.24980000000001}, 'top_100': {0: 98.3687}} INFO 2021-10-11 21:59:32,430 log_hooks.py: 457: Rank: 0, name: test_precision_at_k_list_meter, value: {'top_1': {0: 34.3652}, 'top_5': {0: 11.7659}, 'top_10': {0: 6.9506}, 'top_50': {0: 1.9438}, 'top_100': {0: 1.0727}} INFO 2021-10-11 21:59:32,430 io.py: 63: Saving data to file: manifold://ssl_framework/tree/iseessel/checkpoints/f302378523 /eval_in1k_byol_reference_true_eval_resnet_8gpu_test_only.yaml_11_10_21/metrics.json INFO 2021-10-11 21:59:33,119 io.py: 89: Saved data to file: manifold://ssl_framework/tree/iseessel/checkpoints/f302378523 /eval_in1k_byol_reference_true_eval_resnet_8gpu_test_only.yaml_11_10_21/metrics.json INFO 2021-10-11 21:59:34,340 train.py: 131: All Done! INFO 2021-10-11 21:59:34,340 logger.py: 73: Shutting down loggers... INFO 2021-10-11 21:59:34,341 distributed_launcher.py: 168: All Done! INFO 2021-10-11 21:59:34,341 logger.py: 73: Shutting down loggers... ``` ``` INFO 2021-10-15 15:20:54,205 log_hooks.py: 566: Average test batch time (ms) for 2 batches: 1005 INFO 2021-10-15 15:20:54,205 log_hooks.py: 498: Rank: 0, name: test_accuracy_list_meter, value: {'top_1': {'res5': 0.0}, 'top_5': {'res5': 0.0}} INFO 2021-10-15 15:20:54,206 log_hooks.py: 498: Rank: 0, name: test_precision_at_k_list_meter, value: {'top_1': {'res5': 0.0}} INFO 2021-10-15 15:20:54,206 io.py: 63: Saving data to file: ./metrics.json INFO 2021-10-15 15:20:54,206 io.py: 89: Saved data to file: ./metrics.json ``` Reviewed By: prigoyal Differential Revision: D31548914 Pulled By: iseessel fbshipit-source-id: 7ab7b15b8ad90f0423a7dae98da5b385a1f8be6b
9d49c40
to
64c7dcf
Compare
Summary: Pull Request resolved: facebookresearch#448 Test Plan: 1. Unit Tests 2. Objectnet eval with multiple meters. 3. Objectnet eval with multiple meters + tensorboard. 4. Checking logs are not affected. f302516569 f302378523 ``` INFO 2021-10-11 21:17:38,781 ssl_dataset.py: 224: Loading labels: manifold://ssl_framework/tree/datasets/objectnet/test_labels.npy INFO 2021-10-11 21:17:38,968 ssl_dataset.py: 182: Could not mmap without PathManager. Trying without mmap INFO 2021-10-11 21:59:32,429 trainer_main.py: 204: Meters synced INFO 2021-10-11 21:59:32,430 log_hooks.py: 525: Average test batch time (ms) for 581 batches: 4332 INFO 2021-10-11 21:59:32,430 log_hooks.py: 457: Rank: 0, name: test_accuracy_list_meter, value: {'top_1': {0: 34.3652}, 'top_5': {0: 56.9344}, 'top_10' : {0: 66.52850000000001}, 'top_50': {0: 90.24980000000001}, 'top_100': {0: 98.3687}} INFO 2021-10-11 21:59:32,430 log_hooks.py: 457: Rank: 0, name: test_precision_at_k_list_meter, value: {'top_1': {0: 34.3652}, 'top_5': {0: 11.7659}, 'top_10': {0: 6.9506}, 'top_50': {0: 1.9438}, 'top_100': {0: 1.0727}} INFO 2021-10-11 21:59:32,430 io.py: 63: Saving data to file: manifold://ssl_framework/tree/iseessel/checkpoints/f302378523 /eval_in1k_byol_reference_true_eval_resnet_8gpu_test_only.yaml_11_10_21/metrics.json INFO 2021-10-11 21:59:33,119 io.py: 89: Saved data to file: manifold://ssl_framework/tree/iseessel/checkpoints/f302378523 /eval_in1k_byol_reference_true_eval_resnet_8gpu_test_only.yaml_11_10_21/metrics.json INFO 2021-10-11 21:59:34,340 train.py: 131: All Done! INFO 2021-10-11 21:59:34,340 logger.py: 73: Shutting down loggers... INFO 2021-10-11 21:59:34,341 distributed_launcher.py: 168: All Done! INFO 2021-10-11 21:59:34,341 logger.py: 73: Shutting down loggers... ``` ``` INFO 2021-10-15 15:20:54,205 log_hooks.py: 566: Average test batch time (ms) for 2 batches: 1005 INFO 2021-10-15 15:20:54,205 log_hooks.py: 498: Rank: 0, name: test_accuracy_list_meter, value: {'top_1': {'res5': 0.0}, 'top_5': {'res5': 0.0}} INFO 2021-10-15 15:20:54,206 log_hooks.py: 498: Rank: 0, name: test_precision_at_k_list_meter, value: {'top_1': {'res5': 0.0}} INFO 2021-10-15 15:20:54,206 io.py: 63: Saving data to file: ./metrics.json INFO 2021-10-15 15:20:54,206 io.py: 89: Saved data to file: ./metrics.json ``` Reviewed By: prigoyal Differential Revision: D31548914 Pulled By: iseessel fbshipit-source-id: 1bcc0cbed219ea250dc1aeaaf9df208f6acbe92d
This pull request was exported from Phabricator. Differential Revision: D31548914 |
64c7dcf
to
d817c7e
Compare
No description provided.