Hi I think there is a bug in MSRA evaluation code

[mks0601]

Hi i think there is a bug in MSRA evaluation code
The groundtruth file and the result of REN files has 76375 lines while there are 76391 jpg files in MSRA dataset.
Can you check?
Also, are there no additional results of other methods on MSRA dataset? Only two results are provided.

[xinghaochen]

Hi,

The corresponding image names of the groundtruth and result for MSRA dataset can be found in here. Could you please check which images I am missing?

Actually, there are some other results reported on MSRA dataset. However, most of them do not provide predicted labels online (as far as I know) so I didn't include these results in this repo. I also notice that DeepPrior++ did provide predict labels for MSRA dataset. However, their results seem to be shuffled so I have not included these results.
Also, the predicted labels provided by DeepPrior++ seem to have 76375 lines, which is the same with REN and Pose-REN.

Maybe you can help to find out what the problems are. Thanks.

[mks0601]

can you check

P1/4/depth_499.bin
P1/L/depth_498.bin
P1/L/depth_499.bin

files? In your file list, these files are not in the list

Also, then how did you draw a graph which compares the performance with previous works on MSRA dataset?

[xinghaochen]

It's weird.
I just check my dataset, there are no such files. I look at P1/4/joint.txt, the first line is 499, which indicates that there are 499 images in this folder (000000_depth.bin ~ 000498_depth.bin).

However, I download again the dataset from the author's homepage (Xiao Sun). The P1/4/ folder has 500 images and also the first line of joint.txt is 500.

I downloaded the dataset long time ago (from the co-author Yichen Wei's old homepage). I think maybe the authors re-formatted the dataset at one time, which causes the inconsistency between the datasets that downloaded at different time.

However, as the number of inconsistent files is quite small, I believe the overall performance will be extremely similar.

For comparison, we estimated the curve from the reported figures in prior papers and plotted the graph in our paper. You may also consider sending an email to the authors to ask for the predicted labels :)

[mks0601]

I downloaded the dataset about a month ago, so that is the reason why yours and mine have different files. Thank you for kind answering. I`ll close the issue.

[mks0601]

Can you let me know how to estimate the curve from the reported figures?

[xinghaochen]

I wrote some codes to estimate values in the curve.. However, you may be interested in WebPlotDigitizer

[mks0601]

Thank you. Also, can you share the old MSRA dataset?

[xinghaochen]

I am not sure whether it's ok for me to redistribute the dataset, considering the license or relevant issues.
However, I think the 'old' and 'new ' dataset are almost the same, except for tens of frames that my dataset does not contain. You can easily use the file list I am using to get exactly the same frames with the 'old' dataset.
Hope it helps.

[mks0601]

Great helps. Thank you for your help!