The script computes 3D-NME, not 3D-RSME

[patrikhuber]

In April 2019, we received an inquiry about how to correctly calculate the error value in the table 1 in the paper http://epubs.surrey.ac.uk/846089/1/2018_FG_Evaluation.pdf. It turns out unfortunately that there is a mistake and we computed 3D-NME, and not 3D-RSME.

I would recommend that papers using our evaluation to highlight that error in their paper, and compute both 3D-NME and 3D-RSME, to compare with the values in the original paper.

Below I am copying the conversation from last year, as it will be important for anyone evaluating their algorithm on this data.

---

Inquiry:

I am wondering how to correctly calculate the error value in the table 1 in the above paper.

My understanding is that you calculate 3DRMSE for each images at first, and then calculate mean and standard deviation of 3DRMSE value among all the images.

Is that correct?

---

Patrik:

For each image, we use the script on GitHub to compute all distances of that particular image and save it to the disk (e.g. as F1001_001_distances.txt).
Then for each of those files (i.e. images), we compute the mean distance (i.e. the mean error for that particular image).
This results in a list of values for a particular algorithm and image set (e.g. our algorithm on the HQ set), and these values we then simply plot or run some statistics on (e.g. mean/sdev).

Of course this "aggregates" all the distances from one particular image to one value. (It would perhaps also be interesting to see what e.g. the overall largest error-distance is, or how many distances are below or above a certain value before aggregating, etc.). The participant(s) submitted the "_distance" files to us so we could compute quite a few more interesting values, but because of time constraints we chose to stick with the approach described above for that paper.

---

Inquiry reply:

Thank you for your kind reply.

Now, I am a bit confused.

According to your script, values in the output file (e.g. F1001_001_distances.txt)
seem to represent indices and (not squared) distances.
The next process is still not clear for me.

Then for each of those files (i.e. images), we compute the mean distance (i.e. the mean error for that particular image).

Can I confirm the above process calculates mean of the distances or root mean square of the distances?

---

Patrik:

Yes, you are correct, the values in the _distance files and output of the script are "not squared distances".
I just double-checked the plotting script and it all operates on those "non squared distances". I've attached you the plotting script - apologies it might be slightly messy but it's pretty short.
In fact for the plots, we plot the mean of those distances, so the mean(sqrt((scan-predicted)^2)). It seems to me now that this is actually not correctly the RMSE but as you pointed out, the "mean of the distances".

Sorry for the possible mess.

---

Zhen-Hua:

In the code to generate the data for Table 1: It is the mean and standard deviation of the `3D-RSME' values of all the images.

But, however, as mentioned by Patrik, the script actually calculated the normalised mean distance (3D-NME), mean(sqrt((scan-predicted)^2)), rather than 3D-RSME. So, I think all the terms of `3D-RMSE' in the paper should be replaced by 3D-NME. Then it becomes clear for Table 1.

1. We first calculate the 3D-NME for each image, that is the normalised mean distance between all vertices of the predicted mesh to the ground truth surface.
2. We plot the CED curves using 3D-NME and report the mean and standard deviation of all the images in terms of 3D-NME for table 1.

---

Our apologies for the mistake, and thank you very much to our inquirer for spotting and reporting it!