Evaluation on YUD

[fkluger]

Hi,

I am trying to reproduce your results on the YUD dataset, but I am unable to get it running.

In your paper, you say that you used a network trained on SU3 for the evaluation on YUD.
However, the config/yud.yaml file appears to assume a network trained on NYU instead.
I changed it to match su3.yaml instead, but now I get this error:

Traceback (most recent call last):
  File "eval_manhattan.py", line 315, in <module>
    main()
  File "eval_manhattan.py", line 255, in main
    result = model(input_dict)
  File "/home/kluger/miniconda3/envs/deepvp/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/kluger/miniconda3/envs/deepvp/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 165, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/kluger/miniconda3/envs/deepvp/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/kluger/projects/remote/deepvp/vpd/models/vanishing_net.py", line 40, in forward
    x = self.ht(x)
  File "/home/kluger/miniconda3/envs/deepvp/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/kluger/projects/remote/deepvp/vpd/models/ht/ht_cuda.py", line 53, in forward
    out = self.ht(x)
  File "/home/kluger/miniconda3/envs/deepvp/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/kluger/projects/remote/deepvp/vpd/models/ht/im2ht.py", line 116, in forward
    self.ht_size
  File "/home/kluger/projects/remote/deepvp/vpd/models/ht/im2ht.py", line 53, in forward
    ctx.ht_size[1]
RuntimeError: height_ == height && width_== width INTERNAL ASSERT FAILED at "/home/kluger/projects/remote/deepvp/vpd/models/ht/cpp_im2ht/ht_cuda.cu":38, please report a bug to PyTorch.  image shape and given shape params do not match: (%d x %d vs %d x %d).240320256256

Sounds like there is a problem due to image size mismatch between YUD and SU3, but I don't know the root cause.

Could you kindly help me get this running?

Oh and I may have found a bug in the YUD dataloader. It expects the name of the file with the VP labels to contain the value of C.io.num_nodes:

VanishingPoint_HoughTransform_GaussianSphere/vpd/datasets.py

Line 292 in 4b743cf

with np.load(iname.replace(".png", "_label_" + str(C.io.num_nodes) + ".npz"), allow_pickle=True) as npz:

Meanwhile, the pre-processing script does not include this value in the file name:

VanishingPoint_HoughTransform_GaussianSphere/dataset/yud_process.py

Line 215 in 4b743cf

npz_name = os.path.join(opt.save_dir, f'{idx:04d}_0.npz')

The dataloader then naturally throws an error because it can't find the npz file.

[yanconglin]

Hi, Fkluger,
thank you for the report. I will spend some time and figure it out. It might take some time since I have moved to another institute and previous logs no longer exist.
BTW, are you in a rush?

Yancong

[fkluger]

Thanks for taking the time! I am in a bit of a rush. If you could look into it within the next two weeks I would really appreciate it.

I have now tried using the mappings from NYU, with the following config file:

io:
  # tensorboard_port: 0
  resume_from:
  logdir: logs/
  dataset: YUD
  datadir: dataset/yud_data
  ht_mapping: parameterization/nyu/ht_240_320_403_180.npz
  sphere_mapping: parameterization/nyu/sphere_neighbors_403_180_32768.npz
  num_workers: 2
  focal_length: 1
  num_neighbors: 16
  num_vpts: 3
  num_nodes: 32768
  percentage: 1.0

model:
  batch_size: 4
  backbone: stacked_hourglass
  depth: 4
  num_stacks: 1
  num_blocks: 1
  lpos: 1.0
  lneg: 1.0
  num_channels: 64

optim:
  name: Adam
  lr: 4.0e-4
  amsgrad: True
  weight_decay: 1.0e-5
  max_epoch: 36
  lr_decay_epoch: 24

and run the evaluation script with the SU3 checkpoint:

python eval_manhattan.py -d 0 -o tmp/yud/result.npz  config/yud.yaml  pretrained_models/SU3/checkpoint_latest.pth.tar

This works, but the results are worse than in your paper:

27.51 | 56.63 | 65.17 | 73.11

I have also tried using the correct focal length (focal_length: 2.10912) for YUD but the difference is negligible:

27.71 | 56.69 | 65.21 | 73.13

Is there anything I am missing? Thanks again!

[yanconglin]

I will have to look into the details because there is some rescaling (in terms of image dimensions and focal length) when doing SU3-YUD (if I recall correctly). Say train on SU3 and then test on YUD:

1. rescale YUD from (640x480) to SU3 size (512, 512)
2. load the SU3 mappings and weights
3. pass through the network then you get predictions (from 512x512)
4. rescale those results to (640x480)
5. do the evaluation in the camera space.

I will try to find out the scripts. it seems they are not included in the initial repo. But unfortunately, I can not make any promises.

[fkluger]

Thanks, that's a good pointer. I have now tried the following:

First, resize the images before feeding them into the network:

images = torch.nn.functional.interpolate(images, size=(512, 512))

After prediction, de-normalize the VPs using SU3's focal length, then correct the aspect ratio, and finally normalize the VPs again:

x = vpt_pd[:, 0] / vpt_pd[:, 2] * 2.1875 * 512 / 2.0
y = vpt_pd[:, 1] / vpt_pd[:, 2] * 2.1875 * 512 / 2.0
x = x * 640.0/512.0
y = y * 480.0/512.0
x /= 320.0
y /= 320.0
vpt_pd = np.stack([x, y, np.ones_like(x)], axis=-1)
vpt_pd /= np.linalg.norm(vpt_pd, axis=-1, keepdims=True)

This gives me these results:

28.99 | 65.61 | 76.52 | 85.52

Even a bit better than the paper for AA@3 and AA@5, but slightly worse for AA@10

[yanconglin]

Glad you made it work. As you probably have figured out, the scaling/mapping thing is somehow resolution-dependent, which is unfortunately a drawback of this solution.

#7 (comment)
Oh, forgot to mention that: I did multiple cross-dataset tests: train/test on SU3/ScanNet/NYU and YUD. Only the NYU(train)/YUD(test) is released. That is what you see in the yud.yaml file.

[yanconglin]

feel free to reopen if you have further questions.