Semantic mesh optimization - split by objectID and cull #521

matthewjmay · 2020-03-10T00:27:30Z

Motivation and Context

Previously, semantic meshes were loaded in as a single mesh, making them impossible to cull. This PR leverages the spatial locality of ObjectIDs to split the single mesh into multiple different meshes by object ID, calculate AAbbs, and enables culling on this semantic mesh. See below for a visualization of how this works:

Full benchmarks are currently being ran.
Some example scene A/B tests benchmarking culling on semantic only, and overall rgbd + semantic:

Scene 1:

 ================ Performance (FPS) NPROC=1 ===================================
Resolution      semantic_only                   rgbd_semantic
128 x 128       286.1 /803.2  ( 180.8%) 212.9 /440.6  ( 106.9%)
256 x 256       279.3 /763.5  ( 173.4%) 210.8 /399.2  (  89.4%)
512 x 512       257.0 /613.7  ( 138.8%) 178.9 /291.2  (  62.8%)
 ==============================================================================
 ================ Performance (FPS) NPROC=3 ===================================
Resolution      semantic_only                   rgbd_semantic
128 x 128       374.0 /1479.2 ( 295.5%) 292.2 /786.5  ( 169.1%)
256 x 256       369.6 /1439.2 ( 289.4%) 285.9 /739.8  ( 158.8%)
512 x 512       344.6 /1355.1 ( 293.2%) 270.9 /649.2  ( 139.7%)
 ==============================================================================
 ================ Performance (FPS) NPROC=5 ===================================
Resolution      semantic_only                   rgbd_semantic
128 x 128       379.3 /1575.9 ( 315.5%) 310.5 /909.9  ( 193.0%)
256 x 256       376.5 /1534.8 ( 307.6%) 305.4 /860.3  ( 181.7%)
512 x 512       376.2 /1535.6 ( 308.2%) 297.5 /802.2  ( 169.7%)
 ==============================================================================

Scene 2:

 ================ Performance (FPS) NPROC=1 ===================================
Resolution      semantic_only                   rgbd_semantic
128 x 128       1664.5/2583.4 (  55.2%) 609.7 /769.7  (  26.2%)
256 x 256       1496.5/2296.9 (  53.5%) 564.4 /627.5  (  11.2%)
512 x 512       1022.6/1395.7 (  36.5%) 375.1 /409.7  (   9.2%)
 ==============================================================================
 ================ Performance (FPS) NPROC=3 ===================================
Resolution      semantic_only                   rgbd_semantic
128 x 128       3183.8/6393.6 ( 100.8%) 1154.9/1585.3 (  37.3%)
256 x 256       2998.3/5621.4 (  87.5%) 1080.6/1441.8 (  33.4%)
512 x 512       2895.1/3816.0 (  31.8%) 917.0 /1143.8 (  24.7%)
 ==============================================================================
 ================ Performance (FPS) NPROC=5 ===================================
Resolution      semantic_only                   rgbd_semantic
128 x 128       3108.0/7136.9 ( 129.6%) 1330.7/1868.2 (  40.4%)
256 x 256       2961.8/6342.3 ( 114.1%) 1247.3/1754.1 (  40.6%)
512 x 512       3153.1/5989.5 (  90.0%) 1151.9/1564.9 (  35.9%)
 ==============================================================================

Scene 3:

 ================ Performance (FPS) NPROC=1 ===================================
Resolution      semantic_only                   rgbd_semantic
128 x 128       463.2 /1231.5 ( 165.9%) 309.2 /528.5  (  70.9%)
256 x 256       441.8 /1094.6 ( 147.8%) 290.1 /493.6  (  70.1%)
512 x 512       392.5 /863.5  ( 120.0%) 239.3 /350.6  (  46.5%)
 ==============================================================================
 ================ Performance (FPS) NPROC=3 ===================================
Resolution      semantic_only                   rgbd_semantic
128 x 128       626.2 /2256.7 ( 260.4%) 458.3 /1101.9 ( 140.4%)
256 x 256       619.1 /2135.9 ( 245.0%) 444.1 /1020.1 ( 129.7%)
512 x 512       613.0 /1870.8 ( 205.2%) 407.3 /836.6  ( 105.4%)
 ==============================================================================
 ================ Performance (FPS) NPROC=5 ===================================
Resolution      semantic_only                   rgbd_semantic
128 x 128       625.6 /2540.6 ( 306.1%) 492.4 /1310.6 ( 166.2%)
256 x 256       618.6 /2428.8 ( 292.6%) 478.1 /1244.7 ( 160.3%)
512 x 512       621.3 /2314.1 ( 272.5%) 456.4 /1145.1 ( 150.9%)
 ==============================================================================

How Has This Been Tested

Visual tests using viewer to make sure bounding boxes are calculated correctly
Benchmarks on most scenes
Existing tests pass

Types of changes

New feature (non-breaking change which adds functionality)

Checklist

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have completed my CLA (see CONTRIBUTING)
I have added tests to cover my changes.
All new and existing tests passed.

codecov · 2020-03-10T00:42:24Z

Codecov Report

Merging #521 into master will decrease coverage by 0.26%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master     #521      +/-   ##
==========================================
- Coverage   61.54%   61.28%   -0.27%     
==========================================
  Files         166      166              
  Lines        7594     7627      +33     
  Branches       84       84              
==========================================
  Hits         4674     4674              
- Misses       2920     2953      +33

Flag	Coverage Δ
#CPP	`56.24% <0.00%> (-0.44%)`	⬇️
#JavaScript	`10.00% <ø> (ø)`
#Python	`80.75% <ø> (ø)`

Impacted Files	Coverage Δ
src/esp/assets/GenericInstanceMeshData.cpp	`0.00% <0.00%> (ø)`
src/esp/assets/GenericInstanceMeshData.h	`0.00% <0.00%> (ø)`
src/esp/assets/ResourceManager.h	`100.00% <ø> (ø)`
src/esp/scene/SceneNode.h	`100.00% <ø> (ø)`
src/esp/scene/test/ReplicaSceneTest.cpp	`27.27% <0.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4d59a56...1fa27d6. Read the comment docs.

bigbike · 2020-03-10T00:46:38Z

Wow. Look at this:
256 x 256 369.6 /1439.2 ( 289.4%) 285.9 /739.8 ( 158.8%)
The speedup is almost 4x for this semantic scene!

@dhruvbatra

dhruvbatra · 2020-03-10T02:30:41Z

This is awesome! Great work @matthewjmay!

I think I know which dataset these benchmarks are on. Have you thought about also testing on MP3D and Gibson?

bigbike · 2020-03-10T02:43:17Z

I think I know which dataset these benchmarks are on. Have you thought about also testing on MP3D and Gibson?

I believe the results are from MP3D dataset.
As I advised, Matthew is now running the AB tests on all MP3D models since last Fri.

matthewjmay · 2020-03-10T03:04:47Z

@dhruvbatra @bigbike Confirming these benchmarks are on the MP3D dataset.

dhruvbatra · 2020-03-10T03:11:57Z

That's great, thank you!

mosra · 2020-03-10T19:51:19Z

This is great! Two high-level ideas before I deep-dive into the code:

Instead of populating one separate std::vector for each attribute / each object ID, what about taking the previous state (with per-vertex object ID) and sorting the index buffer in a way that each object ID is in a consecutive set of indices? That way the whole mesh could still be a single uploaded index and vertex buffer like before, but you could be drawing just parts of it -- in other words, there still being one GL::Mesh, but with multiple GL::MeshViews on it, each having a different index range. And then all views that survive culling can be put in an array and drawn in a single call using GL::MeshView::draw() (you'll need to get mosra/magnum@4f0170a that I pushed just now for this, the version currently in habitat has doesn't have this particular overload yet)
(Mutually incompatible with the above but) if each mesh has a single object ID, we wouldn't need the per-vertex object ID anymore, and thus the whole GenericInstancedShader and everything related to it could be dropped, right? Unless I missed something, of course.

matthewjmay · 2020-03-10T23:13:21Z

@mosra, That's great! I was actually thinking of something like your first point while implementing this (a simpler version where individual GL::Meshs share a GL::Buffer backing), but the MeshView should work even better. I'll incorporate it in a later PR once that version is merged in.

I thought about the single uniform object ID per drawable as well, but decided that it wasn't a good idea to constrain us to this design yet. For one, I can't toggle this feature on/off easily for A/B testing (this won't matter once this design is approved/merged), and also I didn't want to constrain us into this 1 Drawable has 1 ObjectID model, since @bigbike is working on some other scene partitioning models.

aclegg3 · 2020-03-16T20:54:13Z

Any blocks on this? Looks like it might be ready to go. @bigbike ?

…arch#521) * split semantic mesh * enable culling on semantic mesh * update test

matthewjmay added 3 commits March 4, 2020 16:31

split semantic mesh

eefbdfb

enable culling on semantic mesh

0d796b6

update test

1fa27d6

matthewjmay requested review from mosra, bigbike, aclegg3 and erikwijmans March 10, 2020 00:27

facebook-github-bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Mar 10, 2020

mosra mentioned this pull request Mar 10, 2020

csv output for ab test #523

Merged

11 tasks

bigbike approved these changes Mar 16, 2020

View reviewed changes

matthewjmay merged commit 05fb588 into master Mar 16, 2020

matthewjmay deleted the semantic-culling branch March 16, 2020 21:15

mosra mentioned this pull request Apr 3, 2020

Port to StanfordImporter and Magnum's builtin per-vertex object ID shader #564

Merged

1 task

Ram81 pushed a commit to Ram81/habitat-web-sim that referenced this pull request Dec 10, 2020

Semantic mesh optimization - split by objectID and cull (facebookrese…

fc74973

…arch#521) * split semantic mesh * enable culling on semantic mesh * update test

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Semantic mesh optimization - split by objectID and cull #521

Semantic mesh optimization - split by objectID and cull #521

matthewjmay commented Mar 10, 2020 •

edited

Loading

codecov bot commented Mar 10, 2020 •

edited

Loading

bigbike commented Mar 10, 2020

dhruvbatra commented Mar 10, 2020

bigbike commented Mar 10, 2020 •

edited

Loading

matthewjmay commented Mar 10, 2020

dhruvbatra commented Mar 10, 2020

mosra commented Mar 10, 2020

matthewjmay commented Mar 10, 2020 •

edited

Loading

aclegg3 commented Mar 16, 2020

Semantic mesh optimization - split by objectID and cull #521

Semantic mesh optimization - split by objectID and cull #521

Conversation

matthewjmay commented Mar 10, 2020 • edited Loading

Motivation and Context

How Has This Been Tested

Types of changes

Checklist

codecov bot commented Mar 10, 2020 • edited Loading

Codecov Report

bigbike commented Mar 10, 2020

dhruvbatra commented Mar 10, 2020

bigbike commented Mar 10, 2020 • edited Loading

matthewjmay commented Mar 10, 2020

dhruvbatra commented Mar 10, 2020

mosra commented Mar 10, 2020

matthewjmay commented Mar 10, 2020 • edited Loading

aclegg3 commented Mar 16, 2020

matthewjmay commented Mar 10, 2020 •

edited

Loading

codecov bot commented Mar 10, 2020 •

edited

Loading

bigbike commented Mar 10, 2020 •

edited

Loading

matthewjmay commented Mar 10, 2020 •

edited

Loading