Higher mesh coverage pose extraction for ImageExtractor #516
Conversation
… pytorch model and tests for that as well as the topdown view
… getTopdownView to return Eigen Matrix
mpiseno
changed the title
|
Can you please post pictures of the kinds of frames you are able to extract? Also (just curious, don't know the right answer) -- would this be better captured as a new camera model in habitat-sim RGB sensor? |
Sure no problem! As for the new camera model, that's something I can bring up, but the way I have it implemented currently is simple and works. Perhaps we could get a more complete coverage of the mesh with a new camera model. This will become more clear when I implement a metric to track how much of the mesh has actually been covered. |
Codecov Report
@@ Coverage Diff @@
## master #516 +/- ##
=======================================
Coverage 62.16% 62.16%
=======================================
Files 167 167
Lines 7860 7860
Branches 84 84
=======================================
Hits 4886 4886
Misses 2974 2974
Continue to review full report at Codecov.
|
Updated the PR with some example output. |
mpiseno
changed the title
| def extract_poses(self, labels, extraction_method): | ||
| r"""Returns a numpy array of camera poses. If extraction method is "closest", this method will | ||
| use bfs to find the closest point of interest to each camera position. If the extraction method | ||
| is "panorama", it will extract a full panorama of camera poses for each camera position |
There was a problem hiding this comment.
When you mention panorama here, does that mean one image or a number of images taking turns? Let's make it explicit what to expect.
| @@ -28,8 +29,10 @@ def __init__(self, topdown_views, sim, pixels_per_meter=0.1): | |||
| self.sim = sim | |||
| self.pixels_per_meter = pixels_per_meter | |||
|
|
|||
| def extract_poses(self, labels): | |||
| r"""Returns a numpy array of camera poses. | |||
| def extract_poses(self, labels, extraction_method): | |||
There was a problem hiding this comment.
This is part of public API, let's add typing to the signature, especially result type.
| in_bounds_of_tdv = lambda row, col: 0 <= row < len(view) and 0 <= col < len( | ||
| view[0] | ||
| ) | ||
| cpi_label_pairs = [] |
There was a problem hiding this comment.
What is tdv, cpi here? Let's make it verbose.
habitat_sim/utils/filesystem.py
Outdated
| import os | ||
|
|
||
|
|
||
| def search_dir_tree_for_ext(mydir, ext): |
There was a problem hiding this comment.
Let's add typing to the function signature.
|
If you have related unit test that would be great to add |
…arch#516) * created infrastructure for data extraction API * mend * added functionality for data extractor to be ran end-to-end through a pytorch model and tests for that as well as the topdown view * fixed incorrect rotations in camera poses * Added more user options for image extraction and added heuristics to BFS * formatting and added pytorch as dependency * dependencies * modifed topdown test * modified test * added reconfig step if a sim is provided to the test (for image shapes) * fixed error with end-to-end test * Removed PyTorch dependecy except for in tests * creating branch for maskrcnn experiments * added close function to extractor to shut down sim * created notebook for training maskrcnn with habitat - doesn't work on devfair yet * added support for train/test split to ImageExtractor * branch init * added functionality for multiple scene files to extract images from * created data extraction API for habitat-sim * height passed in as arg to getTopdownView and removed examples/settings import * added jupyter notebook example for Dataset Extraction API and changed getTopdownView to return Eigen Matrix * formatting * moved PoseExtractor to separate file and other formatting changes * modified training pipeline * removed torch dependency * failing api tests fix * full training pipeline with overfitting on small dataset * fixed merge conflict in pyrproject.toml * created caching for ImageExtractor so scene switches are less frequent * changed file names to match format * formatting * removed jupyter notebook * --ammend * formatting * removed pytorch vision/detection depednecies and updated comments * --ammend * --ammend * --ammend * fixed deps * deps * adding back a deleted file * fixed bug in cache * minor changes for testing * minor test changes * more test changes * initial commit for panorama pose extraction * --ammend * formatting * typing and updated comments * added typing to ImageExtractor constructor and test for panorama Co-authored-by: Michael Piseno <mpiseno@fb.com> Co-authored-by: Michael Piseno <mpiseno@devfair0238.h2.fair>
…nfig --Refactor for removal of Habsim:SceneConfiguration
Motivation and Context
The data extraction in habitat sim is currently limited because it extracts 1 camera pose per camera position. In this PR I created another method for pose extraction, named panorama, in which for each camera position, we can extract multiple different poses by doing a simple 360 degree turn at each point. The purpose is to obtain a more complete mesh coverage and have more image data per scene to train models with.
How Has This Been Tested
Added a simple test that uses the "panorama" method of pose extraction and asserts that the extractor extracted some poses using this method.
Types of changes
Example Output
Below are a series of image that were taken from the same camera position within Replica Apartment 1. With the other method of pose extraction, this camera position would only produce one image.