Skip to content

Latest commit

 

History

History
204 lines (143 loc) · 10.1 KB

04_Generating_your_first_dataset.md

File metadata and controls

204 lines (143 loc) · 10.1 KB
Raw

replicAnt

"generating annotated images of animals in complex environments with Unreal Engine"

by Fabian Plum, René Bulla, Hendrik Beck, Natalie Imirzian, and David Labonte (2023)

Generating your first dataset

We tried to make data generation as straight forward as possible, while providing maximal variability. We now briefly outline how to use the generator project within the Unreal Engine 5 Editor - make sure you have acquired a suitable input subject 3D model before you start (see 01, 02, 03)!

The generator setup process is the same irrespective of the desired output datatypes. A single generated dataset can therefore be used to train a variety of networks provided you have the right data parsers.

For additional details (including which settings have been used in our examples) refer to the publication.

Requirements:

Basic configuration

Most functionality is exposed through the replicAnt Interface.

First, open the replicAnt.uproject file. Then, simply double-click on one of the .umap example files under /Content/Generator_config.

The replicAnt Interface

The replicAnt Interface should open by default, located at the top left of the Editor Window. If that is not the case (sigh), simply right-click on replicAnt_Interface in the Content folder and select Run Editor Utility Widget.

The replicAnt Interface consists of four separate tabs. Each tab configures a different aspect of the data generation process.

1. General

  • Path

    • Export Path - Specifies the output location of the generated dataset

    • Batch Name - The name of the dataset and the sub-folder which will contain it, relative to the specified Export Path.

      NOTE : DO NOT include any "_" underscores in the Batch Name (otherwise little lizards will crawl out from under your bed and bite you - seriously, it will break stuff.). Instead, use "-" or " " to separate words. Underscores are used in parser stages to distinguish different generated passes and data-types.

  • General

    • Iterations - Specify how many iterations (unique samples, comprising image passes and annotation files) you seek to generate

    • Passes - Specify which image passes are to be exported per iteration. By default, all passes are exported. We predominantly use of the FinalColor and ID passes. You can add passes by clicking on the (+) icon, or remove passes by clicking on the downwards-pointing arrow next to each pass element.

    • Randomisation Updates

      • Generate New Ground Every Nth - Update the ground plane every nth iteration

      • Scatter New Stuff on Ground Every Nth - Update all asset scatterers every nth iteration

      • Scatter Colony Every Nth - Update the location and pose of all subjects every nth iteration

        NOTE : The ground plane is the lowest hierarchical level, meaning all elements placed on top of it will also be regenerated every time it is updated. Updating the ground plane is thus computationally expensive. The scattered assets in turn influence the placement of the subjects of colony. Therefore, the lower in this list the element appears, the more seldomly (corresponding to lower values) it should be updated to retain a resemblance of computational efficiency. As all materials, the camera position and its optical parameters are updated at every iteration, high variability can be achieved even with infrequent updates of ground plane and scatterers.

    • Image Settings

      • Width / Height - Output dimensions of all passes

      • Export Image File Format - Output format of the Final Color Pass. All other passes remain unaffected and will be writen out as uncompressed .png files.

      • Compression Quality Random - Random output compression of the Final Color Pass. All other passes remain unaffected and will be writen out as uncompressed .png files.

      • Compression Quality - Fixed percentage output compression of the Final Color Pass. All other passes remain unaffected and will be writen out as uncompressed .png files.

      • Export JSON Data - Export the data (annotation) file.

        NOTE: In all provided examples data (annotation) files are required.

2. Subjects

The Subjects tab allows the user to configure the composition and appearance of the simulated colony.

  • Colony

    • Types - List, comprising all subjects part of the simulated colony. You can add subjects by clicking on the (+) icon, or remove passes by clicking on the downwards-pointing arrow next to each subject element. Make sure, that at least one None entry remains on the list.
    • Number Of Subjects - Number of simulated subjects comprising the colony. In each iteration, one subject will be randomly drawn from the list of subjects and the Variation within the range specified below is applied.

  • Variation

    • Hue Shift Min / Hue Shift Max - range of percentage hue shift (HLS colour space) for the subjects' albedo (colour) map (higher = "more red", lower = "more green")
    • Contrast Min / Contrast Max - range of contrast shift for the subjects' albedo (colour) map
    • Desaturation Min / Desaturation Max - range of desaturation (HLS colour space) for the subjects' albedo (colour) map (higher = "less saturated colours", lower = "more saturated colours")

  • Spawn Preview Colony - Spawns a randomised preview of a colony applying the above settings on the Preview Plane (Located above the simulated environment. Click into the viewport, then hold the left & right mouse button to move the view upwards and view the preview colony)

  • Purge Subjects - Removes all currently spawned subjects (applies to both preview colony and subjects placed in the procedurally generated environment.)

    NOTE : Click on Purge Subjects before generating a dataset, in case you have spawned a preview colony.

  • Preview

    • Colony Preview Distance - Determines the spacing between subjects on the preview plane. This does not affect the spacing of individuals during the data generation process.

3. Environment

In the Environment tab, you can specify elements relevant for lighting, post-processing, and world generation.

  • Environment
    • HDRIs - list of all HDRIs (High Dynamic Range Images) the generator will draw from when lighting the environment. This is where new HDRIs can be added, once loaded into the Unreal Project.
    • Fog Is Volumetric - Whether to tread the fog as volumetric during post-processing
    • Maximum Fog Density - Level of fog density / opaqueness
  • Ground Generation
    • Tessellation Level - number of "sub-divisions" performed on the ground plane (higher = "more finely resolved ground plane", lower = "more coarsly resolved ground plane")

4. Debug

The Debug tab can be used to set the randomisation seed and add delays to the data generation process.

  • Random
    • Random Seed - If enabled, a random seed will be chosen for the dataset generation process and written out to the batch data file.
  • Seed

    • Seed - Explicitly set the Seed to repeatedly produce the same variation.

      NOTE : changing the colony size, ground tesselation, or scattered assets will result in different output appearances, even with the same seed: it cannot be guaranteed that the same camera pose be retained across iterations.

  • Debug
    • Engine Warm Up Time , Iteration Warm Up Time , Frame Warm Up Time - introduces delays at various points in the dataset generation process to combat memory access issues which may cause system instability. Computers have fragile souls.

Generate

When all is configured, click Generate at the bottom of any FARTS_Interface tab - the dataset generation process so begins.

The combination of image passes, and data files constitute synthetic datasets that can be used to train networks specialised for a large variety of deep learning-based computer vision applications, including detection, tracking, 2D pose estimation as well as 3D pose estimation, and semantic segmentation.

Advanced settings

(Following soon)

  • changing subject size (-range)
  • changing camera settings
  • custom animation blueprints
  • configure scene lighting
  • configure asset scatterers

Next up:

05 Adding custom assets and scatterers

In case you encounter any problems, consult our troubleshooting guide, or consider raising an issue on the replicAnt GitHub page.

License

© Fabian Plum, Rene Bulla, David Labonte 2023 MIT License