Add support for fixed external cameras

[rajat2004]

Fixes: #3208
Fixes: #1896,
Fixes: #1709,
Fixes: #610,
Fixes: #599

About

This PR adds the capability to create a fixed camera separate from the cameras on the vehicle, such as a CCTV camera. I'm not aware of any simple way to go about doing this currently, and therefore thought to try it out. I haven't checked the issues very thoroughly, so please let me know if there are other related issues and ways to go about doing this.

PR has been opened to get ideas and suggestions and whether this is actually required. This will likely become a big PR, so it can be split into parts for easier review also.

Currently have added a boolean field external in the getCameraInfo API, went for this since there are currently 7 camera related APIs, which could increase later on as well. Having completely separate APIs will duplicate the code in many places, the new API while doesn't look the cleanest felt like the better choice to me. Do comment if splitting into a different API or some other way is more preferred.

TODO (Comment if something's missing) -

• Camera APIs implementation
  • getImages
  • set/getDistortionParams
  • getCameraInfo
  • setCameraFov
  • setCameraPose
  • Detection APIs
• Allow External Cameras in -
  • Subwindows
  • Recording (This will likely require a bit of tinkering, after Extend Recording to multiple vehicles #2861, possibly even in a separate PR)
• Documentation updates
• Docstrings update
• Unity fixes to preserve earlier behaviour (No external camera implementation currently)

Also removes the simSetCameraOrientation API kept around for backward compatibility

The last commit to add an internal CameraDetails struct is a NFC commit, just to clean up the multiple parameters being passed everywhere. If it's not desirable then the coomit can be dropped easily. Any suggestions on the struct naming are welcome!

How Has This Been Tested?

Example Settings

{
    "SettingsVersion": 1.2,
    "SimMode": "Car",
    "ExternalCameras": {
        "fixed1": {
            "X": 0, "Y": 0, "Z": -10,
            "Pitch": -90, "Roll": 0, "Yaw": 0
        },

        "fixed2": {
            "CaptureSettings": [
                {
                  "ImageType": 0,
                  "Width": 256,
                  "Height": 144,
                  "FOV_Degrees": 90,
                  "AutoExposureSpeed": 100,
                  "AutoExposureBias": 0,
                  "AutoExposureMaxBrightness": 0.64,
                  "AutoExposureMinBrightness": 0.03,
                  "MotionBlurAmount": 0,
                  "TargetGamma": 1.0,
                  "ProjectionMode": "",
                  "OrthoWidth": 5.12
                }
            ],
            "NoiseSettings": [
              {
                "Enabled": false,
                "ImageType": 0,

                "RandContrib": 0.2,
                "RandSpeed": 100000.0,
                "RandSize": 500.0,
                "RandDensity": 2,
                "HorzWaveContrib":0.03,
                "HorzWaveStrength": 0.08,
                "HorzWaveVertSize": 1.0,
                "HorzWaveScreenSize": 1.0,
                "HorzNoiseLinesContrib": 1.0,
                "HorzNoiseLinesDensityY": 0.01,
                "HorzNoiseLinesDensityXY": 0.5,
                "HorzDistortionContrib": 1.0,
                "HorzDistortionStrength": 0.002
              }
            ],
            "X": 10, "Y": 0, "Z": 0,
            "Pitch": 0, "Roll": 0, "Yaw": 0
        }
    }
}

Example API Call -

client.simGetCameraInfo("fixed1", external=True)

An example Python script has been added, the FoV change will be reflected in the info only after #3278

Script Output

$ python external_camera.py
Connected!
Client Ver:1 (Min Req: 1), Server Ver:1 (Min Req: 1)

Saving images to /tmp/airsim_cv_mode
Camera: fixed1, External = True
<CameraInfo> {   'fov': 90.0,
    'pose': <Pose> {   'orientation': <Quaternionr> {   'w_val': 0.7071067094802856,
    'x_val': 0.0,
    'y_val': -0.7071067690849304,
    'z_val': 0.0},
    'position': <Vector3r> {   'x_val': 0.0,
    'y_val': 0.0,
    'z_val': -5.0}},
    'proj_mat': <ProjectionMatrix> {   'matrix': [   [   0.0,
                      1.0,
                      0.0,
                      0.0],
                  [   0.0,
                      0.0,
                      -1.7777777910232544,
                      0.0],
                  [   0.0,
                      0.0,
                      0.0,
                      10.0],
                  [   -1.0,
                      0.0,
                      0.0,
                      0.0]]}}
Press any key to get images
Type 0, size 39292, pos <Vector3r> {   'x_val': 0.0,
    'y_val': 0.0,
    'z_val': -5.0}
Type 1, size 3514, pos <Vector3r> {   'x_val': 0.0,
    'y_val': 0.0,
    'z_val': -5.0}
Type 3, size 4443, pos <Vector3r> {   'x_val': 0.0,
    'y_val': 0.0,
    'z_val': -5.0}
Type 5, size 983, pos <Vector3r> {   'x_val': 0.0,
    'y_val': 0.0,
    'z_val': -5.0}
Type 6, size 8761, pos <Vector3r> {   'x_val': 0.0,
    'y_val': 0.0,
    'z_val': -5.0}
Press any key to change FoV and get images
Type 0, size 34747, pos <Vector3r> {   'x_val': 0.0,
    'y_val': 0.0,
    'z_val': -5.0}
Type 1, size 2437, pos <Vector3r> {   'x_val': 0.0,
    'y_val': 0.0,
    'z_val': -5.0}
Type 3, size 6756, pos <Vector3r> {   'x_val': 0.0,
    'y_val': 0.0,
    'z_val': -5.0}
Type 5, size 983, pos <Vector3r> {   'x_val': 0.0,
    'y_val': 0.0,
    'z_val': -5.0}
Type 6, size 4360, pos <Vector3r> {   'x_val': 0.0,
    'y_val': 0.0,
    'z_val': -5.0}
Old FOV: 90.0, New FOV: 90.0
Type 0, size 46250, pos <Vector3r> {   'x_val': -10.0,
    'y_val': -5.0,
    'z_val': -5.0}
Type 1, size 7357, pos <Vector3r> {   'x_val': -10.0,
    'y_val': -5.0,
    'z_val': -5.0}
Type 3, size 2015, pos <Vector3r> {   'x_val': -10.0,
    'y_val': -5.0,
    'z_val': -5.0}
Type 5, size 1840, pos <Vector3r> {   'x_val': -10.0,
    'y_val': -5.0,
    'z_val': -5.0}
Type 6, size 12998, pos <Vector3r> {   'x_val': -10.0,
    'y_val': -5.0,
    'z_val': -5.0}
Old Pose: <Pose> {   'orientation': <Quaternionr> {   'w_val': 0.7071067094802856,
    'x_val': 0.0,
    'y_val': -0.7071067690849304,
    'z_val': 0.0},
    'position': <Vector3r> {   'x_val': 0.0,
    'y_val': 0.0,
    'z_val': -5.0}}, New Pose: <Pose> {   'orientation': <Quaternionr> {   'w_val': 0.9975020885467529,
    'x_val': -0.002497921697795391,
    'y_val': 0.04991671442985535,
    'z_val': 0.049916788935661316},
    'position': <Vector3r> {   'x_val': -10.0,
    'y_val': -5.0,
    'z_val': -5.0}}
Distortion Params: [0.0, 0.0, 0.0, 0.0, 0.0]
Setting distortion params as {'K1': 0.1, 'K2': 0.01, 'K3': 0.0, 'P1': 0.0, 'P2': 0.0}
Distortion Params: [0.10000000149011612, 0.009999999776482582, 0.0, 0.0, 0.0]

Subwindws Testing:

Subwindow Settings

{
    "SettingsVersion": 1.2,
    "SimMode": "Car",
    "ExternalCameras": {
        "fixed1": {
            "X": 0, "Y": 0, "Z": -5,
            "Pitch": -90, "Roll": 0, "Yaw": 0
        },
        "fixed2": {
            "X": -10, "Y": 0, "Z": -5,
            "Pitch": -45, "Roll": 0, "Yaw": 0
        }
    },
    "SubWindows": [
        {"WindowID": 0, "ImageType": 0, "CameraName": "fixed1", "Visible": true, "External": true},
        {"WindowID": 1, "ImageType": 0, "CameraName": "fixed2", "Visible": true, "External": true},
        {"WindowID": 2, "ImageType": 0, "CameraName": "0", "Visible": true}
    ]
}

The external_camera.py can be easily used for testing all the camera APIs, as well as vehicle cameras by setting the flag. The detection.py script has also been tested to be working after these changes.

Screenshots (if appropriate):

Old FoV (90):

New Fov (120):

Subwindows:

airsim_cv_mode.zip

[zimmy87]

It seems that you have a lot of conflicts. This is most likely due to conflicts with autoformatted code in master. Please follow these steps to resolve this:

1. check out your pull request
2. run git checkout master .clang-format in your repo's root folder
3. if you don't have npm installed, download it from https://www.npmjs.com/get-npm
4. run npm install -g clang-format
5. run clang-format -i path/to/file for each conflicting file
6. run git add -u to track all style changes
7. run git commit -am "apply style from clang-format"
8. run git merge master
9. resolve other non-style conflicts if you have any
10. push changes

Thanks for helping us with these style changes!

[rajat2004]

@zimmy87 Yeah, there are major conflicts with this one, will be a pretty painful one. I'll try to get this done today, but I think it would be best to get #3472 in first since that API will also need to be adapted. This PR was just intended to see if the feature is feasible and useful. Any review regarding the code and the current API modification would be very much appreciated!

[zimmy87]

a few comments, plus I'm having an issue running the external_camera.py script

[rajat2004]

I skipped testing this earlier since I generally keep my GPU disabled, and launching UE4 Editor often causes my laptop to give up. Using the subwindow settings in the description -

[alonfaraj]

@rajat2004 great feature! thank you for the great work.
Do you think it's worth adding a method to get all the external cameras in the scene?
Similar to listVehicles?

[rajat2004]

Hmm, don't have any real preference either way, pretty straightforward to add that API. The listVehicles API itself becomes more useful with the addVehicle API.

[zimmy87]

PR looks good to me; moving ahead with merging; thanks for the contribution @rajat2004!