Offscreen Renderer Produces Blank Images After Many Renders

[callum-ssegrub]

Checklist

• I have searched for similar issues.
• For Python issues, I have tested with the latest development wheel.
• I have checked the release documentation and the latest documentation (for main branch).

Describe the issue

When repeatedly rendering depth images with Open3D's offscreen renderer, I noticed that at some point the renderer starts producing blank images. I wasn't able to locate an obvious source of this behavior - there are no errors/warning logged or obvious memory leaks.
I was able to reproduce this bug on different python versions (3.10 and 3.11) , on machines with and without a GPU, and with different mesh files. On each of my machines and mesh files, the error occurred after 131042 renders. Test code below.
One mesh had 217396 triangles and 651841 vertices, the other had 28398 triangles and 79130 vertices.

Steps to reproduce the bug

import open3d
open3d.utility.set_verbosity_level(open3d.utility.VerbosityLevel.Debug)
import numpy as np 

meshfile = # meshfile l0cation
mesh = open3d.io.read_triangle_mesh(meshfile, True)
renderer = open3d.visualization.rendering.OffscreenRenderer(1052, 780)
fov_type = open3d.visualization.rendering.Camera.FovType.Horizontal
renderer.scene.camera.set_projection(16, 1, 0.1, 500, fov_type)

mat = open3d.visualization.rendering.MaterialRecord()
mat.shader = 'defaultUnlit'
renderer.scene.set_background(np.array([0, 0, 0, 0]))
renderer.scene.add_geometry("mesh", mesh, mat)


num_renders = 0
for i in range(1000000):
    num_renders += 1
    renderer.scene.camera.look_at([0,0,0], [0,0,-300], [1 , 0. ,-0.])
    rendered_image = np.asarray(renderer.render_to_depth_image(z_in_view_space = True))

    num_depth_non_inf = np.sum(rendered_image[~np.isinf(rendered_image)].flatten())
    if num_depth_non_inf == 0: 
        print("Rendered a blank image", num_renders)
        exit()

Error message

No error message

Expected behavior

The renderer keeps producing the same image

Open3D, Python and System information

- Operating system: Ubuntu 22.04 
- Python version: Python 3.11 
- Open3D version: output from python: 0.18.0
- System architecture: x86_64
- Is this a remote workstation?: observed local and remote
- How did you install Open3D?: pip

Additional information

No response