convert selected 2D pixels to 3D points using pyrealsense

[mistimohan]

          You are very welcome, @mistimohan - I look forward to your next report.  Thanks very much for the update!

Originally posted by @MartyG-RealSense in #11788 (comment)
Quite new to github,so please ignore any of my trivial mistakes
Hey sorry for the late response as I have been stuck with another area of the project. I tried the method rs2_project_color_pixel_to_depth_pixel suggested by you but it only provides me depth pixel(I am assuming x and y) not the 3d point .I am doing object detection on rgb image and I need to filter out (/or calculate ) the point cloud of only objectAlso to clearify,my requirement is : **I only need a part of whole point cloud based on desired object in rgb image **rather than whole point cloud using pc.calculate(depth_frame) .So I need a method that converts a set of 2d pixels (or masked rgb/depth image) to a 3d point in space using pyrealsense.
I was hoping that Realsense could or should provide this basic masked rgb image to point cloud conversion as Realsense is being used extensively for object detection and processing.Also why there is a change in orientation compared to real image when I am using convert_depth_to_phys_coord_using_realsense to create point cloud ?


The above image is when i use :roslaunch realsense2_camera rs_aligned_depth.launch

Above image is when i use my code :

/def get_3d_coord(depth_image,camera_info_msg,x1=0,y1=0,x2=480,y2=640):
points=[]
for i in np.arange(x1,x2,4):
for j in np.arange(y1,y2,4):

        result = convert_depth_to_phys_coord_using_realsense(i,j,depth_image[i,j]*0.001,camera_info_msg)
        points.append([result[0],result[1],result[2],result[2]])

[MartyG-RealSense]

The RealSense SDK provides support for basic pointcloud generation but for advanced pointcloud features it is recommended to combine the RealSense data with a specialized external pointcloud library such as PCL or Open3D, as the RealSense SDK does not provide similar in-built features.

---

In regard to object detection, numerous RealSense users make use of YOLO, sometimes also with ROS tools such as darknet_ros for 2D bounding boxes and darknet_ros_3d for 3D bounding boxes - see IntelRealSense/realsense-ros#2311 (comment)

---

The rs_aligned_depth.launch launch file is very old, so I would recommend using the newer rs_camera launch file and enabling alignment in the launch with align_depth:

roslaunch realsense2_camera rs_camera.launch align_depth:=true

---

Regarding convert_depth_to_phys_coord_using_realsense, I am not familiar enough with this code to offer advice on why you are experiencing an orientation problem with it, unfortunately. Though RealSense ROS cameras do use a different axis system to the ROS default of Z = up and Y = forward. For RealSense cameras, the axis system is Z = forward and Y = up. The RealSense ROS wrapper converts the RealSense axis system to the ROS standard axis system.

[mistimohan]

Thank You for your response. I will look more into YOLO and darknet_ros in depth for answers to my queries .

[mistimohan]

Hey @MartyG-RealSense ,I was able to solve the above problem by using intel realsense d435 urdf file and then publishing the camera points data with the right header:
Earlier I thought even if I don't provide any header, the data which I am getting directly from the camera would originally have the right orientation. But as you said, things would mess up due to the different coordinate systems of Rviz and camera d435i.
So I added a header frame of camera_color_optical_frame which shows the right result as I wanted. But I wanted to confirm once that I hope I am choosing the right origin.

def publish_depth_points(points):
     fields = [PointField('x', 0, PointField.FLOAT32, 1),
              PointField('y', 4, PointField.FLOAT32, 1),
              PointField('z', 8, PointField.FLOAT32, 1),
              PointField('intensity', 12, PointField.FLOAT32, 1)]

    header = Header()
    #This ensures that the points are published wrt to present the location and orientation of the camera.
    header.frame_id = "camera_color_optical_frame"
    # header.frame_id = "camera_color_frame"
    header.stamp = rospy.Time.now()
    pc2 = point_cloud2.create_cloud(header, fields, points)
    pub.publish(pc2)

[MartyG-RealSense]

It's excellent news that you achieved a solution, @mistimohan - thanks so much for sharing it! As you have solved your issue, I will close this case. Thanks again!