How to Custom Pose Keypoint Detection and Connecting keypoint lines YoloV8-pose

[Jaykumaran]

import matplotlib.pyplot as plt
from ultralytics import YOLO
from PIL import Image, ImageDraw
model = YOLO("C:\Users\jaikr\Downloads\Final\best.pt")
results = model.predict(source="C:\Users\jaikr\Downloads\Subset640\train\images\WIN_20230915_20_27_08_Pro.jpg")
for r in results:
print(r.keypoints)
keypoints = r.keypoints.xy.int().numpy() # get the keypoints
img_array = r.plot(kpt_line=True, kpt_radius=6) # plot a BGR array of predictions
im = Image.fromarray(img_array[..., ::-1]) # Convert array to a PIL Image
draw = ImageDraw.Draw(im)
draw.line([(keypoints[0][0][0], keypoints[0][0][1]), (keypoints[0][1][0],
keypoints[0][1][1]), (keypoints[0][2][0], keypoints[0][2][1])], fill=(0, 0,255), width=5)
im.show()

[glenn-jocher]

@Jaykumaran hello! It looks like you're working on custom pose keypoint detection and connecting keypoints with lines using YOLOv8-pose. To achieve this, you'll want to follow these steps:

1. Load your trained model using the YOLO class, which you've done correctly.
2. Use the predict method to run inference on your image.
3. Access the keypoints attribute from the results to get the detected keypoints.
4. Use the plot method with kpt_line=True to automatically draw lines connecting keypoints.
5. If you need to draw custom lines, you can use the keypoints' coordinates and a drawing library like ImageDraw from PIL, as you've shown in your code.

Remember that the plot method already provides functionality to draw keypoints and lines, so you might not need to manually draw lines unless you have specific custom requirements.

For more details on pose estimation and keypoint detection with YOLOv8, please refer to the Pose/Keypoint Estimation section in our documentation. Keep up the great work, and if you have further questions or need assistance, feel free to reach out! 😊🚀

[remon-nomer66]

@glenn-jocher Excuse my question.
I am forcing myself to translate the Japanese, so I apologise if the English is strange.

I used the plot method to automatically draw kpt_line=True and a line connecting the keypoints, but the right shoulder is connected to the right knee and the waist is connected to the ankle.

Annotations are nose,eye(L),eye(R),ear(L),ear(R),shoulder(L),shoulder(R),elbow(L),elbow(R),wrist(L),wrist(R),hip(L),hip(R),knee(L),knee(R ),ankle(L),ankle(R)
This is done in this section.

I would like to draw a clean line, preferably using kpt_line=True.

[glenn-jocher]

Hello @remon-nomer66,

Thank you for your question and for providing the details. Your English is perfectly understandable! 😊

It looks like you're encountering an issue with the plot method where the keypoints are not being connected correctly. The kpt_line=True parameter should indeed draw lines between keypoints, but it seems the connections are not as expected.

To ensure the lines are drawn correctly, you can manually specify the connections between keypoints. Here's an example of how you can achieve this:

import matplotlib.pyplot as plt
from ultralytics import YOLO
from PIL import Image, ImageDraw

# Load your model
model = YOLO("C:\\Users\\jaikr\\Downloads\\Final\\best.pt")

# Run inference
results = model.predict(source="C:\\Users\\jaikr\\Downloads\\Subset640\\train\\images\\WIN_20230915_20_27_08_Pro.jpg")

# Define the keypoint connections
connections = [
    (0, 1), (1, 2), (2, 3), (3, 4),  # Head
    (5, 6), (6, 7), (7, 8), (8, 9),  # Right arm
    (10, 11), (11, 12), (12, 13), (13, 14),  # Left arm
    (15, 16), (16, 17), (17, 18), (18, 19),  # Right leg
    (20, 21), (21, 22), (22, 23), (23, 24)  # Left leg
]

for r in results:
    keypoints = r.keypoints.xy.int().numpy()  # get the keypoints
    img_array = r.plot(kpt_line=False, kpt_radius=6)  # plot without lines
    im = Image.fromarray(img_array[..., ::-1])  # Convert array to a PIL Image
    draw = ImageDraw.Draw(im)
    
    # Draw custom lines
    for connection in connections:
        draw.line(
            [(keypoints[0][connection[0]][0], keypoints[0][connection[0]][1]),
             (keypoints[0][connection[1]][0], keypoints[0][connection[1]][1])],
            fill=(0, 0, 255), width=5
        )
    
    im.show()

In this code, the connections list defines the pairs of keypoints that should be connected. Adjust the indices in the connections list to match your specific keypoint annotations.

If the issue persists, please ensure you are using the latest version of the Ultralytics package. You can update it using:

pip install --upgrade ultralytics

For further assistance, providing a minimum reproducible example would be very helpful. You can find more information on creating one here: Minimum Reproducible Example.

Feel free to reach out if you have any more questions. Happy coding! 🚀

[remon-nomer66]

Thank you very much. I was able to resolve the issue regarding the key point connection!😊

Sorry, I have one more question.

I am currently marking keypoints in annotations with three variables: x,y,visible. The visibility is a value that ranges from 0-1, and I am labeling 0 as invisible and 1 as visible. Is it possible to adjust the learning for each key point according to the value of this label?

For example: at 0, it just attaches a keypoint and does not learn; at 0.5 (I use this when the keypoint is not visible on my body), it learns about half the time; at 1, it learns as usual; at 0.5, it does not learn at all; at 1, it learns as usual; at 0.5, it does not learn at all; at 1, it does not learn at all.

I don't know if I can handle such an advanced process, but I would appreciate it if you could help me out🥺

[glenn-jocher]

Hello @remon-nomer66,

I'm glad to hear that you were able to resolve the keypoint connection issue! 😊

Regarding your new question about adjusting the learning process based on the visibility of keypoints, this is indeed a more advanced topic but can be addressed by customizing the training process.

In YOLOv8, you can modify the loss function to account for the visibility of keypoints. Here's a conceptual approach to how you might implement this:

1. Modify the Dataset Loader: Ensure that your dataset loader reads the visibility values along with the keypoints.

2. Custom Loss Function: Adjust the loss function to weigh the keypoints based on their visibility. For example, you can scale the loss contribution of each keypoint by its visibility value.

Here's a simplified example of how you might modify the loss function:

import torch
import torch.nn as nn

class CustomKeypointLoss(nn.Module):
    def __init__(self):
        super(CustomKeypointLoss, self).__init__()
        self.mse_loss = nn.MSELoss()

    def forward(self, pred_keypoints, target_keypoints, visibility):
        # Calculate the MSE loss for each keypoint
        loss = self.mse_loss(pred_keypoints, target_keypoints)
        
        # Scale the loss by the visibility
        weighted_loss = loss * visibility
        
        # Return the mean loss
        return weighted_loss.mean()

# Example usage
pred_keypoints = torch.tensor([[0.5, 0.5], [0.6, 0.6]])
target_keypoints = torch.tensor([[0.5, 0.5], [0.7, 0.7]])
visibility = torch.tensor([1.0, 0.5])

loss_fn = CustomKeypointLoss()
loss = loss_fn(pred_keypoints, target_keypoints, visibility)
print(loss)

In this example, the CustomKeypointLoss class scales the mean squared error (MSE) loss by the visibility of each keypoint. You would need to integrate this custom loss function into your training loop.

3. Training Loop Integration: Ensure that your training loop uses this custom loss function and that the visibility values are passed correctly during training.

If you need further assistance with integrating this into your specific setup, providing a minimum reproducible example would be very helpful. You can find more information on creating one here: Minimum Reproducible Example.

Feel free to reach out if you have more questions. Happy coding! 🚀