models/sam-2/

[giscus[bot]]

models/sam-2/

Discover SAM 2, the next generation of Meta's Segment Anything Model, supporting real-time promptable segmentation in both images and videos with state-of-the-art performance. Learn about its key features, datasets, and how to use it.

https://docs.ultralytics.com/models/sam-2/

[glenn-jocher]

Hey everyone! We wanted to give you a quick update: our team is currently busy integrating SAM 2 (Segment Anything Model 2) into the Ultralytics package. This new model is super exciting, with features like real-time performance and the ability to segment objects it’s never seen before.

We're really pumped about what SAM2 can do, but it's still in the works. Thanks for your patience as we get everything set up! We'll keep you posted as we make progress. Stay tuned!

[glenn-jocher]

Update: SAM 2 is now fully integrated! Update your package to ultralytics>=8.2.70 with pip install -U ultralytics and then for example:

from ultralytics import ASSETS, SAM

# Load a model
model = SAM("sam2_s.pt")

# Segment with point prompt
results = model(ASSETS / "bus.jpg", points=[150, 150], labels=[1], device="cpu")

[sctrueew]

Could you please provide an example of how to perform inference on a video? I would appreciate it if you could include details on the necessary steps and code snippets.

Thank you!

[glenn-jocher]

@sctrueew to perform inference on a video using Ultralytics, you can use the predict mode. Here's a concise example to get you started:

from ultralytics import YOLO

# Load a pretrained YOLOv8 model
model = YOLO("yolov8n.pt")

# Run inference on a video file
results = model("path/to/video.mp4")

# Display results
for result in results:
    result.show()

This code will load the model, run inference on the video, and display the results. For more details, please refer to the Ultralytics documentation.

[VikeIngenior]

Is it usable with ultralytics package at the moment or should we wait until the implementation is done?

[VikeIngenior]

Hello Glenn, I am having problem with using my fine-tuned YOLOv8 model with the SAM2 model. I think my detections with the frame I get from cv2.VideoCapture doesn't match with the frame I get from SAM2 model.

Below, I explained what I'm trying to do and what did I get. I am open to any other approaches for my purpose.

I have imported SAM2 by cloning its github repo to my Google Drive and just imported it on Colab. I am trying to make a "video segmentation". My aim is to keep track of the object even if the camera angle changes for a couple seconds or even minutes. The reason I use YOLO is, it tracks when the object is detected. By using my YOLO model, I get the mask coordinates of my spesific object. And then by adding/substracting a few pixels by the corners of the mask, I give the prompt points to the SAM2 model. I got a result with that block of code, but it gave prompts for every frame, so it was not kind of what I expected:

i = 0
previous_detection = False
with sv.VideoSink(TARGET_VIDEO.as_posix(), video_info=video_info) as sink:
for frame_idx, frame_path in enumerate(frame_paths):
        frame = cv2.imread(frame_path)
        print(previous_detection)
        # Run YOLOv8 detection
        results = model(frame)
        bbox = None
        min_xy_point = None
        max_x_min_y_point = None
        min_x_max_y_point = None
        max_xy_point = None
        for result in results:
          if result.masks:
              mask_coordinates = result.masks.xy[0]  # Use the first mask
              # Compute bounding box coordinates from mask
              x_min, y_min = np.min(mask_coordinates[:, :2], axis=0)
              x_max, y_max = np.max(mask_coordinates[:, :2], axis=0)
              coordinates = np.copy(mask_coordinates)
              min_xy_point = coordinates[np.lexsort((coordinates[:, 1], coordinates[:, 0]))[0]]
              min_x_max_y_point = coordinates[np.lexsort((-coordinates[:, 1], coordinates[:, 0]))[0]]
              max_x_min_y_point = coordinates[np.lexsort((coordinates[:, 1], -coordinates[:, 0]))[0]]
              max_xy_point = coordinates[np.lexsort((-coordinates[:, 1], -coordinates[:, 0]))[0]]
              print("min_xy: ",min_xy_point)
              print("max_x_min_y: ",max_x_min_y_point)
          if result.boxes:
              bbox = result.boxes.xyxy[0].cpu().numpy()

        if bbox is not None:
          detections = bbox  # Assuming detections for the first batch
          if not previous_detection:
            print(detections)
            print(detections[0])
            if len(detections) > 0:
                points = []
                #for detection in detections:
                x1, y1, x2, y2 = detections[0], detections[1], detections[2], detections[3]

                # Calculate the slope and direction
                _, direction = find_slope((min_xy_point[0], min_xy_point[1]), (max_x_min_y_point[0], max_x_min_y_point[1]))
                print(direction)
                if direction == "decreasing":
                  p1 = min_xy_point[0]+5
                  p2 = min_xy_point[1]+5
                  p3 = max_xy_point[0]-5
                  p4 = max_xy_point[1]-5
                else:
                  p1 = min_x_max_y_point[0]+5
                  p2 = min_x_max_y_point[1]-5
                  p3 = max_x_min_y_point[0]-5
                  p4 = max_x_min_y_point[1]+5
                #else:
                  #p1 = x1+5
                  #p2 = y1+5
                  #p3 = x2-5
                  #p4 = y2-5

                points.append([p1, p2]) 
                points.append([p3, p4])

                yolo_mask_points = np.array([[int(min_xy_point[0]), int(min_xy_point[1])], [int(max_x_min_y_point[0]), int(max_x_min_y_point[1])], [int(max_xy_point[0]), int(max_xy_point[1])], [int(min_x_max_y_point[0]), int(min_x_max_y_point[1])]])

                cv2.polylines(frame, [yolo_mask_points], isClosed=True, color=(255, 0, 0), thickness=2)
 
                points = np.array(points, dtype=np.float32)
                labels = np.ones(len(points))
                sam2_model.reset_state(inference_state)
                _, object_ids, mask_logits = sam2_model.add_new_points(
                    inference_state=inference_state,
                    frame_idx=frame_idx,
                    obj_id=1,  # You can assign unique IDs based on detection
                    points=points,
                    labels=labels,
                )

                masks = (mask_logits[0] > 0.0).cpu().numpy()
                detections = sv.Detections(
                    xyxy=sv.mask_to_xyxy(masks=masks),
                    mask=masks.astype(bool)
                )

                annotated_frame = mask_annotator.annotate(scene=frame.copy(), detections=detections)
                sink.write_frame(annotated_frame)
            else:
                # If no objects detected, write the original frame
                sink.write_frame(frame)
            previous_detection = True
          else:
            # If no objects detected, write the original frame and reset previous detection flag
            sink.write_frame(frame)
            previous_detection = False

        if i > 10:
          break
        i += 1
        print(f"{i}th frame")

[VikeIngenior]

I know it's a long code, but I thought it would be necessary for you to understand what I mean. Here, I don't use propagate_in_video method. So this is no different than image segmentation. In order to use propagate_in_video method, I should first assign some points. Here is what I did:

• First, I simply get the first frame by _, frame=cap.read() method.
• Then I got mask coordinates from that frame with this code results = model(frame) and so on (I don't write the whole code to get the coordinates).
• After that, I assigned the first points with that block of code:

points = []
p1 = min_xy_point[0]+5
p2 = min_xy_point[1]+5
p3 = max_xy_point[0]-5
p4 = max_xy_point[1]-5
points.append([p1, p2]) #Left Prompt Point
points.append([p3, p4]) #Right Prompt Point
print(points)

_, object_ids, mask_logits = sam2_model.add_new_points(
    inference_state=inference_state,
    frame_idx=0,
    obj_id=None,
    points=points,
    labels=labels,
)

Finally, here is what I did to keep track of the object I detected:

with sv.VideoSink(TARGET_VIDEO.as_posix(), video_info=video_info) as sink:
    for frame_idx, object_ids, mask_logits in sam2_model.propagate_in_video(inference_state):
        frame_path = frame_paths[frame_idx]
        frame = cv2.imread(frame_path)

        cv2.circle(frame, (int(max_xy_point[0]), int(max_xy_point[1])), 5, (0, 255, 0), -1)
        cv2.polylines(frame, [yolo_bbox_points], isClosed=True, color=(0, 255, 0), thickness=2)
        cv2.polylines(frame, [yolo_mask_points], isClosed=True, color=(255, 0, 0), thickness=2)

        masks = (mask_logits[0] > 0.0).cpu().numpy()
        detections = sv.Detections(
            xyxy=sv.mask_to_xyxy(masks=masks),
            mask=masks.astype(bool)
        )

        annotated_frame = mask_annotator.annotate(scene=frame.copy(), detections=detections)
        sink.write_frame(annotated_frame)

[glenn-jocher]

Thank you for sharing the detailed explanation. It seems like the issue might be related to the synchronization between the frames processed by YOLOv8 and SAM 2. Ensure that the frames fed into SAM 2 match exactly with those processed by YOLOv8. You might also want to verify that the coordinate transformations between YOLOv8 detections and SAM 2 prompts are accurate. If the issue persists, please try updating to the latest versions of both packages to see if it resolves the problem. For further details, you can refer to the SAM 2 documentation.

[VikeIngenior]

Thank you for your answer 🖐

[glenn-jocher]

@VikeIngenior you're welcome! It sounds like you're making good progress. To ensure your frames from cv2.VideoCapture match those processed by SAM 2, verify the synchronization and coordinate transformations. Also, make sure you're using the latest versions of both packages. If the issue persists, please provide more details or specific errors you're encountering. For further guidance, you can refer to the SAM 2 documentation.

[acerolakirsche]

Hey dear people,
i try to find information about the way to work with SAM 2 with my Apple Silicon. Is there a way?
while trying to install it seems to require CUDA :/

[glenn-jocher]

You can run this code on Apple devices with M1/2/3 chips and everything works correctly, though not all SAM or SAM2 operations are supported on MPS, so if you try to use device="mps" you will get an error. You can post this to the torch repo for them to prioritize MPS support for these ops.

from ultralytics import ASSETS, SAM

# Load a model
model = SAM("sam2_s.pt")

# Segment with point prompt
results = model(ASSETS / "bus.jpg", points=[150, 150], labels=[1], device="cpu")

[esssyjr]

How can you fine-tune this model for your specific needs?
Also, does it support receiving annotations?
What of labels?

[glenn-jocher]

@esssyjr to fine-tune SAM 2 for your specific needs, you can use its promptable interface to segment objects by providing points, bounding boxes, or masks. While SAM 2 currently supports inference only, you can use these prompts to iteratively refine segmentation results. For auto-annotation, SAM 2 can generate segmentation datasets quickly by leveraging pre-trained models. For more details, please visit SAM 2 documentation.

[mariswarycharan]

when will training option enable for SAM 2

[glenn-jocher]

@mariswarycharan currently, SAM 2 supports inference but not training. We are continuously working on enhancing the model's capabilities, and training support might be included in future updates. For the latest updates and detailed information, please visit the SAM 2 documentation. Thank you for your interest!

[chauhtnguyen]

Does this support text prompts? For example, I want to search for buildings or cars from an aerial image or video

[glenn-jocher]

@chauhtnguyen sAM 2 does not support text prompts directly. Instead, it uses visual prompts like points, bounding boxes, or masks to specify objects of interest. For your use case of identifying buildings or cars in aerial images or videos, you can provide bounding boxes or points as prompts to guide the segmentation. For more details on how to use these prompts, you can refer to the SAM 2 documentation.

[nihirv]

You can use the autodistill package to use SAM2 for text prompt based inference

[glenn-jocher]

@nihirv sAM 2 does not support text prompts directly. It uses visual prompts like points, bounding boxes, or masks for segmentation tasks. For more details, you can refer to the SAM 2 documentation.

[nihirv]

@glenn-jocher is this a gpt response? 😂

You can do text prompt based segmentation by first running a zero-shot object detection model over the image with your text prompt. Then you can use the center point of the bounding box as a visual prompt to SAM.

Autodistill does it here in an easy to use package: https://docs.autodistill.com/base_models/grounded-sam-2/

[glenn-jocher]

SAM 2 does not support text prompts directly. However, you can achieve text-prompt-based segmentation by first running a zero-shot object detection model with your text prompt and then using the center point of the bounding box as a visual prompt for SAM 2. For more details, you can refer to the Autodistill documentation.

[moehmeni]

Is there an API to fine tune the model on our own labeled photos? also the schema and shape for the inputs

[glenn-jocher]

@moehmeni yes, you can fine-tune the model on your own labeled photos using the Ultralytics API. For SAM 2, you can use the auto_annotate function to create segmentation datasets. For YOLO models, you can train on custom datasets by specifying the dataset path and model configuration. Ensure your dataset follows the required schema and shape for inputs. For more details on SAM 2, please refer to the SAM 2 documentation.

[YiftyD]

Thank you very much for your beautiful work! I have a question regarding prompts with several points. When I prompt your integration of SAM 2 with several points I get several masks - one mask for each prompt point (which is what I want to get), while when I do the same thing using SAM 2 directly, I get one mask. Are there certain parameters that need to be used while initializing sam2 or while doing the prediction?
Thank you in advance

[YiftyD]

Thank you very much for your answer

[VikeIngenior]

How can I use this sample code in order to make a detection (with YOLOv8) and segment that detection through a video (more like a real-time video stream)? I have run the code and it worked and I got a txt file which includes labels but I couldn't know what to do next.

from ultralytics.data.annotator import auto_annotate
auto_annotate(data=frame, det_model=yolo_model, sam_model=sam_model)

[Kavi91]

Hello !

I am trying to test SAM2, but its says "sam2_b.pt is not a supported SAM model. Available models are:
dict_keys(['sam_h.pt', 'sam_l.pt', 'sam_b.pt', 'mobile_sam.pt'])
"

from ultralytics import SAM

Load a model

model = SAM("/backend/model/sam2_b.pt")

Display model information (optional)

model.info()

Run inference

model("Input_videos/video.mp4")

[glenn-jocher]

@zjq1839 if your version is already above 8.2.70 and you're encountering this issue, please ensure that the model path and name are correct. The supported models are sam_h.pt, sam_l.pt, sam_b.pt, and mobile_sam.pt. Double-check the model file name and try again. If the issue persists, please provide more details about your setup. For further information, you can refer to our SAM 2 documentation.

[Kavi91]

Hi Glen, Now its working. and thank you. but when I use the same code to inference with a video it doesn't show the segmentation. can you check what is wrong in this code :

import cv2
from ultralytics import SAM

# Load the SAM model
model = SAM("Aerial_Vision_Analysis_Tool_v1.0/backend/model/sam2_b.pt")

# Display model information (optional)
model.info()

# Input video path
input_video_path = "Input_videos/A normal day Chennai traffic on top angle.mp4"
output_video_path = "output_videos/processed_video.mp4"

# Run inference in streaming mode
results = model(input_video_path, stream=True)

# Define the codec and create VideoWriter object to save output video
cap = cv2.VideoCapture(input_video_path)
fps = int(cap.get(cv2.CAP_PROP_FPS))
width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
fourcc = cv2.VideoWriter_fourcc(*'mp4v')
out = cv2.VideoWriter(output_video_path, fourcc, fps, (width, height))

# Process the streaming results
for result in results:
    # Assuming result contains the processed frame
    processed_frame = result.orig_img  # Access the original frame after processing

    # Display the processed frame
    cv2.imshow('Inference', processed_frame)

    # Write the frame to the output video file
    out.write(processed_frame)

    # Break the loop on 'q' key press
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

# Release everything when done
out.release()
cap.release()
cv2.destroyAllWindows()

[glenn-jocher]

@Kavi91 hi Glen, glad to hear it's working now! For the video segmentation issue, ensure that result.orig_img contains the processed frame with segmentation. If it doesn't, you might need to access the segmentation masks directly from the result object and overlay them on the original frame before displaying or saving. For more details, refer to the SAM 2 documentation.

[Kavi91]

Hello Glenn, I used default real time inference code but it gives output like below. but I couldn't visualize the result. can you give me a example code to see the visual inference

video 1/1 (frame 1/631) /home/kavinda/Input_videos/a.mp4: 1024x1024 315.1ms
video 1/1 (frame 2/631) /home/kavinda/Input_videos/a.mp4: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1 8, 1 9, 1 10, 1655.6ms
video 1/1 (frame 3/631) /home/kavinda/Input_videos/a.mp4: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1 8, 1 9, 1595.9ms
video 1/1 (frame 4/631) /home/kavinda/Input_videos/a.mp4: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1 8, 1 9, 1606.8ms
video 1/1 (frame 5/631) /home/kavinda/Input_videos/a.mp4: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1608.5ms
video 1/1 (frame 6/631) /home/kavinda/Input_videos/a.mp4: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1 8, 1 9, 1608.6ms
video 1/1 (frame 7/631) /home/kavinda/Input_videos/a.mp4: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1606.0ms
video 1/1 (frame 8/631) /home/kavinda/Input_videos/a.mp4: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 3066.4ms
video 1/1 (frame 9/631) /home/kavinda/Input_videos/a.mp4: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1612.2ms
video 1/1 (frame 10/631) /home/kavinda/Input_videos/a.mp4: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1614.8ms
video 1/1 (frame 11/631) /home/kavinda/Input_videos/a.mp4: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1 8, 1 9, 1 10, 1615.5ms

[glenn-jocher]

To visualize the segmentation results, ensure you are accessing the correct processed frames. Instead of result.orig_img, you should use result.plot() to overlay the segmentation masks on the original frames. This should display the segmented output correctly. If you need further assistance, please refer to the SAM 2 documentation.

[ckn3]

Hey there,

I was wondering how can I set foreground and background points and then visualize the segmentation properly? Here is my current code, but it gives each point's segmentation a distinct color.

from ultralytics import SAM
# Load a model
model = SAM("sam2_b.pt")

# Display model information (optional)
model.info()

# Segment with point prompt
results = model("frame_50.jpg", points=[[960,530], [1340, 980], [1880,1170], [1160,1230],[880,520]], labels=[1,1,1,0,0])

for result in results:
    masks = result.masks  # Masks object for segmentation masks outputs
    result.save(filename="seg/result.jpg")  

[glenn-jocher]

Thank you for the detailed explanation. It seems the "labels" are not effectively guiding the segmentation as intended. This could be due to how the model processes multiple points. To address this, you might need to post-process the masks to unify the colors for foreground and background points. If the issue persists, please check if there are any updates or fixes in the latest version of the package. For further details, you can refer to the SAM 2 documentation.

[sctrueew]

Hi @glenn-jocher,

I'm using GPU 4090
I am currently using the following code:

python
Copy code
import cv2
from ultralytics import SAM

model = SAM(r"C:\Users\Asus\Downloads\sam2_t.pt")
model.info()
model(0)
Despite having a GPU and having installed PyTorch, I am experiencing very slow performance. Could you please provide assistance or suggestions to improve the speed?

Thank you.

0: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1 8, 1 9, 1 10, 5735.2ms
0: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1 8, 1 9, 1 10, 1 11, 1 12, 1 13, 2621.7ms
0: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1 8, 1 9, 1 10, 1 11, 1 12, 1 13, 2641.8ms
0: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1 8, 1 9, 1 10, 1 11, 1 12, 2641.3ms
0: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1 8, 1 9, 1 10, 1 11, 1 12, 1 13, 2607.4ms
0: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1 8, 1 9, 1 10, 1 11, 2685.5ms
0: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1 8, 1 9, 1 10, 1 11, 1 12, 2670.0ms
0: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1 8, 1 9, 1 10, 1 11, 1 12, 2646.2ms
0: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1 8, 1 9, 1 10, 1 11, 1 12, 2612.6ms
0: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1 8, 1 9, 1 10, 2612.1ms
0: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1 8, 1 9, 1 10, 1 11, 2623.1ms
0: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1 8, 1 9, 1 10, 2619.8ms
0: 1024x1024 1 0, 1 1, 1 2, 1 3, 1 4, 1 5, 1 6, 1 7, 1 8, 1 9, 1 10, 1 11, 1 12, 1 13, 2655.6ms

[glenn-jocher]

Hi @sctrueew,

Thanks for reaching out. To improve performance, ensure you're using the latest package version and that your GPU is properly utilized by PyTorch. You can also try reducing the input image size or batch size. If the issue persists, please provide more details about your environment and setup.

[RunsenXu]

Same, here.

The inference speed is very low, with 2.2s per image for SAM-t.

[glenn-jocher]

Hi,

To improve inference speed, ensure you're using the latest package version and that your GPU is properly utilized by PyTorch. You can also try reducing the input image size or batch size. If the issue persists, please provide more details about your environment and setup.

[CA4GitHub]

The web page shows the example below, but what options do I have for saving the results using ultralytics built in methods?
For example, is there a way to run the example below and save the segmented results as a video?
If not a video, what's the simplest way to save the segmented results (no bounding box overlays or other stuff) for each frame in the video?

from ultralytics import SAM

model = SAM("sam2_b.pt")

model.info()

model("path/to/video.mp4")

[glenn-jocher]

result.plot(boxes=False) should omit the boxes.

I think if you don't pass any additional arguments the SAM models will by default segment the entire image.

[esssyjr]

I used the above code you provide, it took more than an hour processing but still didn't work. I have to cancel it.
What do you think the issue might be?

[glenn-jocher]

@esssyjr it sounds like the process might be getting stuck due to the complexity or length of the video. Please ensure you are using the latest version of the package. Additionally, you can try running the model on a shorter video or a smaller segment to diagnose if the issue persists. If the problem continues, please provide more details about the video and system specifications.

[esssyjr]

Thanks for the response.

I tried two videos on my local machine which are 4 and 5 seconds in .MOV and .MP4 format, it didn’t work for more than an hour. But, when I used it on google colab it worked successfully. Is it showing that my processor is low for the task ?

[glenn-jocher]

It seems like the issue might be related to your local machine's processing power. Google Colab provides powerful GPUs which could explain why it worked there. You might want to try running the model on a machine with a more powerful GPU or consider using cloud-based solutions for heavy processing tasks.

[dvolgyes]

The SAM and SAM 2 documentation have an identical block called 'SAM comparison vs YOLOv8'.
I think that is misleading. One key selling point of SAM2 its close to real time speed, so repeating the results of SAM 1 makes no sense. It would be very nice to add SAM2 too, or SAM2 only, but there is nothing helpful having the SAM1 results repeated in SAM2 docs.

My wish is having updated results in the section, but if that is significant work, then rather remove it. If an update is performed, then enumerating all the SAM2 variants would be nice.

[glenn-jocher]

@dvolgyes both SAM and SAM2 speeds are nearly identical and both extremely slow compared to what you might be used to with YOLOv8

[iholy19]

Hello Glenn,
I tested sam2-l.pt. It works perfectly. I noticed that the size of model downloaded from ultralytics is 429MB but the model released by Meta, sam_hiera_large.pt is 857MB. I am wondering if it is the same model. Please help me clarify

[glenn-jocher]

@iholy19 hello, the SAM 2 model you downloaded from Ultralytics is optimized for size and performance, which is why it is smaller than Meta's original release. Despite the size difference, it retains the core functionalities and capabilities. For more details, you can refer to the SAM 2 documentation.

[Sahilrism]

Hello sir , I am trying to run the auto annotate code but I am getting the error please help to solve it

Code :
"from ultralytics.data.annotator import auto_annotate

auto_annotate(data="defect_tyre/inside2.jpg", det_model="best.pt", sam_model="sam2_b.pt")"

Error:
"AttributeError: 'Sequential' object has no attribute 'append'"

Requirement : I want my model to improve its accuracy over time without the need for retraining each time I add more data. I'm looking for a way to enable my model to learn and adapt on its own, similar to reinforcement learning in computer vision. Could you suggest how to achieve this ?

[glenn-jocher]

@Sahilrism please ensure you're using the latest version of the Ultralytics package. The error you're encountering suggests a potential issue with the model loading. For your requirement of continuous learning without retraining, consider using online learning techniques or reinforcement learning frameworks, but note that this is beyond the current scope of YOLO and SAM models. For more details on SAM 2, visit SAM 2 Documentation.

[FOSSEnjoyer69]

how can i use the mask data from the results object to create a mask image?

[glenn-jocher]

@FOSSEnjoyer69 to create a mask image from the results object, you can use the Masks attribute. Here's a minimal example:

import cv2
from ultralytics import SAM

# Load the model
model = SAM("sam2_b.pt")

# Run inference
results = model("path/to/image.jpg")

# Extract mask data
mask = results[0].masks[0].numpy()

# Save mask as an image
cv2.imwrite("mask_image.png", mask * 255)

This will save the mask as a binary image. For more details, refer to the SAM 2 documentation.

[FOSSEnjoyer69]

is it the same with video?

[glenn-jocher]

Yes, SAM 2 can be used for video segmentation as well. It supports real-time processing and handles video-specific challenges like occlusion and reappearance effectively. For more details, you can refer to the SAM 2 documentation.

[shady764]

Hey! I am trying to do video segmentation with point prompting. But it seems to me that only the first frame is segmented with the prompt. For all the other frames, everything seems to get segmented. How can I resolve this?

[glenn-jocher]

@shady764 to ensure consistent segmentation across all frames, make sure you're using the memory mechanism in SAM 2, which helps track objects over time. If the issue persists, try updating to the latest version of the package. For more details, you can refer to the SAM 2 documentation.

[shady764]

How do I use the memory mechanism in SAM 2?

[glenn-jocher]

To use the memory mechanism in SAM 2, ensure it's enabled to maintain object tracking across frames. This feature helps manage occlusions and reappearances by storing and utilizing past frame information. For more details, please refer to the SAM 2 documentation.