Questions about segment support.

[Edwardmark]

❔Question

There exists some code about segments, so does yolo-v5 support train detector like mask-rcnn with segmentation prediction?

Additional context

Could you please provide usage to train like mask-rcnn? Thanks in advance.

[Edwardmark]

@glenn-jocher would you please give some comments? Thanks very much.

[glenn-jocher]

@Edwardmark yes, YOLOv5 now has partial support for segmentation labels. Currently segmentation labels of the following format are supported:

img.txt file of the following format (each row can be any length, and row lengths can vary within a file).

class x1, y1, x2, y2, x3, y3, ... xn, yn
class x1, y1, x2, y2, x3, y3, ... xn, yn
class x1, y1, x2, y2, x3, y3, ... xn, yn

rather than the current box format

class xywh
class xywh
class xywh

This allows for better bounding box transformation during augmentation (rotation, scale, translation, etc.), helping to reduce some problems associated with augmenting box labels such as #2151.

We do not have support for full segmentation training yet though, which would require substantial changes to model architectures and training pipelines (IoU functions, test metrics, visualization tools, etc.). We are hoping to introduce this at some point but I don't have a timeline for you at the moment.

[glenn-jocher]

[Edwardmark]

@glenn-jocher so the segment is only used for augmentation, but not for real segmentation task, is that right?

[Edwardmark]

@glenn-jocher but how the bbox is provided? I think the bbox and segmentation coordinates should be provided both.

[glenn-jocher]

@Edwardmark the segmentation labels naturally contain their own extents, so converting a segment into a box is super easy:

yolov5/utils/general.py

Lines 287 to 293 in ad05e37

	def segment2box(segment, width=640, height=640):
	# Convert 1 segment label to 1 box label, applying inside-image constraint, i.e. (xy1, xy2, ...) to (xyxy)
	x, y = segment.T # segment xy
	inside = (x >= 0) & (y >= 0) & (x <= width) & (y <= height)
	x, y, = x[inside], y[inside]
	return np.array([x.min(), y.min(), x.max(), y.max()]) if any(x) else np.zeros((1, 4)) # cls, xyxy

When loading the data, the YOLOv5 dataloader examines each label to determine whether it is a segment label or a box label:

yolov5/utils/datasets.py

Lines 465 to 471 in ad05e37

	with open(lb_file, 'r') as f:
	l = [x.split() for x in f.read().strip().splitlines()]
	if any([len(x) > 8 for x in l]): # is segment
	classes = np.array([x[0] for x in l], dtype=np.float32)
	segments = [np.array(x[1:], dtype=np.float32).reshape(-1, 2) for x in l] # (cls, xy1...)
	l = np.concatenate((classes.reshape(-1, 1), segments2boxes(segments)), 1) # (cls, xywh)
	l = np.array(l, dtype=np.float32)

Actual segmentation models have very different architectures than detection models. For a segmentation version of YOLOv5 you'd basically want the backbone followed by an inverted backbone to return the image to the original size. Also depending on the segmentation task (semantic or instance) the output may be quite complicated, particularly for instance segmentation.

[Edwardmark]

@glenn-jocher Thanks, buddy.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[daikankan]

what if the mask of a single object can not be represented with a single polygon line label ? such as separated parts mask or there is a hole in a mask polygon. Any more strong representation of mask label ? @glenn-jocher

[saitarslanboun]

Is there any progress with segmentation update? I am excited to see the Yolov5 with instance segmentation.

[github-actions]

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 🚀 resources:

• Wiki – https://github.com/ultralytics/yolov5/wiki
• Tutorials – https://docs.ultralytics.com/yolov5
• Docs – https://docs.ultralytics.com

Access additional Ultralytics ⚡ resources:

• Ultralytics HUB – https://ultralytics.com
• Vision API – https://ultralytics.com/yolov5
• About Us – https://ultralytics.com/about
• Join Our Team – https://ultralytics.com/work
• Contact Us – https://ultralytics.com/contact

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐!

[jieruyao49]

@Edwardmark yes, YOLOv5 now has partial support for segmentation labels. Currently segmentation labels of the following format are supported:

img.txt file of the following format (each row can be any length, and row lengths can vary within a file).

class x1, y1, x2, y2, x3, y3, ... xn, yn
class x1, y1, x2, y2, x3, y3, ... xn, yn
class x1, y1, x2, y2, x3, y3, ... xn, yn

rather than the current box format

class xywh
class xywh
class xywh

This allows for better bounding box transformation during augmentation (rotation, scale, translation, etc.), helping to reduce some problems associated with augmenting box labels such as #2151.

We do not have support for full segmentation training yet though, which would require substantial changes to model architectures and training pipelines (IoU functions, test metrics, visualization tools, etc.). We are hoping to introduce this at some point but I don't have a timeline for you at the moment.

@glenn-jocher How do I get annotations like “class x1, y1, x2, y2, x3, y3, ... xn, yn” in a VOC dataset? Why is x1y1 presented in # 2188 in decimal form?

[glenn-jocher]

@jieruyao49 I don't know about VOC segmentations. If they are available you'd have to convert them to the above YOLO segmentation format for use with YOLOv5.

[ryouchinsa]

Using the script general_json2yolo.py, you can convert the RLE mask with holes to the YOLO segmentation format.

The RLE mask is converted to a parent polygon and a child polygon using cv2.findContours().
The parent polygon points are sorted in clockwise order.
The child polygon points are sorted in counterclockwise order.
Detect the nearest point in the parent polygon and in the child polygon.
Connect those 2 points with narrow 2 lines.
So that the polygon with a hole is saved in the YOLO segmentation format.

def is_clockwise(contour):
    value = 0
    num = len(contour)
    for i, point in enumerate(contour):
        p1 = contour[i]
        if i < num - 1:
            p2 = contour[i + 1]
        else:
            p2 = contour[0]
        value += (p2[0][0] - p1[0][0]) * (p2[0][1] + p1[0][1]);
    return value < 0

def get_merge_point_idx(contour1, contour2):
    idx1 = 0
    idx2 = 0
    distance_min = -1
    for i, p1 in enumerate(contour1):
        for j, p2 in enumerate(contour2):
            distance = pow(p2[0][0] - p1[0][0], 2) + pow(p2[0][1] - p1[0][1], 2);
            if distance_min < 0:
                distance_min = distance
                idx1 = i
                idx2 = j
            elif distance < distance_min:
                distance_min = distance
                idx1 = i
                idx2 = j
    return idx1, idx2

def merge_contours(contour1, contour2, idx1, idx2):
    contour = []
    for i in list(range(0, idx1 + 1)):
        contour.append(contour1[i])
    for i in list(range(idx2, len(contour2))):
        contour.append(contour2[i])
    for i in list(range(0, idx2 + 1)):
        contour.append(contour2[i])
    for i in list(range(idx1, len(contour1))):
        contour.append(contour1[i])
    contour = np.array(contour)
    return contour

def merge_with_parent(contour_parent, contour):
    if not is_clockwise(contour_parent):
        contour_parent = contour_parent[::-1]
    if is_clockwise(contour):
        contour = contour[::-1]
    idx1, idx2 = get_merge_point_idx(contour_parent, contour)
    return merge_contours(contour_parent, contour, idx1, idx2)

def mask2polygon(image):
    contours, hierarchies = cv2.findContours(image, cv2.RETR_CCOMP, cv2.CHAIN_APPROX_TC89_KCOS)
    contours_approx = []
    polygons = []
    for contour in contours:
        epsilon = 0.001 * cv2.arcLength(contour, True)
        contour_approx = cv2.approxPolyDP(contour, epsilon, True)
        contours_approx.append(contour_approx)

    contours_parent = []
    for i, contour in enumerate(contours_approx):
        parent_idx = hierarchies[0][i][3]
        if parent_idx < 0 and len(contour) >= 3:
            contours_parent.append(contour)
        else:
            contours_parent.append([])

    for i, contour in enumerate(contours_approx):
        parent_idx = hierarchies[0][i][3]
        if parent_idx >= 0 and len(contour) >= 3:
            contour_parent = contours_parent[parent_idx]
            if len(contour_parent) == 0:
                continue
            contours_parent[parent_idx] = merge_with_parent(contour_parent, contour)

    contours_parent_tmp = []
    for contour in contours_parent:
        if len(contour) == 0:
            continue
        contours_parent_tmp.append(contour)

    polygons = []
    for contour in contours_parent_tmp:
        polygon = contour.flatten().tolist()
        polygons.append(polygon)
    return polygons 

def rle2polygon(segmentation):
    if isinstance(segmentation["counts"], list):
        segmentation = mask.frPyObjects(segmentation, *segmentation["size"])
    m = mask.decode(segmentation) 
    m[m > 0] = 255
    polygons = mask2polygon(m)
    return polygons

The RLE mask.

The converted YOLO segmentation format.

To run the script, put the COCO JSON file coco_train.json into datasets/coco/annotations.
Run the script. python general_json2yolo.py
The converted YOLO txt files are saved in new_dir/labels/coco_train.

Edit use_segments and use_keypoints in the script.

if __name__ == '__main__':
    source = 'COCO'

    if source == 'COCO':
        convert_coco_json('../datasets/coco/annotations',  # directory with *.json
                          use_segments=True,
                          use_keypoints=False,
                          cls91to80=False)

To convert the COCO bbox format to YOLO bbox format.

use_segments=False,
use_keypoints=False,

To convert the COCO segmentation format to YOLO segmentation format.

use_segments=True,
use_keypoints=False,

To convert the COCO keypoints format to YOLO keypoints format.

use_segments=False,
use_keypoints=True,

This script originates from Ultralytics JSON2YOLO repository.
We hope this script would help your business.

[glenn-jocher]

@ryouchinsa sorry for any confusion, but I wanted to clarify that the repository you mentioned at https://github.com/ryouchinsa/Rectlabel-support/ is not an official Ultralytics repository. If you find the scripts useful, feel free to use them, but since they are not part of the official YOLOv5 repository, the Ultralytics team cannot provide official support for them.

If you have any inquiries related to the YOLOv5 repository, feel free to ask!

[ryouchinsa]

@glenn-jocher, we will make a PR about this script to your official repository. Please let us contribute to your official repository if this script would be useful for your company and users.

[glenn-jocher]

@ryouchinsa thank you for your interest in contributing! We appreciate your willingness to share your script. Before making the PR, please note that any contributions to the official YOLOv5 repository need to align with the project's guidelines and goals. Feel free to submit your PR, and our team will review it. We value community contributions that benefit the YOLOv5 user community.