Seeking Advice for Extracting Images from Album Pages with High Precision

[hsnawn]

I’m currently working on a project where I need to extract individual images from scanned album pages with extreme precision. Each scan is at a resolution of 5792x5792.

Here’s my current workflow:

• Using YOLOv9 object detection (trained at size 640) to detect and crop individual images.
• Using YOLOv8 segmentation (trained at size 640) to extract the images from the cropped images.

While the results are decent, I am aiming for highly accurate masks and facing some issues:

• Some parts of the images are missing in the mask.
• Parts of the background are included in the mask.
• The masks are not smooth and appear choppy.

I have also tried using Mask RCNN, but the results were slightly worse compared to YOLOv8 and YOLOv9 Segmentation. Additionally, I attempted mask refinement techniques, but they ended up degrading the mask quality.

I’m looking for suggestions on any preprocessing, post-processing, or potentially a different approach that could help me achieve my goal with very high precision.

Any advice or recommendations would be greatly appreciated!

Thank you in advance!

[pderrenger]

@hsnawn hello,

Thank you for sharing your detailed workflow and the challenges you're encountering. It's great to see your dedication to achieving high precision in your project. Here are a few suggestions that might help improve your results:

1. Higher Resolution Training: Given that your scans are at a very high resolution (5792x5792), training your models at a higher resolution than 640 might yield better results. You can try training YOLOv8 and YOLOv9 models at resolutions like 1024 or even higher if your hardware allows it. This can help the models capture finer details.

2. Data Augmentation: Enhance your training dataset with more diverse augmentations. Techniques like random cropping, rotation, and color adjustments can help the model generalize better and improve mask accuracy.

3. Post-Processing Techniques:

   • Morphological Operations: Applying morphological operations like dilation and erosion can help refine the masks. For instance, dilation can fill small holes in the masks, and erosion can remove small noise.
   • Contour Smoothing: You can use contour smoothing techniques such as Gaussian blurring or applying a convex hull to smooth out the choppy edges of the masks.

4. Ensemble Methods: Combining predictions from multiple models can sometimes yield better results. You can use an ensemble of YOLOv8 and YOLOv9 models to get more robust predictions.

5. Fine-Tuning: Ensure that your models are fine-tuned on a dataset that closely resembles your specific use case. This can significantly improve the precision of the masks.

6. Custom Loss Functions: Experiment with custom loss functions that penalize false positives and false negatives more heavily. This can help the model focus on getting the masks more accurate.

Here’s a small code snippet to illustrate how you might apply morphological operations using OpenCV in Python:

import cv2
import numpy as np

# Load your mask image
mask = cv2.imread('mask.png', 0)

# Apply dilation and erosion
kernel = np.ones((5, 5), np.uint8)
dilated_mask = cv2.dilate(mask, kernel, iterations=1)
eroded_mask = cv2.erode(dilated_mask, kernel, iterations=1)

# Save or display the refined mask
cv2.imwrite('refined_mask.png', eroded_mask)

Lastly, ensure that you are using the latest versions of the Ultralytics packages to benefit from the latest improvements and bug fixes.

I hope these suggestions help you achieve the high precision you’re aiming for. If you have any further questions or need more assistance, feel free to ask!

[hsnawn]

@pderrenger Thank you for your suggestions.
Higher Resolution Training: I have trained on imgsz=1024, the results are slightly improved compared to 640. However, when I increase size to 1536 or 2048 the mAP is close to 0.

My training parameters are:
yolo train segment model=yolov8l-seg.pt data=/home/abark/projects/datasets/rect1535/data.yaml epochs=300 patience=30 imgsz=1536 iou=0. device=0,1 batch=6 lr0=0.1 lrf=0.1

[pderrenger]

@hsnawn, thank you for the update and for sharing your training parameters. It's great to hear that you've seen some improvements with a higher resolution. However, the drop in mAP at even higher resolutions is concerning. Here are a few suggestions to help address this issue:

1. Learning Rate Adjustment: A high initial learning rate (lr0=0.1) might be too aggressive, especially for larger image sizes. Consider reducing it to something like lr0=0.01 or even lower to see if it stabilizes the training.

2. Batch Size: Ensure that your batch size is appropriate for the GPU memory. Sometimes, larger image sizes require a smaller batch size to fit into memory. You might want to experiment with reducing the batch size further.

3. Gradient Accumulation: If reducing the batch size affects the training stability, you can use gradient accumulation to simulate a larger batch size. For example, if you reduce the batch size to 3, you can accumulate gradients over 2 steps to simulate a batch size of 6.

4. Warmup Period: Increasing the warmup period can help the model adjust to the larger image sizes gradually. You can set warmup_epochs to a higher value, like 5 or 10.

5. Data Augmentation: Ensure that your data augmentation techniques are suitable for the larger image sizes. Sometimes, augmentations that work well on smaller images might not be as effective on larger ones.

Here’s an example of how you might adjust your training command:

yolo train segment model=yolov8l-seg.pt data=/home/abark/projects/datasets/rect1535/data.yaml epochs=300 patience=30 imgsz=1536 iou=0. device=0,1 batch=3 lr0=0.01 lrf=0.1 warmup_epochs=5 accumulate=2

Additionally, please ensure that you are using the latest versions of the Ultralytics packages to benefit from the latest improvements and bug fixes. If the issue persists, providing a minimum reproducible example can help us diagnose the problem more effectively. You can find more details on how to create one here: Minimum Reproducible Example.

I hope these suggestions help! Feel free to reach out if you have any more questions or need further assistance. 😊