Introducing Autopolygon With SAM2

Introduction

Welcome to the jungle! In this guide, we’ll walk you through the process of training Start Of The Art instance segmentation models using the Autopolygon tool to label the data.

By the end of this guide, you’ll have a better understanding of how to prepare your data, build and train a model, and evaluate its performance.

In terms of model choice, we will train both a YOLO12 and a YOLO11 segmentation model. We will then test them on a video stream to see how they perform.

What is Autopolygon?

Its a tool in our annotator that allows you to label your data in a more efficient way. It uses a combination of AI and human intelligence to create high-quality segments.

Here is a short video showing the Autopolygon tool in action:

What is Segmentation and when would I need it?

Segmentation is the process of dividing an image into multiple segments. It is used in various applications such as medical imaging, autonomous driving, and satellite imaging.

In these applications, detailed analysis of object shapes and structures is crucial, making segmentation a better choice than classification or object detection.

A detailed use case would be for example detecting sea lice and welfare defects on salmon fish. You would segment out the individual fish as the first step in the process to get your region of interest (ROI). Then you can use an object detection model to detect the lice and defects on a cropped fish segment. This is allows the detection model to focus on the important parts of the image and ignore the rest along with relating the detections to the correct fish.

Annotating the data

You can download the unlabeled dataset and follow along if you wish.

We will create a new instance segementation project in Oslo and upload our unlabelled data. We will then use the autopolygon tool to label the data.

Under the hood the Autopolygon tool runs a version of SAM2 foundational model to create segments.

Exporting the data

Go to the exports tab and create an export for the project in the YOLO segmentation format.

Training the models

We have created 2 simple notebooks for training these models. You can find them in the following links:

One thing to note is we have trained the YOLO11 model from a starting point with pretrained weights. When training a YOLO11 model from scratch with a small dataset, it is recommended to use a pretrained model to avoid overfitting.

At this point I could not find any pretrained weights for the YOLO12 Segmentation model, so we trained it from scratch, it was unrealistic to train on full COCO dataset for 300-500 epochs.

Results

When comparing the results between training sessions for V11 and V12 you can see immediately that V11 has better performance. This is expected due to the pretrained weights used in the training process vs. training from scratch.

compare segmentation models — Confusion Matrix Yolo11-Seg Training

From looking at the precision / recall graphs it looks like we could have trained for more epochs ( we trained for 200 epochs) to get better results. I would next try 300 epochs and see if that improves the results.

Video Example

Here is a video example of the YOLO11-Seg model we trained in action:

You can see it performs well with fox class but not with the cat as it thinks its a dog! Next step would be to improve the training set with more cat images and train again.

Also it does not perform so well with occluded objects, this is a common problem with segmentation models.

We will try to see if we can get better results by combining an object detection model with SAM2 in the future post.

Conclusion

In this guide, we introduced you to the Autopolygon tool and showed you how to train two instance segmentation models using annotated data. The Autopolyon tool is a powerful tool that can help you label your data more efficiently and accurately, and can also be used for creating object detection datasets.