Custom Auto-Label
Custom Auto-Label refers to the AI-enabled automatic labeling feature that is trained and customized using a labeled dataset that a user provides. Users can easily train a machine learning model without any additional coding whenever they obtain a small labeled dataset in the process of creating a new dataset and apply the trained Custom Auto-Label AI to a set of unlabeled images to accelerate the data labeling process.
Custom Auto-Label ultimately aims to leverage AI to reduce the amount of time and costs that go into a labeling project. The resources that a user saves by using this AI-enabled feature can then be re-invested in other high-stakes areas such as sourcing and labeling additional batches of edge case or hard example data.

Using Custom Auto-Label

1. Export labeled datasets

To create a Custom Auto-Label AI, you first need a labeled dataset. Label a small set of raw data (manually, or by importing existing labeled data), and then export the labeled dataset.
❗You need at least 100 labels to create Custom Auto-Label AI.

2. Create Custom Auto-Label AI

There are two ways to create Custom Auto-Label AI.
 1. Go to the Label Exports Tab
  (1) Click the Label Exports tab in the Project sidebar.
  (2) ClickCreate a Custom Auto-Label AI to start training a custom auto-label model on the exported dataset.
  (3) Check the expected amount of Auto-Label credits to be used, and click OK .
 2. Go to the Custom Auto-Label AI Tab
  (1) Click the Custom Auto-Label tab in the Project sidebar.
  (2) Click + Create Custom Auto-Label AI at the top right, and select a labeled dataset from the Export History.
  (3) Click Confirm to see the expected amount of Auto-Label credits to be used and the remaining amount, then click Confirm again.
  • Simply click theApply button on the righ-hand side of the card to apply Custom Auto-Label AI.
  • The performance of each object class is evaluated through precision and recall scores. Toggle/Expand the card to see the full evaluation.
❗ Apply button and precision/recall index are only supported for custom autolabeling AI cards created after 2021-09-15
Related Manual: What are Precision and Recall?

3. Inference

You can use Custom Auto-Label in the same way you use a general Auto-Label model by configuring it in the Project Configuration page.

3-1. Configure Auto-Label

You can configure the Auto-Label AI you created in the following ways.
 1) Auto-Label Settings button
  (1) Map each object class and Custom Auto-Label AI on the Auto-Label Settings tab, and complete the Project Configuration (Configure Auto-Label).
 2) Apply button
  (1) Click Apply on the right side of each card in the Custom Auto-Label tab. This will display a popup window for Apply Custom Auto-Label AI. (2) Select the object class you want to apply to the Custom Auto-Label AI, and click Apply.
(3) A popup window will appear when the Custom Auto-Label is successfully applied to the project, and you can optionally choose to navigate to either Labels or Auto-Label tab by clicking the link.

3-2. Run Auto-Label

  • If there are many unlabeled images in the project, it is recommended that you apply the Custom Auto-Label to a subset of the data (i.e. a few hundred to a few thousand) and then repeat the process above to keep improving the accuracy of the Custom Auto-Label AI.

4. Modify

Once Custom Auto-Label is complete, review the labeled data and correct if there are any errors.

5. Export Modified Dataset

Re-export the dataset, but this time including the newly labeled data that you've just created in the steps above. The re-exported dataset should now contain more labeled data and should be able to train a new Custom Auto-Label AI model that performs better than the model created in the previous step.

6. Repeat (Create Again a Custom Auto-Label)

Once again, click Create a Custom Auto-Label AI in Labels Export to create a new Custom Auto-Label AI.
  • Repeat steps 2-6 to increase the performance and accuracy of the Custom Auto-Label AI. The higher the performance, the fewer labels that need to be corrected.

Additional Resources

Which annotation types does Custom Auto-Label Support?

Name
Custom Auto-label AI
Image Category
O
Bounding Box
O
Polygon Segmentation
O
Keypoint
X
Polyline
X
  • Precautions when using image category custom autolabeling
    • Among the response types, only multiple selection and multiple choice are available. Free response is not available.
    • If you want to use a custom autolabeling AI model created in another project, the following conditions must be met.
      • Same response type (multiple selection, multiple choice)
      • Same option name (order and attribute name are irrelevant)
❗ Refer to the following manual for support information for each annotation type of 'auto-labeling' using the suite's own model (Pre-trained AI).

Which usage plan do I need to use Custom Auto-Label?

Currently, the Custom Auto-Label feature is only available to Enterprise plan users. Auto-Label Credit is limited by Suite user plan and can be found in Settings > Billing & Usage tab.

Auto-Label Credits

Auto-Label Credits are deducted whenever you create a new Custom Auto-Label AI or apply it to label raw data. Please refer to the relevant documentations for more information.
Related Manual: Billing & Usage > Auto-Label Credits

Note

The Apply button and precision-recall metric are only supported for Custom Auto-Label AI created after 2021-09-15.

Frequently Asked Questions (FAQ)

How can I improve the performance of my Custom Auto-Label AI?

  • The performance improves as the size of the labeled dataset used for training increases and as the accuracy of labeled dataset improved. The size of a labeled dataset refers to both the number of raw data in the dataset, as well as the number of annotations in each piece of raw data.
  • We recommend that you update and recreate a Custom Auto-Label AI as you gather more and more labeled raw data to continuously improve its performance.

What is the minimum number of labeled data required for Custom Auto-Label training?

The minimum number of images required for Custom Auto-Label training is 1,000-2,000. If there are only a small number of annotated objects in each image, then you will likely need more labeled data for adequate model training.

Will creating Custom Auto-Label stop when I run out of Auto-Label credits?

Yes, Custom Auto-Label creation will stop when all credits are exhausted. In this case, a pop-up will appear where you can choose whether to proceed or stop. If you do decide to proceed, you will be charged for overage.

What are Precision and Recall?

Precision and recall are measures used to evaluate the performance of machine learning models.
  • Precision: the fraction of relevant instances (where the actual value is 'true') among all retrieved instances (where the model classifies as 'true').
    ex. Proportion of actual dogs among the objects labeled as 'dog' by Custom Auto-Label AI.
  • Recall: the fraction of retrieved instances (where the model classifies as 'true') among all relevant instances (where the actual value is 'true').
    ex. Proportion of properly labeled dogs among all 'dog' objects.
Both metrics represent 'the proportion of a real object X properly labeled as X', but precision is calculated from the model's point of view, while recall is calculated from the object's point of view. The metrics to focus on may vary depending on the nature of the project.
For example, if the average number of objects in one label is very large, or if there are many overlapping objects in one label, it is advantageous to apply a model with high precision. If you apply a low-precision model, a lot of false detections will occur, and it will become difficult for a human to manually correct it during the QA process. Therefore, even if the detection rate is low—in other words, the recall rate is low—reducing false positives in object detection with high precision Auto-Label AI can greatly increase labeling efficiency.
On the other hand, in the case where it is convenient to edit or remove annotations, it can be more efficient to select a model with high recall, rather than redrawing annotations from scratch.
Better models have higher values for both precision and recall, but there is often a trade-off between precision and recall. Higher precision leads to lower recall, while higher recall results in lower precision. These measures should give you a good sense of which metric to focus on, depending on how you design the project.
Any other questions? E-mail us at [email protected]