Manual review is one of the most basic quality evaluation methods, in which all or a subset of the labeled data are directly inspected by human reviewers during the validation stage.
The review can be performed on the entire labeled dataset or a sample of the dataset. A full inspection can improve the quality of your dataset, but will require more time.
In the review stage, the reviewer evaluates labeled data and can approve or reject them. In addition, the Suite provides role-based access control for features such as issue tracking, labeling status management, and review workflow so that customers can build an optimal workflow and communicate effectively especially during the data inspection and correction stages.
Manual Review Process
1. Assignee — Submit/Skip Labels
After completing a labeling task in Label Mode, an Assignee submits the task or skips the label that is not suitable for labeling.
When labeling, the label status is In Progress. When the label is submitted or skipped, the label status changes to Submitted·Skipped.
The review round starts at the same time as the label submission and skip, and the review status changes to Pending.
*Review round: A series of processes in which a labeler submits a label and a reviewer reviews the label to decide whether to approve or reject it is called a "review round."
2. Reviewer — Review Labels
❗ Only Owner·Admin·Manager can be reviewers. Labelers cannot be reviewers.
2-1. Run the Review Mode by Clicking Review on the top of the project sidebar or clicking on a particular label in the label table.
ℹ️ When you click Review, a Full-screen Review Mode is launched that allows you to review labels that are pending review one by one.
ℹ️ When you click on a particular label, a Modal Review Mode is launched that allows you to review only that label.
2-2. After reviewing the label, you can select whether to approve or reject the label with the Approve and Reject buttons on the top right.
Approving a label means that it is suitable for model training.
Once the label is approved, the review status changes to Approved, and the review of the label ends.
Rejecting a label means that it is not suitable for model training.
The reviewer must enter the reason when rejecting the label. This reason is recorded in the Issue Thread.
Rejecting the label will change the review status to Rejected, and the label will be re-assigned to the previous labeler. The label status is also changed to In Progress.
3. Existing Labeler — Revising Rejected Labels
3-1. The rejected label will be re-assigned to the previous labeler.
The labeler can modify the label by checking the reason for rejection via the issue thread.
When the labeler starts the Label Mode, the rejected label is surfaced first.
3-2. Labelers must resubmit labels after editing.
When the labeler submits the labels after editing, the next review round begins.
Repeat steps 1~3 to review labels in the queue.
Manual Review Statistics
In the project Overview, you can check the statistics on the progress of the manual review in nearly real-time. When you click the Review chart highlighted below, the list of labels that have been approved or rejected will be displayed.
Labeling Status is divided into three states : In progress, Submitted or Skipped.
Each submission status is divided again into two states : In Progress: Rejected, Not Submitted/Submitted, Skipped but Approved, Pending Review.
To get a more detailed look at the labels within each stage, you can click on specific card which will guide you to the Labels tab. From here you can see the desired list of labels.
Review Status and Reviewer Filter
There are three label review status: Pending, Approved, and Rejected.
The Filter function on the Labels tab allows filtering based on the Label Review Status (Approved, Rejected, Pending) and the reviewer.
Click the Reviews column to arrange the high review round and rejected labels first.
If you hover in the review column, you can check the reviewer information shown via text bubble.
Review Statistics by Labeler User
In the User Reports tab, you can check the number of approved and rejected labels per each labeler user.
What is the difference between entering Review mode by clicking the Review button on the project sidebar and by clicking a specific label on the label table?
Clicking the Review button opens the Full-screen Review mode. You can get labels pending review assigned and review them in order.
Clicking a specific label opens the Modal Review mode. You can review only the selected label.
On ‘Full-screen Review Mode’, you can get labels pending review assigned and review them in order.
What happens if I change the status of a label that has been reviewed to In Progress?
If you change the status of a label that has been reviewed to In Progress, the review history will disappear temporarily. Relabeling and submitting that label will restore the original review history.
What happens if I change the 'Assignee' of a label that has been reviewed?
If you change the 'Assignee' of the review label, all the previous review history will be deleted.
When 2 or more reviewers review a label, who is the 'Reviewer' filter applied to?
Reviewer filter is applied based on 'most recent reviewer'. In other words, to filter the labels reviewed by A in the first review round and by B in the second review round, the Reviewer must be selected as 'B'.