Changelogs
Consensus Labeling
Video Data SDK / CLI Support
And many others!
1. Labeling Status on the Project Overview Page
With the new manual review feature, the review status view in the Project Overview page has been improved to provide greater detail on the number of labels in each status.
You can now check the number of labels per each status: In progress, submitted, and skipped, depending on whether or not the label is submitted
Each status is further divided into 2 statuses, depending on whether or not the label has been reviewed
Note: Reviewers can approve/reject both submitted labels and skipped labels. Rejected labels will be assigned back to the labeler.
You can now simply click on each status in the Overview Page and get the filtered list of data immediately.
2. Filter by Auto-Label Difficulty
You can now filter labels by ‘Auto-Label Difficulty’ levels: Easy, Medium, and Hard
1. Suite UI/UX Renewal
1-1. New Navigation Bar
.png?alt=media&token=61eeb721-26b8-4656-b95f-5ed18ecc6b44)
The navigation bar has been changed from being located horizontally at the top to vertically on the left.
A new sidebar has been added to show all the project features at once. You can now quickly navigate to the desired tab.
1-2. Data Upload UI/UX Update
.png?alt=media&token=5fbfc07e-bb13-4668-8512-6ff1ab7c8961)
The user interface of Data Upload has been updated. You can now easily follow the instructions on how to upload data step-by-step on the page.
The CLI(Command Line Interface) Upload option has been added. You can now choose one of the four methods (File, Cloud, URL, or CLI) to upload data.
↗ Related Manual:
1-3. Review Mode, Label Mode
.png?alt=media&token=1d616122-8851-4700-bc32-88c38da37a8f)
The nomenclature for what we used to call 1) Detail View and 2) Annotation App has been changed to 1) Review Mode and 2) Label Mode.
You can create and edit annotations in Label Mode; check the results, add comments and perform quality assurance in Review Mode.
In Review Mode, you can view detailed information of a label in the Label Information panel. You can also use the Approve/Reject button for Manual Review.
You can quickly find all the information on the labeling task (object classes and annotation types) in the Tasks panel; Generated annotations are listed in the Object panel, and you can leave comments/feedback in the Issues panel; You can view keyboard shortcuts in the Shortcut panel and configure labeling conditions in the Adjustments panel.
Click on the edit button at the bottom right of the page in Review Mode. You can switch to Label Mode without having to open a new tab.
↗ Related Manual:
2. Manual Review
Labelers can now submit labels and have them reviewed by reviewers/validators. Reviewers can evaluate the quality of labels and either approve or reject each label. Manual Review is the most basic method of quality assurance in Suite.
↗ Related Manual:
Add Data to Project faster
The time it takes to assign large-scale data to a project has been reduced.
Improved performance of Custom Auto-Label
With the implementation of a new, improved deep learning architecture and training module, the object detection performance of our custom auto-labeling model has been improved.
1. Export History
You can now edit the name of each export job on the Export History page.
You can delete unwanted export history by clicking on the Trash icon in the upper right corner of each export.
2. Profile Picture
You can now add or change your profile picture on Suite. The default profile picture is generated using the first letter of your name with random background color.
Go to My Profile page on My Account tab and click on the ‘Edit’ button to make changes.
3. User Report
You can now view “Category Count” and “Labeling Time” information on our User Reports under the Analytics tab.
1. Improved Data Upload
You can now directly add data to your project on the project overview page.
Click on the “assign existing data” button under the “To Do” section to start adding existing data.
If your project is empty, you can also add your existing data to the project from the Label List by clicking on the “assign existing data” button.
For video projects, you can now upload more than 100,000 frames via Cloud Upload.
1. Import existing custom keypoint templates into a new project You can load and re-use keypoint templates that you have already created in different projects.
2. Suite SDK Superb AI Suite SDK is now available via Github. Please refer to the following link for more guides and example SDK use-cases: Superb AI Suite Examples SDK
1. Keypoint Auto-Labeling
Auto-labeling is now available for keypoints detection.
We currently support the ‘person’ class for our Keypoint Auto-Labeling. To use this pre-trained AI, the object class you select must be configured on the ‘Human template’.
2. Custom Keypoint Template
You can now add your own custom keypoint template.
When creating a project, you can configure a custom keypoint template in the Object Class section. Please copy and paste the JSON file that contains the location, name and color information of the keypoints. Or, you can edit the JSON template directly on the web browser.
You can create multiple custom keypoint templates in a single project.
3. Custom Keypoint Auto-Labeling
You can create a Custom Auto-Label model that is trained on your custom keypoint labels. Label your data using the custom template, export the labeled dataset, and start training the Custom Auto-Label model. ❗ You can apply Custom Auto-Label when the keypoint configuration settings for the custom auto-label and the target object class are the same.
1. Google Cloud Platform (GCP) Integrations You can now access data stored on your Google Cloud Platform. 1-1. Cloud Upload through Integrations
To use the Cloud Upload function, you must first add GCP from the integrations menu.
Please refer to the tutorial below for detailed instructions on adding a new integration and uploading data from your GCP storage. Google Cloud Platform →
1-2. Enhanced Data Security with Read-Only Upload
On your cloud storage, you can choose to give read-only access to your data assets. When you turn this option on, the data remains solely within your managed storage, and no copy of the data will be created on Suite.
Restrictions:
GCP Integration is currently limited to image data only.
There are some restrictions on the use of ‘Raw Data Download’
A retrieval cost applies when you request or read data stored on your cloud storage.
2. Frame-Level Categorization
For video projects, you can now set up the annotation type ( image category/property) per frame on the Project Configuration page.
You can select frame-level properties for each object ID.
3. Analytics - ‘Dataset’ filter You can now find analytics by dataset on the Analytics tab.
When uploading data to a project, you will be notified of the connection failure with an alert if your data is a duplicate or does not match with the project type.
The position of the icon that indicates the difficulty level of Auto-Label has been changed as below.
1. New Image and Video Annotation App (Beta) The annotation app for video and image projects has been updated.
Custom keyboard shortcuts You can now customize your shortcuts as you like. If you want, you may revert back to the default key settings. Click on the keyboard button in the upper right corner to set up the shortcuts.
Resizing annotation elements You can resize each annotated area (left and right panels and timeline) within the annotation app.
Non-stop labeling mode
The non-stop labeling mode has been added.
With this mode enabled, you can immediately work on the next label after submission.
You can select or clear the checkbox, ‘Continue?’, at the top right corner of the annotation app.
2. Multiple Categorization
You can now perform multiple categorization tasks within a single project.
On the Project Configuration page, you can set up the desired response method among multiple choice, multiple selection and free response.
1. AWS (S3) Integration You can now access data stored on your AWS S3 cloud storage. To use the Cloud Upload function, you must first add S3 from the integrations menu. We currently support integration with AWS S3. If you need any further help with other cloud integrations, contact us at [email protected].
1-1. Cloud Upload through Integrations
In order to upload data from a cloud storage such as AWS S3, your account first needs to be integrated with the cloud storage.
Once the integration is complete, you can easily access data on your cloud storage without having to enter the Access Key and Secret Key each time.
Please refer to the tutorial below for detailed instructions on adding a new integration and uploading data from your integrated cloud storage.
1-2. Enhanced Data Security with Read-Only Upload
You can now upload data on your cloud storage (such as S3 bucket), and choose to give read-only access to these files. When you select this option while uploading, your Superb AI Suite account will only have read-only access to data stored in your cloud storage. This means that your data will remain solely within your managed storage, and you can always remove the integration to disconnect access.
There are some restrictions on the use of Raw Data Download
Please refer to the user manual below for a step-by-step guide to upload data.
2. Custom Auto-label
Cross project function You can now apply custom auto labels that you created with one dataset to another project with different datasets.
3. New Video Annotation Type New annotation types (keypoint and image category) have been added, and there are now four annotation types available for Video projects.
1. Cloud (AWS S3) Upload
2. Video Cloud Upload
Import mp4 file You can now import mp4 files when using the video Cloud Upload function. Once you select the desired frames per second (FPS) on the upload page, the files will be converted into image sequences before they are uploaded. Notes
The size of the mp4 file that you can upload is limited to 100MB, and the video length is limited to 5 minutes.
Please note that each frame is considered as one data asset, so please choose your FPS carefully. (If a video data contains 100 frames, it will be counted as having used 100 data assets)
3. Video Auto-label
You can now apply Auto-Label to projects that contain video data.
Auto-label for Video Projects include object tracking function. Auto-label AI will detect and track objects throughout the video and automatically provide unique object tracking IDs.
Like projects that contain image data, you can configure Auto-label on the Project Setting page.
You can then use the feature by clicking on the Auto-label button on the Label List page.
Please note that when applying Auto-label to videos, you'll be metered based on the number of frames Auto-label is applied to. (Each frame is considered as one data asset)
4. User Statistics & Report
You can now view the total number of object annotations per each annotation type. ❗ You must click on the Update Now button to view the latest statistics. Simply refreshing the page won’t work.
When you press the button at the top-right corner of the Annotations Summary chart, you will see the types of annotation in the table header. Click on the download (↓) button to get an excel file with the annotation types abbreviated as follows:
(B): Box, (P): Polygon, (KP): Keypoint, (PL): Polyline, (TB): Tilted Box
Usability Improvement: Label Filtering You can now apply filters a lot easier and faster and get the search results immediately, without having to click on a button start the filter action.
You can now find statistics by user in real time! On the Analytics tab > User Analytics menu, click on the Update Now button to update the statistics information.
You will be notified whenever a new version is updated. When the new version is updated, you will see the following message. Click on the Update button to refresh the page.
We’ve improved the toolbar for the video annotation app and the user interface of the Issue Thread.
Raw Data Download
You can now download the raw data that is used in an image project.
Go to the project setting page to download.
❗ Only the Owner & Admin-level users can access this feature.
Project Duplicate
You can now duplicate the selected project with the same configuration.
Go to the project setting page to duplicate.
❗ Only the Owner & Admin-level users can access this feature.
❗ Note that the project’s dataset does not get duplicated.
Label Assignment (Proportional Distribution)
You can now select an option of “Proportional” to distribute an equal number of labels to the selected users.
Delete Users (in bulk)
You can now delete users in bulk
New Filter Option
Tag > Contains only: filter out labels with selected tags
Added Features for Video Work App
Merge: If you labeled the same object with different IDs in more than two frames, you can choose to merge these into a single ID.
Split: In case you labeled different objects with the same ID, you can choose to split them into different IDs.
Undo
There have been various bug fixes
Video Data Projects
You can now annotate video data by using polygon segmentation.
We've added Class List, issue thread page, label information, object property from the Detailed View Page.
New shortcuts on Video Annotation App
Copying a single annotation from the nearest frame to the selected frame
d : selected annotation
shift + d : All annotation
Moving from frame to frame
, : previous frame
. : next frame
Assign Feature
Users can adjust the number and the order of labels when assigning multiple labels to multiple users at once.
When the data-project matching process fails, you will see the following message:
.png?alt=media&token=f74512b5-9211-4b6e-b922-09e387f9a4f3)
A filter option has ben added to the label list
Open Issues > by any one of: filter based on the users who created issue threads
Auto-label Request > is completed: filter based on labels that are annotated with the auto-labeling feature
Export → Fixed an issue of an export failure when the number of labels exceed 100,000
User stats → Fixed an issue of an User Statistics Export failure when the number of labels exceed 100,000
Video Object Tracking Beta Version
You can annotate video data on Superb AI Suite. Select “Video” as a data type when you are creating a new project.
Uniform distribution feature (labels)
Now, you can assign an equal number of labels to selected assignees. To do so, follow the steps below. 1. On the label List, select the labels to assign and click the ASSIGN button. 2. Select the members to assign labels. 3. Select the Assignment order. 4. Lastly, confirm the distribution settings and click the Apply button to complete the process.
Billing
With the integration of StripeStripe integration , paid users can now be added.
Updated User Statistics
User Statistics of video data projects has been added.
For the image data projects, the time spent(sec) entry has been added to the user statistics.
SDK Status filter has been updated.
Fixed an issue while updating labels with existing tags.
The location of the spirals are modified when they are loaded from the Analytics Project Tab.
User Statistics
On the Label List tab, you can request user statistics of selected labels by clicking the
USER STATSButton. This feature will display the summary of data labeling work per user and it will be sent out via email. From the Analytics tab > User Analytics page, you can request user statistics for all labels viaREQUEST USER STATSbutton. This feature will display the requested data on demand in the table format. You can see the latest version of your data anytime by just clicking theUpdate Nowbutton.
Data Upload via S3 You can upload your data to our Suite directly from AWS S3.
Minor Web UI bug fixes
You can list data which are not assigned to any project on our Data List page.
You can edit label tags directly from Detailed View.
Auto-labeling for Segmentation
We've added polygon segmentation support to our Auto-labeling AI!
Our Auto-label can annotate around 100 common objects using bounding box and polygon segmentation.
Here's our recommended workflow: 1) run
auto-label→ 2) sort your images based on "difficulty" (measured by our auto-label) → prioritize on reviewing the ones with high difficulty score.With this, you can boost the time to training by 9.7x!
Sample Project
When you first create an account, you'll now see a sample project with annotations, label tags, and various other features demonstrated for you to browse through!
We've made huge improvements on our latency around the globe, thanks to our new server locations!
Improvements to our Analytics page UI and many more UI and performance improvements!
Auto-labeling
Finally, auto-labeling is coming out of Beta testing!
Using our proprietary technology, such as Bayesian Deep Learning and Uncertainty Estimation, our Auto-labeling engine can annotate around 100 common objects using bounding box (polygon segmentation coming soon).
Not only that, auto-labeling engine scores each image in terms of how difficult it was to label it (High, Medium or Low difficulty). With this information, you can focus on reviewing and verifying the images that are difficult to label.
Overall, our Auto-label can boost the time to training by 9.7x! 🚀🚀
We've added a search bar to the Project List page!
We've added a cumulative vs. daily option on the "Submitted Labels" analytics chart.
And many more UI and performance improvements!
Edit project configurations
You can now make changes to project configurations after creating it. Whenever you have a need to add additional object classes, attributes or delete them, head over the Project → Settings → General menu.
Please note that you may not change the data type (image, video, text). Also, labeled data may become incompatible if any existing object classes or attributes are deleting or modified in the project configurations.
Export Progress
Large datasets, especially those with segmentation masks, can take a bit of time to prepare an export. You can now see when an export task will be completed!
Command-line Interface
Finally, after a rigorous beta-testing period, we've publicly released our command-line interface (CLI)!
You can upload or download both raw data and label (JSON) files using the CLI, and also check on the project progress and many more.
Head over to the My Account → Advanced menu to first create an Access API Key, and you can read more about it on our Github page.
Exporting labeled dataset is now 10 times faster!
UI improvements on the
Analytics PageAlso shipped minor bug fixes and performance improvements!
Data Analytics
Once you select a project from your Project List, you'll now see the "Analytics" tab. Here, you can grasp at a glance the project progress, statistics and distribution of your dataset as well as assignee-related information.
You can also download the raw data as a CSV file, or download the chart as an image!
Image Filter
Whenever any part of your image is too dark, i.e. night scenes, satellite imagery, etc., you can now apply brightness and contrast filters to help you annotate more efficiently.
Improved UI and bug fixes.
UI improvements on the
Details Viewand when creating a new project.Also shipped minor bug fixes and performance improvements!
Suite User Manual
Our user manual is finally here! More resources, including tutorial and use cases are coming soon.
Redesigned Export tab
We've redesigned our Export tab. We now have a sample export download and detailed explanations on what to expect when you export your labels — such as the file format and folder directories.
Resend activation email
Whenever an invited user loses the activation email, you can now use the Resend Email button. Go to your My Account page → Account Members menu, and you should see the message "Pending activation" next to users that have not activated yet.
Improved UI and bug fixes.
Performance improvements on our Annotation App!
Also shipped many UI improvements and minor bug fixes on the Annotation App!
Invite an admin user.
You can now invite a user as an Admin. Admin users can do everything except for billing management. (Available for Enterprise Plan only. Contact us for more information!)
Support user access
In your account settings (My Account → Advanced), you can toggle access to Superb AI's support user. Whenever you need help with your account (i.e. setting up a new data labeling project, etc.), we will ask you to toggle this on so that we can access your account. Don't forget to turn it off once we are done!
Use @tag to mention a user
When you create a new issue thread or reply to one, use the @tag to mention a user in your account. A mentioned user will get a notification.
Integrate with Slack to receive notifications
Head over to your project settings (Settings → Integrations), and you will be able to integrate with Slack! As of now, all issue thread related notifications (new issues, replies, @mentions) will be sent to both your web and Slack. Note that each project can be integrated with one Slack channel only!
Improved UI and bug fixes.
Everything loads faster now! We've reduced image loading speed by 400ms in the US Region.
We can now handle larger data exports! Go on and export more data at once!
Account avatar.
You can now add a profile image in your account settings page. This image will appear alongside your name throughout the Suite.
Advanced label tag filter.
Now you can filter labels better with our advanced tag filter. We originally had contains all of and contains any one of filters and we've now added three more — does not contain any one of, is empty, and is not empty.
Improved naming rule.
You can now use capital letters, numbers, spaces when creating a new label tag.
Many other UI improvements on the
Annotation App!
Issue notification.
You will now receive notifications whenever a new issue is created on labels, or someone replies to issue threads you've participated in. Email and Slack notifications are coming soon!
Delete empty projects.
Now you can delete projects that you don't want anymore. Head to the Settings Tab on any Project Page, and you'll see a delete button. Please note that you can only delete projects that are empty (i.e. that doesn't have any labels)!
Improved UI and bug fixes.
Improved Annotation App toolbar UI and shortcuts.
Many other bug fixes on
Label Tabfilters, user activation email and non-english language on user profile!
In order to provide the best possible user experience, we are sending out invitation emails in batches. Please sign up to our waitlist and we promise we will reach out to you as soon as possible!
Once you complete our onboarding call, you can access the free tier of Superb AI Suite by logging in on our website.
Add existing data to project.
Go to the
Data List Taband use theEdit Project Assignmentbutton to aadd existing data or dataset to a project.Delete data from your account.
Again, on the
Data List Tab, select the data or dataset you want to remove from your account and click on theDelete Itemaction button. Be careful as you cannot restore any data you've deleted!Annotation App Recovery mode.
If something happens (i.e. your network is down, etc.) while you are working on the
Annotation App, your work in progress can be recovered!
Improved UI.
You can create a new project from the dropdown menu at the upper-left corner of the
Project Page.Performance improvement on the Annotation App.
Now you can move around images inside the
Annotation Appwhile pressing the spacebar much faster!Huge improvements in the
Annotation AppUI/UX and looks!5+ other bug fixes!
Delete labels
You can now delete labels inside each project. Inside the
Label Tab, select the labels you would like to delete, and click on the Delete action button.
Improved UI.
You can move between projects more easily. Choose from the list of projects on the dropdown menu at the upper-left corner of the
Project Page.Project Configuration is now shown on the
Overviewtab.Lots of
Annotation AppandLabel Tabmodal view UI improvements.Bug fixes.
Disabled default browser action when using trackpads in the
Annotation App.Fixed the undo, redo bug in the Video
Annotation App(beta).10+ other bug fixes!
Improved
Detail ViewUI.The annotation ID # is now visualized along with other annotation properties.
Toggle which annotations are visualized by Class Groups.
Other UI Improvements.
When creating a new custom project, you can now edit the name of each keypoint in the keypoint template page.
Better support for non-standard screen dimensions.
Filters are now limited to a maximum 30 options to guarantee performance.
Bug fixes.
6 bug fixes in the
Work App.
Create a custom project.
You can now create custom projects for image data. We currently support the following four annotation types — bounding box, polygon segmentation, keypoint and polyline.
Directly download individual label files.
Open up the
Detail Viewfor a label and click on the "Download" button to download a single label JSON file.
Export a set of labels.
Apply any filter or search, select which labels you want to export, and then click on the "Export" button. You can download them when the export is completed.
See export history and download exported labels.
Each export job will be in one of these five statuses — In Queue, Processing, Ready for Download, Canceled and Failed.
When an export job is completed (Ready for Download), you can download a zip file containing the label JSON files.
Improved UI.
Adjust the location of each annotation in
Work Appusing keyboard arrow keys. Added "Q" shortcut to toggle label visualizations.OverviewandLabel Viewnow loads faster.Bug fixes.
Multi-language support for guideline document file names in
Overview.
Overview dashboard for every project.
When you click on a project in the
Project List View, you'll now see theOverviewpage — a dashboard summarizing the status of each project.
Checkoff from a To Do list
Shows a list of common To Do's for setting up a project — i.e. uploading data, guideline documents, and inviting project members. They are checked off when completed.
Useful links for managing your project
We've selected a few actions that our users commonly use (i.e. applying common filters). You can click these to instantly apply filters.
Check the status of labels
Admin and Manager users can see the status of entire labels in the project. Worker users can see the status for the labels they are assigned to.
Upload project guideline documents
Admin and Manager users can upload guideline documents (pdf, etc.), and all users can download them.
See the number of users involved in a project
Check how many users are invited to your project, broken down by roles.
Filter by project assignment
Inside
Data View, select the dataset name and project name to see the list of data assets
.gif?alt=media&token=df8065d8-d998-4798-a7a1-06114697ad2d)
Edit project assignment for each data.
Inside
Data View, select the data you want to add to a project and click on the "Edit Project Assignment" action button.
Improved UI.
Data key (name) search is now case-insensitive.
Changed cursor icon to improve visibility.
New version updates are now automatically loaded without refreshing the page.
Bug fixes in
Label ViewAssigning label search results used to assign all labels in a project. This bug is now fixed.
View and manage data assets on
Data ViewYou can move to the
Data Viewfrom the navbar on your account page. Here, you'll see the list of datasets you've uploaded. Click on each dataset to view a list of data assets.
Filter by project assignment
Inside
Data View, select the dataset name and project name to see the list of data assets
Edit project assignment for each data.
Inside
Data View, select the data you want to add to a project and click on the "Edit Project Assignment" action button.
Improved UI.
You now select a dataset (or create a new dataset) when uploading data assets.
Bug fixes and performance improvements in
Detail View
Create an issue thread in
Work App.Click on the Issues button on
Work App(located at bottom-left) and click on the image area to create an issue.Filter labels by open issues.
See labels that have (or don't have) open issues. We've added a new column in
Label Viewto show the number of open issues for each label.
Add tags to labels.
Create custom tags and add them to selected labels.
Filter labels by tags.
See labels that contain
all of(AND condition) orany one of(OR condition) the tags you choose.
Improved
Work Appperformance.It doesn't slow down when you move the image around or zoom in/out.
A lot of bug fixes around creating and editing annotations.
Improved UI.
Long data key (name) is no longer shortened as "..." and you can see the entire key by side-scroll.
Added more shortcuts to
Work App.
Create an issue thread.
Move to the Issues tab on
Detail Viewand click on the image area to create an issue.
Resolve and Re-Open issue thread.
Change the status of each issue thread by resolving or re-opening it.
See the list of issue threads for each label.
You can see the list of issue threads and their status in
Detail Viewby clicking the Issues tab.
Improved
Work Appperformance.It doesn't slow down when there are a lot of annotations added to a label.
Improved UI
User invitation menu now shows a list of existing users.
Assignee: "none" now shows up as "-".
Separated
Change StatusandAssignactions into two buttons.User list now shows each role groups separately.
