How to outsource image annotation

If you are looking to outsource image annotation, there are many points to consider before contacting image annotation companies. With an influx of AI training data companies, choosing the best one for your project can be a difficult task.

As a provider of image tagging services, TELUS Digital has the experience and knowledge that can help save you from unnecessary headaches when outsourcing training data. From standards of quality to platform considerations, below are five basic tips on how to outsource image annotation.

1. Examine workbench demos or data samples

Evaluating the company background is one of the most rudimentary, yet important things to do. Of course, you should research the company’s proven experience in the market. However, since AI training data companies often provide multiple services, you should look for proof of their image annotation capabilities. Browse the company’s site and look at the quality of the photos, case studies and solutions. Most of the leading image annotation companies will provide graphics, gifs, or videos to demonstrate their services.

2. Decide how you want the data verified

Based on the needs of your project, it is important to accurately identify how the service will be handled. One element you should decide beforehand is how you want the image annotations to be verified.

A good example of this is image classification. With image classification, you task annotators with selecting the appropriate label for each image. However, with human-powered classification there is a chance of human bias affecting the data. To combat this, there are multiple levels of verification you can implement:

• Single pass: one annotator labels the image.
• Double-blind verification: two annotators label the image without seeing each other’s work. If the labels don’t match, a supervisor observes the case and decides the correct label.
• Multi-blind verification: three or more annotators label the image without seeing each other’s work. If the labels don’t match there are two options: consensus or adjudication. With consensus, majority rules and the most common label among the annotators is chosen. With adjudication, a supervisor is brought in and decides the correct label.

While different companies may use different terms, these are the standard systems used in data entry and annotation. Stricter verification will take more time and may come with additional costs. Be sure to decide which verification method is best for your project and budget.

3. Define your standards of quality

Aside from verification methods, you must also clearly dictate your standard of quality. Many companies will say they provide “accurate” training data. However, what does “accurate” mean to you?

For example: with bounding boxes, cuboids, and polygons, how precisely do you need these annotations to be drawn? Some projects may require annotators to zoom in to the very edge of the object and adjust the the annotation down to the pixel. Other projects may allow for a margin of error. The same goes for accuracy of the image data as a whole. Some clients require 100% accuracy in the data. Meanwhile, others may allow for 90% accuracy.

The important point is that you dictate these terms and establish guidelines before consulting with the outsourcing company. Give the company examples of what you consider correct image annotations, and incorrect image annotations. Explain the exact format or file types you require, how you want the data batched, and quality control systems you want implemented. The more information you give the company, the better the final product will be.

4. Determine who will be annotating your images

Every AI training data company has their own workflows, business model and staffing system. If you are annotating images used in a specialized field, it is important to determine exactly who will be working on your project. Does the company provide onsite staff, remote staff or both? How are the workers trained and do they hold certain qualification standards? You should ask these questions to every outsourcing company you speak to.

While some annotation tasks may not require specialist knowledge, the same cannot be said for annotating lesions or tumors in CT scans. If you require experts in a certain field to annotate your images, make sure the company has the resources to source specialized staff.

5. Decide on an image annotation platform

One important element to consider is whether the company has their own image annotation platform. Annotating on the company’s proprietary platform has many benefits. Firstly, the company owns the platform and can make changes to it if your project requires a specific feature. If the company crowdsources workers, their annotators should already know how to use their platform. Therefore, they should not require additional platform training.

On the other hand, you may have your own custom platform or a third party’s platform that you want to use. If you need your images annotated on an external platform, companies may charge extra fees to train their staff on that platform. Be sure to inquire about platform fees when getting a quote.

When outsourcing image annotation, you may run into companies that only provide the image annotation software. You may also find companies that only provide the staff. It will be much more efficient for you in the long run if you find a company that provides both.

Looking for more advice on sourcing annotated image data or need a quote for image annotation services right now? Get in touch with our sales team to learn more.