1. Insights
  2. AI Data
  3. Case Study

Enhancing an autonomous vehicle bot’s scenario understanding with tailored fine-tuning datasets

Discover how TELUS Digital assisted a leading U.S. autonomous vehicle (AV) developer in refining and improving their AV bot.

Visualization of the interaction of self-driving autonomous vehicles.
  • Share on Facebook
  • Share via email

100K
descriptive text annotations

95K
response rankings and ratings

98%
SFT quality score

95%
RLHF success rate

The challenge

Our client, an L4 and L5 autonomous vehicle manufacturer, needed fine-tuning data to refine and improve the decision making capabilities of their AV bot for various driving situations. The data, consisting of video recordings of the vehicle navigating real-world scenarios, required high-quality annotation. As their long-time partner, the client was familiar with our 2D/3D annotation expertise in the automotive sector, and ability to source experts with a thorough understanding of U.S. traffic rules. They leveraged our fully-managed data labeling solution for diverse and complex use cases to support their initiative.

The TELUS Digital solution

The fine-tuning process happened in two phases. First, supervised fine-tuning was done to train the AV bot to better comprehend driving scenarios and formulate adaptive motion plans, even in edge cases. To accomplish this, we assembled a team of 25 driving experts from the U.S. and the Philippines who generated a fine-tuning dataset of over 100,000 high-quality text annotations (responses). These annotations involved analyzing lidar and 2D data to describe driving scenarios in alignment with traffic rules, safety guidelines and human expectations. This required thorough attention to detail, including the precise identification of objects in the source video dataset, followed by the prescription of appropriate driving actions.

Throughout the project, we employed a comprehensive technical and linguistic quality-assurance rubric. This was used to evaluate categories such as vocabulary, fluency, mechanics, timestamp, location, reasoning, action, track identification and track behavior. Each annotation was graded as critical, good or great.

Second, reinforcement learning from human feedback (RLHF) was done to benchmark the trained AV bot’s responses against critical parameters to evaluate the vehicle’s performance. The AV bot’s responses were compared and ranked against the expert-generated responses in three categories: the action taken by the AV bot, the reasoning behind the decision and the scenario location. Complex rules and ranking priorities were applied to identify critical errors, resolve ties and conduct holistic comparisons for more than 95,000 individual response ratings.

Adding annotation to driving scenes video. Example annotation: “I am at an intersection with a considerable amount of vehicle and pedestrian traffic. I have stopped due to a red signal. A white SUV has passed in front of me (ID: xxxxx) and a pedestrian (ID: xxxxx) is crossing the intersection in the opposite direction. Immediately after, a bicyclist (ID: xxxxx) and another pedestrian (ID: xxxxx) proceed to cross the intersection in front of me as well. Additionally, the first pedestrian is crossing the road again to the opposite side. I can proceed after the bicyclist and the two pedestrians have crossed the road completely and the traffic signal turns green.”

The results

For SFT, we achieved an impressive quality score of 98% based on two criteria. The first was the customer’s linguistic rubric, which defined the quality of the vocabulary and clarity of the response. The second was the accuracy of the decision made, reasoning and the precise scenario location.

For RLHF, we achieved a success rate of up to 95% compared to the golden dataset provided by the client. We employed a hierarchical ranking system that equated certain options on a case-by-case basis to ensure accurate comparisons.

The high-quality datasets and response ratings significantly improved the AV bot’s scenario comprehension and motion-planning capabilities. They also enhanced the AV bot’s ability to handle edge cases, contributing to safer and more reliable autonomous driving technology.


Check out our solutions

Enrich your data with our range of human-annotation services at scale.

Learn more

Related insights