Auto-Annotating UI Elements for VOD Apps at Scale

The solution was designed around two core components: computer vision models and active learning.

The client was facing a major bottleneck in their software testing process due to the manual annotation of UI elements across applications.

The old Comcast process was not only labor-intensive but also inefficient, especially given the need to support multiple devices with varying aspect ratios. The sheer amount of time spent manually tagging UI elements hindered productivity and delayed testing schedules.

1. Computer Vision-Based Pre-Annotation: The tool utilized advanced image recognition models to detect and display specific UI elements. We trained a selection of open-source models, including:

YOLO (You Only Look Once) network - a highly efficient real-time object detection system.

DETR - a transformer-based object detection model from Meta.

These models were embedded into an AWS serverless automated testing framework, allowing the tool to automatically classify and annotate predefined UI elements.

2. Active Learning for Continuous Improvement: To maximize the tool's accuracy and adaptability, we implemented an active learning approach. This strategy enabled the human testers to:

• Review the pre-annotated labels.
• Make corrections when necessary.
• Re-train the model based on those corrections, improving its performance over time.

By integrating this iterative process, the system became more effective at recognizing existing UI elements and capable of learning new ones, thus enhancing both the accuracy and scalability of the solution.

The automated annotation tool significantly improved Comcast's software testing workflow, delivering the following outcomes:

• Pre-annotation accuracy of 92%, drastically reducing the need for manual intervention.
• 5x-10x reduction in human workload, speeding up the testing process across different devices and resolutions.
• A scalable solution that continued to learn and improve through the active learning feedback loop, ensuring even higher efficiency over time.