Real-Time Object Tracking on the Nexys A7-100T

Real-Time Object Tracking on the Nexys A7-100T (Hardware accelerated)

In the robotics field, visual tracking systems are traditionally implemented using high level software running on a CPU or GPU. While software solutions are flexible, they suffer from sequential processing bottlenecks and operating system overhead, leading to non-deterministic latency.

To demonstrate the massive advantage of hardware level parallel processing, I designed and implemented a real-time, closed loop object tracking system entirely on a Diligent Nexys A7-100T FPGA. The system utilizes a custom computer vision pipeline to identify a distinct target (a tennis ball), calculate its centroid, and drive a 2-axis pan-and-tilt servo gimbal to keep the object centered in the camera's field of view.

System Architecture & Pipeline

The architecture is designed as a continuous, real-time data pipeline divided into three main clock domains: a 100 MHz board clock, a 24 MHz camera clock, and a 25 MHz VGA/DSP clock.

• An OV2640 CMOS camera streams 16-bit RGB video, which is captured and concatenated into a 12-bit format. Because the camera pushes data asynchronously, the pixels are written into a dual-port Block RAM (BRAM) acting as a buffer between the camera's write domain and the VGA's synchronous read domain.
• As pixels are fetched from BRAM for the VGA display, they are immediately routed into the DSP pipeline for digital white balancing and color filtering. Every pixel is analyzed and processed the exact nanosecond it is received.
• A centroid calculator accumulates the coordinates of positively matched pixels within a strict bounding box (Region of Interest) to pinpoint the object's physical center of mass.

RGB to YCbCr

Tracking an object based purely on raw RGB is highly susceptible to shadows and glare. Perfect color tracking is typically done in the HSV color space; however, converting RGB to HSV in hardware requires complex division and floating-point logic. Initial attempts at HSV conversion took 3 clock cycles, causing timing mismatches that overwrote pixel data in the BRAM.

To resolve this and maintain my 1-clock-cycle constraint, I implemented a custom Luminance/Chrominance (YCbCr) color space conversion.

By calculating the distance of the red and blue channels from this base luminance, the hardware isolates the target color while actively ignoring skin tones and background reflections.

The control loop and mechanical latency

The calculated coordinates are passed to a gimbal tracker that acts as a hardware proportional controller. The absolute error distance between the object and the screen's center is calculated and bit-shifted to determine dynamic movement steps.

A major system limitation was the physical speed of the SG90 servos. While the FPGA calculates the object centroid at 25 million times per second, the physical servo requires approximately 100ms to mechanically rotate. If the PWM target updates faster than the servo can move, the system overshoots. To prevent endless mechanical oscillation, a 24-bit timer forces a 120ms wait state in the control loop.

Future improvements

• Predictive Tracking: Adding a derivative term to the proportional controller would allow the system to calculate the velocity of the target, helping to predict fast-moving objects.
• Edge Detection: Integrating a Sobel edge filter to identify the shape of the object would drastically improve tracking reliability.
• Depth sensing: Incorporating an ultrasonic or LiDAR sensor to capture the Z-coordinate of the object for full 3D spatial awareness.

Github Repo: https://github.com/CarlosT25-png/object-tracker-fpga-a7100t

Live Demo