Nref HW points on the ground plane for a top view query located at (h, w) in the TPV. These reference points are used to construct the deformable convolutional kernel, which is then applied to the image feature maps to extract relevant information for the query. The resulting feature map is then aggregated by taking the weighted sum of the features at the sampled reference points, where the weights are learned through a separate attention mechanism. This process is repeated at multiple scales and possibly multiple cameras to generate multi-scale, multi-camera features that capture rich spatial and semantic information from the input images. Overall, image cross-attention enables TPVFormer to effectively leverage visual information to enhance object detection and tracking in complex 3D environments.
