Marker-free Real-time Needle Tracking for CT-guided Procedures Using RGB Cameras

Overview

This study presents a novel marker-less, real-time needle tracking system utilizing low-cost RGB cameras and a U-Net-ConvNeXt architecture, achieving clinically accurate tracking across diverse needle types and challenging conditions. The system outperforms traditional marker-based methods, especially under occlusion and needle deflection, and is supported by a publicly available, fully annotated multi-view needle dataset.

Background

CT-guided interventions require precise needle placement for biopsies and tumor ablations, but needle tracking is limited by radiation exposure and imaging speed constraints. Existing tracking systems rely on markers that are prone to occlusion and require complex setup, limiting their clinical utility. Marker-less tracking offers a promising alternative by eliminating markers and enabling tracking of arbitrary needle geometries without modification. However, prior approaches have struggled with real-time performance, accuracy, and generalizability to diverse needle types.

Data Highlights

The tracking system uses Logitech StreamCam webcams (1920×1080, 60 FPS) calibrated with a 5×7 ChArUco board achieving mean reprojection errors of 0.46 ± 0.12 pixels. The dataset includes over 35,000 segmentation-mask-annotated frames captured simultaneously from two to three cameras at 60 FPS, covering multiple needle types and environmental conditions. The system achieves sub-pixel accuracy in tip and base localization and employs a Kalman filter and RANSAC-based triangulation for robust 3D tracking.

Key Findings

• The marker-less system achieves clinically accurate real-time needle tracking without requiring instrument modification or system recalibration.
• It outperforms traditional infrared and image marker-based systems under static, dynamic, occlusion, and needle deflection scenarios where marker-based systems fail.
• The approach generalizes well to previously untrained needle types, demonstrating robustness across diverse geometries.
• Multi-view RGB camera configurations and a hybrid U-Net-ConvNeXt architecture enable sub-millimeter tracking accuracy with low-cost hardware.
• The publicly available multi-view annotated dataset facilitates further research and development in needle tracking for CT-guided interventions.

Clinical Implications

This marker-less tracking system simplifies procedural setup by eliminating the need for markers and calibration, reducing workflow complexity and potential tracking interruptions due to occlusion. Its ability to track diverse needle types in real time enhances guidance accuracy during CT interventions, potentially improving procedural safety and outcomes. The low-cost hardware and open dataset support broader clinical adoption and innovation.

Conclusion

The proposed marker-free, real-time needle tracking system represents a significant advancement for CT-guided interventions, providing accurate, robust, and generalizable tracking without the limitations of marker-based approaches. Its integration into clinical practice could improve needle guidance precision and procedural efficiency.

References

1. Hein et al. 2023 -- Marker-less surgical drill tracking using RGB cameras
2. Agethen et al. 2022 -- RGB-only tracking with recurrent neural networks
3. Hasan et al. 2021 -- Integrated detection and pose estimation for needle tracking
4. Trojak et al. 2020 -- Mixed reality system for CT-guided biopsies
5. Logitech International S.A. -- StreamCam Webcam specifications
6. ChArUco board calibration method [10]
7. Public multi-view needle dataset [7]