Partners: David Rolnick, Michael Bunsen, Francis Pelletier, Anna Viklund, Mohamed Elabbas

SSEC Engineers: Carlos Garcia Jurado Suarez, Bhagyashree Wagh

Research Goals and Domain

The Antenna project studies entomological biodiversity through cutting-edge technologies that are revolutionizing the way insect images are collected, identified, and monitored using AI-powered camera traps, machine learning algorithms, and scalable data platforms deployed across diverse ecosystems. Entomological biodiversity is crucial for healthy ecosystems due to their role in key factors such as pollination, decomposition, and food sources. The platform is designed to serve researchers and partner organizations conducting ecological studies, particularly those deploying custom detection and classification models. 

Software Problem

The current implementation of Antenna supports third-party machine learning (ML) models through a service API specification, requiring the platform’s underlying processing services (PS) to expose public endpoints. This architecture introduces several challenges. First, requiring public endpoints increases setup complexity and may be blocked by firewalls. Second, the push-based request-response model limits batch processing optimizations, such as asynchronous data loading. Finally, scalability burdens fall on the PS implementors, who must manage load balancing and concurrent request handling. These limitations hinder performance, extensibility, and ease of use, especially for large-scale jobs. 

Software Solution

SSEC is rebuilding the ML processing service, to allow PS greater capabilities and accessibility when they interact with Antenna job queues. The updated pull-based architecture eliminates the need for public endpoints and enables asynchronous, scalable processing. The system will support any ML framework (e.g., PyTorch, TensorFlow, R) and allows PSs to be deployed across multiple workers or multi-GPU/CPU systems. Authentication workflows ensure secure integration and job queue management; offering features like retries, backoff, and visibility timeouts. The API includes endpoints for listing jobs, pulling task batches, submitting results, and reporting failures. Reference implementations and starter frameworks are planned to simplify onboarding, including examples for Jupyter notebooks and command-line interfaces. 

Impact

The new backend architecture significantly improves scalability and reliability. Large jobs, such as those involving one million images, can be processed without blocking other services. The system supports concurrent execution across multiple PS, ensuring that long-running tasks do not impede smaller ones. The pull-based model reduces engineering complexity for third-party developers and facilitates broader adoption of Antenna across research institutions. This re-architecture will also enable cloud and hybrid deployments to allow for greater scale as well as reduce the operational cost. By enabling efficient, extensible ML processing, the platform advances the capabilities of ecological monitoring and collaborative model development by accommodating large datasets and diverse ML pipelines.