NEEDLE
Orchestration Framework and Toolkit for the deployment of Neural Simulation-based Inference (NSBI) methods in High Energy Physics (HEP) at the Large Hadron Collider and beyond.
Simulation-based Inference at scale for Particle Physics
NEural-basEd Diffusion Likelihood Estimations (NEEDLE) is an inter-experiment research project, aimed at developing a computationally efficient training and inference toolkit (the NEEDLE Framework), and develop new thermo-equilibrium inspired models (diffusion/flow matching library).
The NEEDLE Framework provides scalable workflow orchestration for the training of NSBI models on large computing infrastructure. We achieve this by harnessing a DAG workflow that atomizes the training of each neural surrogate into individual Tasks. These tasks are unique, reproducible and provide automatic checkpointing of each step. In addition, trainings are submitted to batch systems with support for htcondor and slurm.
Research Focus
Tools that help NSBI methods scale to large-scale datasets within the HEP ecosystem and concentrate on diffusion/flow-matching models for neural likelihood, posterior, and likelihood ratio estimation.
Infrastructure
Our DAG-based orchestration system is designed for reproducible, scalable ML workflows. Our architecture supports distributed training, efficient data pipelines, and seamless integration with LHC computing resources.