2025 | Laboratory for Analytic Sciences (LAS)

Faster, Smarter, User-Aligned: EvalOps and the Future of Integrated Evaluation for the IC

The four EvalOps phases, each of which has a corresponding worksheet that can be used in a modular way.

Abstract

The development of AI-enabled tools for the Intelligence Community (IC) faces a persistent challenge: while technical systems and Machine Learning models advance rapidly, ensuring those tools reliably enhance analyst workflows and operational effectiveness remains complex and time-consuming. EvalOps addresses this challenge by extending MLOps principles to focus on rapid, scalable, and systematic evaluation of human-machine interfaces. Rather than treating evaluation as an afterthought, EvalOps embeds stakeholder feedback, usability assessment, and iterative testing from the start of system development and sustains it throughout. During SCADS 2025, we expanded EvalOps through a series of four structured design worksheets that guide project teams through the process of integrating evaluation from the earliest stages on. Combined with multi-level instrumentation and targeted empirical studies, these efforts enabled faster interface refinement, improved alignment with analyst workflows, and accelerated time-to-deployment across operational contexts.

BibTeX

@article{shrestha2025faster,
  title={Faster, Smarter, User-Aligned: EvalOps and the Future of Integrated Evaluation for the IC},
  author={Shrestha, Bijesh and Shrestha, Hilson and Bonilla, Karen and Harrison, Lane T and Crouser, R Jordan},
  year={2025},
  publisher={Laboratory for Analytic Sciences}
}