A draft reporting standard for AI agent calibration.
VACRA.AI publishes the Calibration Reporting Standard (CRS): a 13-field record of whether an AI agent's stated confidence matches how often it is right, for financial services.
Why this draft exists
SR 26-2 (Federal Reserve, FDIC and OCC, 17 April 2026) states that generative AI and agentic AI models are not within the scope of its guidance (footnote 3). It adds that a banking organization's risk management and governance practices should guide governance of such tools.
CRS is one author's proposal for a calibration reporting format.
What the Calibration Reporting Standard defines
CRS defines a 13-field record of an AI agent's calibration over a reporting period, compiled by the deploying institution and completed by an independent third-party attester, and a rule that determines one of three compliance states from the calibration rate and the expected calibration error (ECE). The record format and the rule are set out on the specification page.
Calibration is the match between an agent's stated confidence and how often its outcomes are correct. ECE is a standard way to measure it (Guo et al., 2017).
The reported calibration rate and ECE meet the CRS thresholds for the reporting period. This is not an assessment of safety, security or suitability for any use.
One or more reported values fall outside the COMPLIANT range but inside the remediation range. The specification sets a re-attestation window.
The reported ECE is above the upper threshold. The specification sets an emergency re-attestation window.
Enquiries
Questions and comments on the CRS draft are welcome. No paid service is offered. The author has a commercial interest in CRS adoption.
Email the authorcontact@vacra.ai