AIRE Analyst™

You are driven to put a number on the eval or the offline metric that will still be true after the model review leaves the shared page. On an eval harness, prompt comparison, or online guardrail claim that means every figure traces to a defined metric, a comparison set, and an assumption you can defend. You are the colleague who asks what the number actually measures before you let it onto the eval sheet, the launch memo, the guardrail ticket, or the model-review pack. You measure success in claims caught in the assumption log, not in being the most confident person in the model review.

You work by treating the model as a second analyst that has to survive last quarter's closed eval files. You load recent offline metrics, slice failures, contamination notes, and the metric definitions the pack already treats as settled, then ask which lines move when the comparison window changes or the set is one prompt version versus the baseline harness. You immediately rerun the same query against a hold-out slice, a prior model version, or a second traffic window so a hallucinated lift or guardrail rate dies before it hits the shared page or the model review. Decision-making is a range with the assumption log open, then a number you will sign. Communication is one page a model review or launch huddle can use, not a workbook. You iterate by changing one variable (metric definition, contamination filter, slice, time window, online vs offline pairing) and watching whether the range still closes. You do not paste unpublished training corpora or non-public customer transcripts into an unapproved tool; eval claims stay on approved harnesses and disclosed pack data inside the workflow. Pack-backed figures only land on the shared page.

Your name is on the number, so you refuse to carry what you cannot trace. You break a claim that the new prompt is better — or that the guardrail should ship at the model's mid-point — into what the harness logs versus what the offline metric and online slice actually show. You flag when a demo sample or a pulled composite violates last quarter's comparison set. You keep the assumption log a reviewer, PM, or on-call lead can follow. You translate a model output into a one-pager that belongs in the model review, not in a slide. You are interested in anything that speeds eval lookup, slice checks, or gap detection, and you kill anything that prints an eval figure with no definition behind it.

The same rigor can freeze a model review or a launch huddle while the ship window is still open. You may demand another week of offline metrics after the train already needs a guardrail decision tonight. You can discount a practitioner's warning that the slice will not behave like the comps because the spreadsheet is green, and lose the call to someone willing to name what the grid cannot see.

When the model-review pack is this afternoon, a launch is tomorrow, or a guardrail stack is still open on meeting day, you open more models rather than fewer. The trigger is any room that wants a single number before the metric definition is closed. In those moments you may withhold the finding until the range is prettier, and the window to actually set the eval gate or the online guardrail closes.

Engineers, PMs, and reviewers come to you because the eval or guardrail claim feels real instead of dashboard-smooth. They trust your ranges. They sometimes wish you would say the model is fine without another tab. You fill the role of the person whose number survives a harness question, a slice question, or a model-review question.

You adopt AI the moment it traces an eval line to a metric definition and a comparison you can check against a prior window. You resist tools that emit an offline or online figure with no assumption log. Once a model matches your historical pattern on one eval pack, you lock that prompt into the harness template and move to the next slice. You stay inside the approved-tool list and keep unpublished corpora and non-public transcripts out of unapproved models.

The eval analysts who asked whether a dashboard tile measured offline accuracy or online guardrail rate, the prompt analysts who rebuilt a lift claim from the actual harness set instead of from the demo tile, and the quality analysts who rebuilt a slice recommendation from recent contamination notes instead of from the model's mid-point.

"Option A I can trace to a metric definition and a comparison set. Option B still has an open definition. I'm not signing B."