Abstract
Consider: a company commits to mitigating targeted harassment. Three teams — Trust & Safety, fairness research, and red teaming — are each responsible for operationalizing this commitment. Yet none of their evaluations trace directly to the policy's harm definitions, and results rarely feed back into deployment decisions.
This gap is increasingly prevalent for AI-centric products. Unlike traditional software, AI systems produce emergent behaviors difficult to anticipate in policy language and harder to capture with static evaluations. The result is a sociotechnical gap: safety policies describe intentions, while evaluation pipelines operate on different assumptions, owned by different teams, with limited traceability between them.
Using a system safety lens, participants work in cross-functional groups to (1) draw the control structure that operationalizes a real policy commitment, (2) identify where policy-to-evaluation gaps live, and (3) generate the research questions and tools that could close them.
Agenda
- 010–30 min
Introduction & talk
Shalaleh & Renee
Welcome and overview; state of policy and evaluations (Renee); control-structure terminology with a quick worked example (Shalaleh); Q&A.
- 0230–65 min
Activity 1 — Connecting policy to evaluations
Shalaleh
Groups identify stakeholders and draw the control structure for the policy on their table (worksheet steps 1–4).
- 0365–100 min
Activity 2 — Identifying & resolving gaps
Renee
Groups locate policy-to-evaluation gaps, brainstorm ways to close them, and name the research questions or tools that would help (worksheet steps 5–8).
- 04100–120 min
Share back & debrief
Shalaleh & Renee
Groups share their control structure and one gap-closing strategy or research question they surfaced; connect findings, share next steps, and conclude.
What each activity produces
Activity 1 · 35 min
Connecting policy to evaluations
Groups work through steps 1–4 of the worksheet: identify the actors, establish power dynamics, and map control actions and feedback into a drawn control structure.
Activity 2 · 35 min
Identifying & resolving gaps
Groups work through steps 5–8: connect evaluations back to the policy, name the gaps, prioritize them, and propose closures — including research questions and tools that would help.
The worksheet, at a glance
The full 8-step map groups take through in the room — from choosing a policy to closing evaluation gaps.
Open full image in a new tab ↗Who this is for
Trust & Safety leaders, operations managers, policy professionals, ML engineers, evaluation researchers, and product managers working on AI systems — especially those responsible for implementing or overseeing safety policies who need greater clarity on how those commitments translate into technical and operational controls.
Ready to map?
The worksheet works offline. Your draft saves locally and you can export to Markdown or print as PDF.