CCAR-P practice questions: Stakeholder communication and lifecycle management.

An executive asks whether to fund the agent system. What framing lands?

• Cost per completed task, measured reliability, the known failure modes, how each is caught, and what fixing them takes
• The eval harness design, benchmark scores against the previous model, and the token budget per task
• A live demo of the happy path with a projection of headcount savings once the agent reaches full autonomy
• The architecture diagram, the tool inventory, and the prompt techniques that produced the accuracy gains

A new model version is available. What is the responsible upgrade path for a production agent?

• Run the new model in shadow mode on live traffic for a quarter and adopt it only if no user reports a regression
• Run the golden eval set on the new model, compare against the current baseline, then roll out gradually with monitoring
• Pin the current model until the provider's deprecation date, then switch in one release with a rollback tag ready
• Point staging at the new model on release day and promote to production after a week with no complaints from the engineers using it

Which metrics tell you a production agent is drifting?

• Ticket volume from users, time to first response from the team, and the count of open bugs tagged to the agent
• Tool error rate, cost per completed task, latency, and eval scores tracked over time
• Token volume per day, API error codes, and the number of prompt versions shipped in the quarter
• Uptime of the agent service, p99 latency of the model endpoint, and the provider's status page history

When should a deployed agent system be retired?

• When the model it was built on is deprecated, since a migration costs about as much as a rebuild
• When the original sponsor leaves and no team is willing to take over on-call for it
• When its measured value no longer covers cost and upkeep, and a simpler alternative covers the need
• When its eval scores fall below the launch baseline for two consecutive review periods

A stakeholder wants "an agent for everything". What do you establish before building?

• Success metrics, the tasks in scope, and which decisions stay with people
• A pilot with the widest possible scope so that real usage reveals which tasks are worth keeping
• The full tool inventory the agent would need and the MCP servers that expose each system
• The model tier, the token budget per task, and the evaluation harness for the first release

A CFO asks 'what does this agent cost us per month and what happens if it breaks?' The right answer includes:

• Cost per task and per month, failure modes, how they are detected, and time to recover
• Total tokens per month and the provider's published availability figures
• Model names, token counts per task, and the prompt caching rate that keeps the monthly bill down
• A live demo plus a commitment to SLA-backed uptime from the model provider

Which metric set belongs on an agent's monthly review?

• Token volume, cache hit rate, and API error rate by endpoint
• User count, sessions per user, and feature requests logged
• Completion rate, quality score, cost per task, incidents, and time to recover
• Prompt versions shipped, eval pass rate, and model release notes reviewed

A stakeholder wants a feature that the evals show is unreliable. Your move?

• Ship it behind a feature flag so that it can be turned off quickly if users begin to complain about it
• Show the eval results, state the risk, and propose a gated rollout or scope reduction
• Raise the eval threshold and ship once a stronger model tier clears it
• Decline the request and escalate to the sponsor with the eval numbers attached

How should prompt changes be communicated to stakeholders of a production system?

• Only when the change alters user-visible behavior, via the release notes
• Through the change ticket that the engineer files when editing the prompt
• As versioned releases with eval results and a rollback plan
• In the monthly review, summarized as the number of prompt edits made

Which event should trigger a lifecycle review of an agent system?

• A model upgrade, a change in tool capabilities, or a shift in the task's inputs
• A new release of the agent framework or SDK that the runtime uses, since dependencies change behavior
• A change in the on-call rotation or the team that owns the agent
• A quarter end, so that reviews line up with the budget cycle

A client asks for a single number: 'how accurate is it?' What do you give them?

• A range spanning the best and worst eval runs from the past quarter
• Accuracy on a defined eval set with the date, the failure categories, and what happens on failure
• The eval pass rate as a single percentage, so the client has the number they asked for
• A comparison to human accuracy on the same task, drawn from the client's own quality assurance records for the past year

How do you set expectations for an agent's first month in production?

• A commitment to the eval accuracy figure as a production SLA from day one
• A soft launch to a friendly team with no metrics until usage has settled down
• Full autonomy on the low-risk tasks from day one and a promise to review the remaining tasks after 30 days
• A staged plan with metrics, human review levels, and the criteria for expanding autonomy

A stakeholder wants to skip evals to hit a launch date. Your position as the architect?

• Ship with production monitoring in place of evals, since live traffic is the more realistic test
• Hold the launch until the full eval suite passes, whatever that does to the date
• Agree, provided the stakeholder signs a written risk acceptance and the full evals run during the week after launch
• Explain the risk concretely, offer a reduced-scope launch that keeps the eval gate, and document the decision

Which artifact keeps a long-running agent engagement healthy across team changes?

• The chat history between the team and the client, archived and searchable
• The original proposal and statement of work, kept current through change orders
• A living runbook: architecture, gates, evals, known failure modes, and on-call procedures
• Recorded handover sessions from each departing engineer, stored with the project

When a model provider announces a deprecation, what is the lifecycle response?

• Switch to the successor in production at once, since the provider has already validated it
• Stay on the deprecated model until the retirement date to avoid churn, then switch
• Freeze the agent's scope and run both models in parallel for the foreseeable future
• Plan the migration: run evals on the successor, compare, canary, cut over, and update docs

How should success be reported to executives for an agent that handles support tickets?

• The number of tickets the agent touched, the average handling time, and the uptime of the service
• Resolution rate, customer satisfaction, cost per ticket, escalation rate, and incidents, trended
• Tokens per day, cache hit rate, and average latency per response
• Prompt versions shipped and the eval pass rate per release