The definition

What is AI observability?

AI observability is continuous, evidence-based visibility into AI running in production - usage, reliability, and safety - so the people accountable for AI can spot problems early and govern with data instead of anecdotes. It's the monitoring layer that makes AI governance enforceable.

01

Usage

Who uses which agents and Copilot apps, how often, and in which departments.

02

Reliability

Response times, outcomes, tool success, and the error codes behind failed conversations.

03

Safety

Prompt-injection and jailbreak attempts, harmful-content categories, and responsible-AI blocks.

04

Alerts

Thresholds that surface the change worth acting on - before it becomes an incident.

Agent telemetry

Every Copilot Studio agent, measured

Sentinel pulls Copilot Studio telemetry from Application Insights and trends it over time, so every agent's usage and health is visible - and the agent that suddenly takes off, or starts failing, stands out.

  • Conversations and users - daily conversations, unique users, and new versus returning users, per agent.
  • Outcomes - resolved, escalated, and abandoned conversations, so a struggling agent shows up.
  • Topics, tools, and latency - what users ask, which tools an agent calls and whether they succeed, and how fast it answers.
  • Transcripts - drill from a spike to the individual conversations behind it.
app.princetonsentinel.com/dashboard/agents
Copilot Studio agent telemetry with conversation KPIs, daily conversation trend, and conversation outcomes
AI safety monitoring

Catch prompt injection, jailbreaks, and harmful content

Copilot Studio blocks some attacks; others get through. Sentinel collects the platform's own safety events and runs its own classifier over every user prompt each hour - including the ones the platform let through - so your team reviews structured evidence, not screenshots in a chat thread.

  • Safety events - content-filter, jailbreak, injection, DLP, and connector blocks, by agent, channel, and month.
  • Attack classification - prompt-injection and jailbreak attempts identified and counted per agent.
  • Harm categories - content-safety harm categories charted, so you see which risks dominate.
  • Review queues - block-worthy prompts and prompts flagged for a human, counted up front.
app.princetonsentinel.com/dashboard/agents/safety
AI Safety dashboard with safety event KPIs, events by month and by agent, and prompt classification counts
Prompt verdicts

Evidence for every flagged conversation

Each flagged conversation opens to its verdict and the exchange behind it, so a reviewer can tell an attack from a curious user or a false alarm in seconds.

  • Verdict - attack label, threat category, severity, confidence, and recommended action.
  • Admin review flag - verdicts that need a human are routed to a review queue.
  • Prompt and safety event together - the prompt and the platform's safety event side by side - visible to admins only.
app.princetonsentinel.com/dashboard/agents/safety/session
AI Safety conversation detail with a jailbreak verdict, critical severity, block recommendation, and the safety event log
Copilot usage analytics

Measure Microsoft 365 Copilot adoption

Copilot licenses are expensive and adoption is uneven. Sentinel shows who is enabled, who is actually active, and where Copilot is landing - without storing a single prompt or response.

  • Enabled vs active users - adoption trended over time, so idle licenses are obvious.
  • Prompts submitted - volume over time and by time of day.
  • By app and department - adoption across Microsoft 365 apps and your top departments.
  • Counts, not content - Copilot prompt and response text is never stored.
app.princetonsentinel.com/dashboard/copilot
Microsoft 365 Copilot usage analytics with adoption trend and per-app breakdown
Alerts & reporting

Know when something changes

Observability only helps if someone notices. Sentinel raises agent, AI Safety, and Copilot alerts and tracks every agent's risk day by day, so the right people hear about a spike before it becomes an incident.

  • 21 AI alerts - shadow and dormant agents, expired credentials, tenant-wide permission grants, prompts flagged for review, and idle Copilot licenses.
  • Banded by urgency - Urgent, Warning, and Info alerts on the Executive Summary, each clicking through to the rows behind it.
  • Daily agent risk history - see whether an agent is getting safer or riskier over time.
  • Scheduled PDF reports - Executive Summary and Risk Summary PDFs emailed daily, weekly, or monthly.
app.princetonsentinel.com/dashboard/risk
Risk hub with agent-risk and data-risk gauges and summaries
FAQ

AI observability questions

What is AI observability?

AI observability is continuous, evidence-based visibility into AI systems running in production - their usage, performance, reliability, and safety - so the people accountable for AI can spot problems early and act on data instead of anecdotes. For Microsoft 365, that means monitoring Copilot Studio agents and Microsoft 365 Copilot across the whole tenant.

What's the difference between AI observability and AI governance?

Observability tells you what your AI is doing: usage, reliability, and safety signals. Governance decides what should happen about it: inventory, permissions, risk thresholds, and controls such as quarantine. Princeton Sentinel does both in one console - the telemetry feeds the agent risk scores, and the risk scores point admins at the agents to contain.

Where does Sentinel's agent telemetry come from?

From the Application Insights telemetry your Copilot Studio agents emit, combined with the agents registered in Dataverse. Microsoft 365 Copilot usage and adoption come from Microsoft Graph. A scheduled worker syncs it into Sentinel's database, so dashboards query a local copy instead of live APIs.

Does Sentinel block risky prompts in real time?

No. Sentinel classifies prompts after the fact - it doesn't sit inline in front of Copilot. When it finds a prompt-injection attempt, a jailbreak, or harmful content, admins can block that user from the agent or quarantine the agent, and the action is logged.

Does Sentinel store Copilot prompts?

Microsoft 365 Copilot prompt and response text is never stored - only usage counts. Copilot Studio conversation text used for safety classification is visible to admins only and is deleted after 90 days by default.

Watch your agents in the demo

We'll walk agent telemetry, prompt-attack classification, and Copilot adoption on a tenant that looks like yours.

Request a demo