Implementing Conditional Access is a critical step in safeguarding identities and resources. But deploying these policies without understanding their real-world effects can lead to disruptions. That’s why Microsoft provides two essential features—report-only mode and per-policy reporting—to help you assess and refine your policies before they go live.
This article walks you through how to use these tools to evaluate Conditional Access effectively.
What is Report-Only Mode?
Report-only mode allows administrators to test Conditional Access policies without enforcing them. When users sign in, the policies are evaluated silently in the background. The system records what the outcome would have been, but it does not actually block or grant access.
✅ This gives IT teams a safe way to preview policy effects and fine-tune configurations before fully deploying them.
⚠️ Note: Report-only mode doesn’t support policies scoped to User Actions.
During evaluation, the system logs expected outcomes in the Conditional Access and Report-only tabs of the Sign-in logs. It’s a low-risk way to spot potential disruptions early.
What is Per-Policy Reporting?
Per-policy reporting, now generally available, provides a detailed breakdown of how each Conditional Access policy impacts sign-in activity across your organization. Whether the policy is enabled or in report-only mode, this feature shows:
-
Which users are affected
-
How frequently the policy is triggered
-
The outcomes of those evaluations
With clear, actionable visualizations, this feature helps administrators understand and optimize policy behavior at an individual level—ensuring your access controls are both secure and seamless.
Avoid Unexpected Prompts on Mobile and Mac Devices
Policies requiring device compliance can trigger certificate prompts on macOS, iOS, and Android, even when in report-only mode. These can confuse end users and repeat until the device is compliant. To prevent this, exclude these platforms from report-only policies involving device compliance checks during the testing phase.
Understand Evaluation Results
Every sign-in that encounters a policy returns a clear result. Here’s what each one means:
| Evaluation Result | What It Means |
|---|---|
| Report-only: Success | All conditions and controls were satisfied. |
| Report-only: Failure | Conditions were met, but one or more required controls failed. |
| Report-only: User action required | User would need to act (e.g., perform MFA), but wasn’t prompted in simulation. |
| Report-only: Not applied | Policy conditions didn’t match (e.g., user or location excluded). |
| Success (active policy) | Access was allowed as all conditions were met. |
| Failure (active policy) | Access was denied due to failed conditions. |
| Not applied (active policy) | Policy didn’t apply to this sign-in. |
Use Built-In Tools to Review Impact
Microsoft provides several tools to help visualize and analyze Conditional Access policies and their impact:
1. Policy Impact View (Preview)
This view gives a snapshot of policy effects on interactive sign-ins over the last 24 hours, 7 days, or 1 month.
-
Helps spot patterns in user sign-ins
-
Shows sign-in samples for deeper review
-
Available to users with Security Reader permissions
2. Conditional Access Insights and Reporting Workbook
A powerful dashboard for analyzing multiple policies at once—especially when in report-only mode.
-
Customize views based on users, apps, and timeframes
-
Visualize combined policy effects
-
Ideal for enterprise-wide policy testing
3. Sign-In Logs with Report-Only Tab
For granular review, use the Sign-in logs to check how each policy behaved during a specific sign-in.
-
Shows whether a policy was in report-only or active mode
-
Details evaluation outcomes
-
Helps troubleshoot user-specific cases
Move Policies from Report-Only to Enforced
Once you’ve reviewed results using the tools above and are confident in your policy configurations:
-
Go to the policy settings
-
Change Enable policy from Report-only to On
This shifts the policy from simulation to enforcement—without surprises or disruptions.
By combining report-only mode with the now generally available per-policy reporting, administrators gain full visibility into how Conditional Access policies behave across the organization. These tools help you:
-
Validate policy conditions
-
Fine-tune enforcement logic
-
Prevent misconfigurations
-
Roll out policies smoothly and confidently
Before enabling any Conditional Access policy, test, analyze, and refine it. This approach not only enhances security—it safeguards the user experience too.
