Streetlight effect killing M365 Governance

A blog by Martin Erasmuson.

In establishing a governance framework for their Microsoft 365 tenancy, many organisations look at the plethora of reporting options available in the M365 Reports & Admin Centre and before they know it, are producing a bunch of colourful reports, and impressive dashboards.  In doing that, many fall victim to ‘the streetlight effect’ in reporting on inconsequential stuff, mostly because it’s interesting or easy. 

What is the streetlight effect? 

One night a police officer on the beat notices an obviously drunk person on their hands and knees searching for something under a streetlight. Asking what they’re looking for the drunk replies they’ve lost their keys.  The police officer helps search for a few minutes before asking the drunk if they’re sure this is where they lost their keys. The drunk replies no, they lost them over there, pointing to a dark carpark, but the light is better here.

Reporting could, or should, support strategic objectives and proposed benefits most likely identified in the original M365 Business Case; along with the key performance indicators (KPI) underpinning the wider M365 governance framework.  And remember, sometimes the most important KPI can be less than straightforward to obtain data for, at least compared with the off the shelf M365 reports.

Let’s look at both empirical, and conjecture-based reporting examples.

Empirical reporting

Governance Objective:  Manage the M365 tenancy licensing within a specified budget.

KPI: App usage within the licensing range

Report objective (in user story format): As a M365 tenancy administrator; I want to identify the number each month of PowerAutomate (flow) operations; to confirm that is within what our M365 licensing allows, and no additional fees will be incurred.

Report: Power Platform Admin Centre > Analytics > Power Automate

(Note: a regular, scheduled report would also give you a trend)

Conjecture-based reporting

Governance objective: our staff are using SharePoint to store and manage organisational content.

KPI: 

  1. Staff feel adequately trained in using M365 and SharePoint to save and manage theirs and their teams content

  2. Teams and individuals are saving and managing their content in SharePoint

Combined reporting objectives (in user story format)

  1. As a SharePoint Administrator; I want to see how many documents are being created in, saved to and edited in SharePoint; in order to gauge the level of use and activity

  2. As a SharePoint Administrator; I want to know the level of confidence staff have in using SharePoint to create and manage content; to establish and training or knowledge gaps

Reporting (two reports required):

  1. M365 Reports & Admin Centre > SharePoint activity and/or SharePoint site usage.

  2. Survey Monkey (Self-reporting) > ‘which statement best describes how you use SharePoint: 1 = I know what to do and I know how to do it; 5 = I have no clue what to do and if I did, I would know how to do it’ (Survey monkey or similar).

Using the above scenario, if the SharePoint activity/site-usage report indicates none or limited activity, plus the Survey Monkey result with a bunch of 4 or 5 scores; that indicates some impediment to use.

I advise against falling victim to the seductive array of shiny reports in the M365 Reports & Admin Centre before first establishing why (M365 strategic objectives) and what (governance framework) success will look like; then how (reporting objectives) and where (the report itself) the data will come from.

 

Up next: the Microsoft 365 governance framework.  What that might look like, and how to begin to implement it.

Want to talk about something in this blog?  Please feel free to contact me.

(C) Creative Commons - Attribution 3.0 Aotearoa New Zealand

You are free to:

1. Share — copy and redistribute the material in any medium or format for any purpose, even commercially.

2. Adapt — remix, transform, and build upon the material for any purpose, even commercially.

3.       The licensor cannot revoke these freedoms as long as you follow the license terms.

Under the following terms:

1. Attribution - You must give appropriate credit , provide a link to the license, and indicate if changes were made . You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.

2. No additional restrictions - You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.

Notices:

You do not have to comply with the license for elements of the material in the public domain or where your use is permitted by an applicable exception or limitation .

No warranties are given. The license may not give you all of the permissions necessary for your intended use. For example, other rights such as publicity, privacy, or moral rights may limit how you use the material.