How can we help you?

High Variance Treatment

When an assessment is taken by multiple team members, the presence of a few outliers could impact the overall Agility Index Score of the assessment, which can impact the team’s current state (Pre-Crawl, Crawl, Walk, Run, Fly). In some instances, customers have raised concerns about team assessment results showing the team is in a Run/Fly state while believing the team is actually in a Walk state. 

To combat this situation, we created a feature called High Variance Treatment that is applied to maturity competencies where assessment responses exhibit high variance and a few outliers. This feature adjusts the Agility Index Score and dimension scores using statistical outlier treatments algorithmically as per industry standards. The adjusted score will be automatically reflected across the platform, including APIs, growth journeys, radar results pages, dashboards, benchmarking, and reports.

Watch this video to learn more!

Download the presentation at the bottom of this article. 


How does it work?

The High Variance Treatment will be applied for assessments that have:

  • Maturity competencies that are answered by at least 4 team members and have 50% or higher variance (ie., the difference between the minimum and maximum member score for that competency); and
  • Maturity competencies that have a score of > 50 units and > 50% variance even after the outlier treatments are applied.

If both of the above criteria are met, the maturity competency scores will be pulled down by:

    • 10 units if the Competency Score is between 50-60 units.
    • 20 units if the Competency Score is between 60-70 units.
    • 30 units if the Competency Score is above 70 units.

Please refer to the attachment at the bottom of this article for the detailed methodology of how maturity competency scores could be adjusted with the High Variance Treatment.


Where do the changes take place?

These adjusted scores will be automatically reflected across the platform (ie., APIs, growth journeys, radar results pages, dashboards, benchmarking, and reports).

On the team's radar results page for their assessment, the red line displaying the average of the team's responses will be lowered in the competencies that meet the above criteria, the red line will never be raised. The team member's actual responses (the black dots) will remain in place. Outlier responses will not be removed from the radar.

Here are the assessment results before the High Variance Treatment is applied:

HVT Feature Off

Here are the same assessment results after the High Variance Treatment is applied:

HVT Feature On

You will also see a new Overall Score section below the Analytics section on a team's radar results page displaying the team's updated Agility Index score/state and adjusted score/state in each dimension.

Screenshot 2024-05-14 at 2.06.44 PM.png

On the Enterprise Agility Dashboard, the team's adjusted score may put the team in a different state. For example, the team's overall score may be adjusted enough to show in the Agility Index widget as in the Walk state instead of the Run state. The Maturity Metrics scores will also be adjusted accordingly and displayed in the appropriate widget.

EA Agility Index and Matury Metrics Widgets


What methodologies are used?

Z-score and Interquartile Range (IQR) are two statistical methods used for identifying and handling outliers in data. Outliers are data points that differ significantly from other observations, which could be due to variability in the measurement or it could indicate experimental errors. We use both methods to determine if the maturity competency scores meet the criteria for the application of the high-variance treatment.

Z-score
The Z-score, also known as the standard score, measures how many standard deviations an element is from the mean of the dataset. The formula to calculate the Z-score of a value is:

Z=(X−μ)/σ

where:

X is the value,
μ is the mean of the dataset, and
σ is the standard deviation of the dataset.

Values with a Z-score above or below a certain threshold (commonly 2 or 3) are considered outliers. This method assumes that the data follows a normal distribution, and it's particularly sensitive to outliers because the mean and standard deviation are easily influenced by extreme values.

IQR (Interquartile Range)
The IQR is a measure of statistical dispersion or variability based on dividing a data set into quartiles. Quartiles divide a rank-ordered data set into equal parts. The values that divide each part are known as the first quartile (Q1), the second quartile (Q2, or the median), and the third quartile (Q3).

The IQR is calculated as:

IQR=Q3−Q1

Outliers can then be identified by using the IQR. A common rule of thumb is that a data point is considered an outlier if it is:

Below Q1 - 1.5 x IQR or
Above Q3 + 1.5 x IQR

This method does not assume a normal distribution of the data and is, therefore, more robust for data with an unknown distribution or when the data is skewed.

Was this article helpful?
0 out of 0 found this helpful
    Download article

Comments

Please sign in to leave a comment.