Survivorship bias is a processing error and form of selection bias that happens when people only look at information or things that have made it past a selection process and overlooking those that didn’t due to lack of visibility. A classic example is when statistician Abraham Wald examined aircraft that had returned home from World War II missions and made savvy recommendations where armor could be added to areas that showed the least damage of these planes. He inferred that planes that did not return home had been hit in those areas and therefore proved what needed to be bolstered. Without accounting for survivorship bias, Wald would have made his recommendations based only on the planes that did return home, without factoring in the planes that did not. Survivorship bias is present in finance, education, the military, and can also be a key hindrance to IAM teams. It is particularly prevalent in access certifications and surveys, resulting in decisions being made based on incomplete datasets.
The Importance of Access Surveys
It’s often said that you can’t manage what you don’t see, but it’s also true that you can’t analyze data that you don’t have. For years, Identity Governance & Administration (IGA) solutions have helped organizations with running surveys and certifications for determining who has access to what and why, and then making decisions about how access can be appropriately updated as needed. As people move roles, or join or leave the organization, their roles change, and therefore their access rights must change with them. Not only do these surveys help maintain security, but auditors also need to know this information, as well as why access was granted in certain cases, whether it was appropriate, and reasons for why exceptions are made. Surveys are an essential component of any IGA program, but complicating factors in running access surveys are never-ending, particularly within the context of today’s modern businesses.
Suffering from Survey Fatigue
IAM leaders and administrators may view longer surveys as a necessary evil to gain a deep understanding of access rights throughout the ever-changing organization, but less is often more. It isn’t a total shock to think that lengthy surveys can lead to surveyed workers overlooking questions, skipping them, or rubber-stamping answers. Survey fatigue is a common problem for business users who are routinely sent surveys that stretch on for tens, hundreds, and yes, even thousands of questions at a time. At Omada, we regularly speak with customers that want to better refine their survey processes, and in looking at 42,381 surveys, totaling nearly 600,000 total questions, we found that survey fatigue can very easily lead to survivorship bias, and incomplete data.
First, we found that the rejection rate, or the percentage of questions that, when asked if access was needed to a certain data set, application or otherwise was answered with ‘no’, for surveys that were 0-25 questions was 6.5%, whereas in surveys that were more than 1,000 questions was only 1.1%. Digging into this data we found that the reasons for this were simple. When faced with longer surveys, people rejected fewer and fewer questions and instead fell back on their default positions that they should maintain their access status quo. This was either due to them skipping questions altogether, or just hitting “accept” repeatedly until reaching the end of the survey. While the intention of long surveys is likely to gather as much data as possible, this has been proven to have diminishing returns. Again, we found that less was more.
There are two primary problems with this long-winded survey approach. For one, skipping questions can lead to survivorship bias, where administrators only are analyzing data they have from answered questions and results in an incomplete painting of the access landscape. Second, long surveys take up a lot of time for business users and results in lost productivity, frustration, and survey fatigue. The question is, how can organizations lessen the burden these surveys have on people while still gathering important information about access rights?
What Can Be Done to Combat Survey Fatigue
While organizations shouldn’t abandon survey questions for the sake of brevity, a clear best practice is to shorten surveys where it makes sense. Further, designing the survey process in a way where if the initial user is unable to answer some of the questions they can be reassigned to another user can be very helpful. This takes very little time for the administrator to set up and permit this type of action when launching the survey and can have a dramatic positive effect.
Next, using tags and tag categories helps simplify and organize findings. These tags and tag categories are used to establish and operationalize the risk management strategy and put relevant controls in place based on the findings from a survey. This also allows the administrator or system owner to create a survey which is sent to users and presents them with tasks to classify the survey in a timely fashion. Once the questions are completed, the classifications needs to be approved by the by the category owner.
Finally, establishing how often a survey should run and automating that process where possible can save a lot of time for IAM leaders and business users alike. The administrators can also input what should happen when responses to the survey are submitted, or when answers are not given, as well as set notifications and reminders that should be sent, and determine who can monitor and manage the campaigns and surveys. This will also help business users settle into routines for allocating time, but not too much of it, towards accurately filling out access surveys.
Omada IdentityPROCESS+ lays out how surveys are created in the design certification campaign process and scoped based on factors like risk classification, systems and resource types for maximum effectiveness, without over burdening the people tasked with filling out the surveys.
