The use of safety performance indicators (SPI), safety performance targets (SPT) and their connection to safety objectives tends to be a challenge among business aviation organizations. Unfortunately, this has led to some organizations simply establishing SPIs and SPTs to satisfy regulatory or industry best practice guidelines. In other words, the use of SPIs and SPTs has been relegated to a “check the box” exercise rather than providing the organization with meaningful information upon which to make important decisions. In this post, we will briefly discuss what SPIs and SPTs are, where they fit in the concept of safety performance management and discuss what your organization can be measuring for success.
The safety performance indicator (SPI) is nothing more than a measurement - a data point if you will - meant to inform on performance. In ICAO’s Document 9859 4th Ed., it loosely defines an SPI as something that is used to help senior management know whether the organization is likely to achieve its safety objectives. Again, the SPI informs and ICAO states that it informs on the progress of meeting safety objectives and informs on the performance of processes and risk controls in the organization. In the former, the SPI is coupled with a target (SPT) that defines where the organization wants to be in terms of measurement, which is defined by the objective.
For example:
A flight operations department may have the objective to reduce or maintain landings outside of the touchdown zone to less than 1 per 100 operations.
The measurement (SPI) is landings outside of the touchdown zone while the target (SPT) is less than 1 per 100 operations.
The SPI is only what is being measured and informs management where the operation is in relation to where it wants to be (i.e., the SPT).
The SPI is also used to inform on the performance of processes and risk controls in the organization, as stated earlier. In this case, we are measuring the outcomes of organizational activities, policies, and processes to determine if those elements are functioning as intended and designed. If we were to put this into our everyday context - imagine the engine temperature gauge in your car. If the engine is performing as designed, the engine gauge should provide a “normal” measurement. If the engine gauge goes outside of “normal” limits then we know something is wrong and have the information needed to decide to fix it. It is the same with our risk control within the organization. We need to measure if our risk controls are working and functioning within limits.
For example:
The fatigue management system is a risk control and let’s say that an organization is measuring fatigue levels prior to a flight (i.e., a scale of 1 - 5 where 1 is dead tired and 5 is wide awake). Let’s say that the organization finds that a value of 4 is average, it might establish the “yellow zone” at 3 and the “red zone” at 2.
Now, the organization can measure the performance of its fatigue management system (a risk control) and make decisions regarding its performance through the limits that have been established.
We come full circle when we realize that if the average fatigue levels have dropped to 3, the organization can establish an objective to increase the fatigue scores (SPI) to, at, or above 4, by a designated time in the future (SPT).
In other words, if an adverse trend or unacceptable value of the SPI when measuring risk control performance manifests, the organization can establish an objective to move that value back to optimal levels.
Often, organizations are tempted to measure the number of incidents or accidents to determine how risk controls are performing, however we have few data points here (thankfully!). These are deemed high consequence / low probability measurements and happen very infrequently, but are generally catastrophic, costly, or both.
Where the good stuff is in the low consequence / high probability measurements. These are the things we can measure that happen before the low probability / high consequence events. This type of measurement is also referred to as a precursor indicator by ICAO.
For example:
An approach that is not stabilized (high probability / low consequence) can be a precursor indicator to controlled flight into terrain (CFIT) or a runway excursion (both low probability / high consequence indicators).
The approach that is not stabilized SPI informs on the risk control’s (i.e., stabilized approach policy) performance and can also provide information on chances for a CFIT event or runway excursion.
Therefore, focusing on measuring precursor events can inform the organization’s trajectory towards the low probability / high consequence events.
Organizations are always curious as to what to measure in their operations. In terms of establishing and measuring objectives, simply measure what the objective is aimed at. In regards to measuring the performance of risk controls, ICAO recommends that the organization determine what the high-risk operations are or where the greatest risk exposure is and measure the risk control performance in this context. However, ICAO also provides a list of what can be measured, which is especially applicable to small organizations:
Events including structural damage to equipment
Events indicating circumstances in which an accident nearly occurred
Events in which operational personnel of members of the aviation community were fatally or seriously injured
Event in which operational personnel became incapacitated or unable to perform their duties safely
Rate of voluntary occurrence reports
Rate of mandatory occurrence reports
This is obviously not an all-inclusive list, but it is a good start. These are very general measurements that can allow the organization to cast a wide net in terms of measuring safety performance. The idea is that if there is a trend or preponderance of data in one of those areas, the organization then can drill down to specifics. At Baldwin Safety and Compliance, we also have generated a list of items that can be measured as viable SPIs [see below]. This list is grouped by functional area and then further refined by different objectives the organization may be interested in attaining. We have a mix of low consequence / high probability and high consequence / low probability indicators that might be helpful to the business aviation flight operator. It is important to note that SPIs need to be useful to an organization, therefore, all organizations may have differing SPIs. Please select the ones that would be most useful to your organization based on your operations.
Takeaways:
The use of SPIs and SPTs should be more than a “check the box” exercise for your organization but should provide meaningful information on the safety performance of your organization.
The SPIs are meant to support! They should provide information on the attainment of safety objectives (with SPTs) and the performance of risk controls in your organization.
In regard to risk control performance, ICAO does provide some good indicators, especially for small operators.
Baldwin Safety and Compliance also has some good low consequence / high probability as well as high consequence / low probability indicators that can be used in your organization’s safety performance monitoring and measurement system [see below].
It is important to note that SPIs should be applicable to your organization and not merely “window dressing”. Select the right indicators and your organization can be on the path to measuring for success.