Sofema Aviation Services (SAS) www.sassofia.com considers the role and purpose of Aviation Safety Management System (SMS) – Safety Performance Indicators (SPI).
Introduction – Maintenance Key Performance Indicator (KPI)
Typically measures how well your operation is doing at achieving its maintenance goals, for example reducing downtime or costs. KPIs can be used as a benchmark to understand your organisation’s current position as well as to facilitate the development of Key Performance goals.
Maintenance key performance indicators (KPIs) vary according to the company, its goals, strategies, and action plans. However, there is a set of indicators that is well-regarded and used more often. Consider the following Features:
- The selection of KPIs can relate to many different elements and aspects including financial – efficiency of the process – health and safety (SMS) considerations – defects and pilot reports (PIREPS) following maintenance.
- Once a detailed assessment has been made it is possible to consider which are the most relevant indicators applicable to the delivery of the most effective maintenance.
o How do these indicators relate to each other and do they meet the requirement?
o Is the measurement process of the chosen indicators effective?
Notes
- KPIs are not Safety Performance Indicators (SPI) nor are they directly related to the achievement of safety objectives (although they can be a feature within the overall safety management system).
- Whilst KPIs are closely related to the performance level you want to achieve, don’t take them for goals themselves. (This means an indicator is only a metric which we use quantitatively to demonstrate the performance of a given maintenance activity, asset or department.)
- When developing KPIs it is important that they relate to the entire process including the various inputs and outputs so that they are able to genuinely benchmark the performance of the maintenance.
Safety Performance Indicator
A safety performance indicator (SPI) is defined in the ICAO Safety Management Manual as a measure (or metric) used to express the level of safety performance achieved in a system.
- Generally expressed in terms of the frequency of occurrences of some event causing harm, e.g., A number of events/year or / 000’s of Hrs of operation.
- A set of indicators should also include leading indicators.
Note: These indicators do not require a safety event to take place and are metrics that provide information on the current situation that may affect future performance.
What is the main difference between Safety Performance Indicators (SPI) and Safety Performance Targets (SPT)?
ICAO defines:
- Safety Performance Indicators as “A data-based parameter used for monitoring and assessing safety performance”.
- Safety Performance Targets as “The planned or intended objective for safety performance indicator(s) over a given period”.
The following table is the EASA proposed baseline set of SPIs (Note: this is essentially for Operators – Currently EASA has not created a baseline set of SPIs for use of an EASA Part 145 Aircraft Maintenance Organisation (AMO))
SPI/SPT Development Considerations
- Specific understanding regarding the selection of the relevant safety concern to be treated to ensure both validity & relevance.
- Avoid complexity wherever possible – simple is better and can clearly reflect relevant safety concerns.
- Use clear definitions to avoid ambiguity.
- Phased approach to implementation – achieves the ultimate objective in stages.
- Avoid high workload analysis.
- Ensure a broad coverage is important do not only pursue extremes (to capture systemic exposure that can agglomerate into more significant events.
- Make sure time frames are clear and understood (data capture rate).
- Avoid quantity over quality measure what is important, not easy.
Leading & Lagging SPIs and SPTs
Lagging indicators are reactive in nature that measures an organization’s performance like the number and types of incidents that occurred based on the information from past incidents and accidents such as reactionary analysis. Reactionary analysis is analyzing past data to find loopholes in processes and policies.
Leading indicators are proactive and preventive measures that can shed light on the effectiveness of safety and health activities and reveal potential problems in a safety and health program.
Many employers are familiar with lagging indicators:
- Incorrect application of the MEL / 1000 Departures
- Missed Airworthiness Directive (AD) / 10,000 FH
Leading SPIs/SPTs – sometimes known as “Process SPIs/SPTs” – measure situations which have the potential to become or contribute to high severity/low probability negative outcomes:
- Leading SPIs support proactive development of the organisation’s safety management System – Capability to improve safety exposure.
o SPI – 0.75 events of foreign object damage FOD) to aircraft per 10,000 Maintenance Activities (Ramp Transit & Minor Check).
o SPT: To reduce to 0.5 events of foreign object damage to aircraft per 10 000 Maintenance Activities (Ramp Transit & Minor Check) within 12 months from the introduction.
Further Considerations Lagging & Leading Indicators
- Lagging safety performance indicators should match the above criteria.
- For leading indicators it should be validated that they are representative of measurement for accident risk.
- In general it can be said that it is not unreasonable to assume there is a positive relationship between a correct functioning system of organisation and aviation safety.
- Effort is required to validate that positive relation. Some of the indicators suggested are Boolean; they are either true or false. Boolean indicators are straightforward to quantify, comprehendible, and cost-efficient.
- To assure sensitivity to changes safety performance measurement must be done frequently increasing the costs of obtaining the results; a proper balance needs to be found.
- Lagging indicators are metrics for safety events that have already occurred.
- Risk controls will attempt to address many leading and lagging indicators. When erosion of risk controls occurs, the severity of lagging indicators will increase and there will be a negative impact on the percentage of standard operating procedure (SOP) hazards identified.
Implementation of Maintenance Safety Performance Indicators
Safety performance indicators are important because they measure the proper functioning of the organisations barriers that exist as well as considering also the interfaces between our organisation and customers as well as other related (contracted & subcontracted) organizations.
Consider the following:
- The main goal of aviation safety performance indicators is to provide an indication of the probability of an incident/accident.
- This can be done using lagging and leading indicators.
- The total set of indicators should remain manageable.
- Each individual indicator should match the following criteria:
o Quantifiable and permitting statistical inferential procedures
o Valid or representative of what is to be measured
o Provide minimum variability when measuring the same conditions
o Sensitive to change in environmental; or behavioural conditions
o Cost of obtaining and using measures is consistent with the benefits
o Comprehended by those in charge with the responsibility of using them
Examples of Suitable SPIs for the Maintenance Environment
- Operational Incidents due to maintenance
- Operational Accidents due to maintenance
- TIR Technical Incident Reports
- IFSD Engine Inflight Shutdowns
- MEL Extensions Requested extensions for MEL items
- AD Airworthiness Directives irregularities
- Average risk level determined for all occurrences
- Maintenance of Unscheduled Engine Removals
- Failures in emergency equipment during
- Voluntary Reports Voluntary Occurrence Reports,
- Number of major risk incidents (as defined in SMM)
- Number of MORs
- Number of internal audits
- Number of audit findings per audit
- Number of safety committee meetings
- Safety committee attendance of key personnel
- Number of ERP drills
- Number of hazard/safety reports
- Number of safety newsletters issued
- Number of formal risk assessments
- Number of safety surveys
- Number of airworthiness incidents (as defined in SMM)
- Number of flights flown with operational MEL restrictions
Next Steps
Follow this link to our Library to find & Download related documents for Free.
Sofema Aviation Services www.sassofia.com and Sofema Online www.sofemaonline.com provide Aviation Safety Management System Training as Classroom, Webinar & Online courses. For additional information please visit our websites or email team@sassofia.com
Tags:
Aviation Maintenance, aviation safety, EASA Aviation SMS, EASA Part 145, Hazard and Risk Identification, Leadership, Maintenance Key Performance Indicator, Part 145, Safety, Safety Performance Indicator (SPI), SAS blogs, SPI
