To Evaluate, or Audit, That is the Question - When it Comes to Your IEPs
Congratulations on making your SMS more proactive by implementing an Internal Evaluation Program (IEP)! By now, you have probably visited the FAA and ICAO websites to download pertinent Advisory Circulars and Annex Guidance material. After digesting numerous pages of guidance, examples, and practices, you may have derived an IEPs objective is:
To systematically collect information about activities, characteristics, and outcomes of identified policies, programs, and procedures that are designed to continually assure the quality and safety of your operations.
To use the collected IEP information to make operating judgments about specific programs, improve program effectiveness, modify quality and safety controls/thresholds, and make informed decisions about future program development.
Within your IEP, suppose you want to collect information on a new parts supplier for your maintenance shop. You send your maintenance manager to visit the office of the parts supplier. They tour the facility, speak to key personnel, and sample the quality of the parts to be supplied. Does the manager write an ‘evaluation’ or ‘inspection’ report? Could the manager count this event as a supplier ‘audit’ giving recommendations of further ‘review’?
Let’s break down some terminology -
Are systematic and objective assessments of an ongoing or completed project, programs or policy, its design, implementation success, and results
Initially employs a rigorous methodology of collecting and documenting business processes, their organizational structure, responsibility, authority, controls, interfaces, and process measures[i]
Can be conducted internally or external to an organization with the understanding that an internal evaluator is directly accountable to the organization (function) they are evaluating, whereas an external evaluator is not
Are efforts to focus on the aspects of SMS safety promotion, continuous quality improvement, learning and development, and directing cultural change
Produce findings that are in “non-conformance” to internal company policy, operating objectives, and non-regulatory industry standards
Create targeted reviews and follow-up by managing “opportunities for improvement”
Are an assessment of the adequacy of management controls to ensure economical and efficient use of resources, the safeguarding of assets, the reliability of financial and other operating information[ii]
Are a measurement of compliance to regulations, rules, and established policies; the effectiveness of risk management; and the adequacy of organizational structures, business systems and processes.
Are focused on compliance while evaluations are more intricately linked to the organizations learning and development
Focus on the general examination of a specific organizational unit, issue, or practice to ascertain the extent it adheres to normative standards and good practices[iii]
Performed to make further recommendations for improvement or to validate an open corrective action requests generated by a recent IEP event
Performed, potentially as a no-notice event, when there is a perceived risk of a systemic non-compliance or direct threat to personnel