Congratulations on making your SMS more proactive by implementing an Internal Evaluation Program (IEP)! By now, you have probably visited the FAA and ICAO websites to download pertinent Advisory Circulars and Annex Guidance material. After digesting numerous pages of guidance, examples, and practices, you may have derived an IEPs objective is:
To systematically collect information about activities, characteristics, and outcomes of identified policies, programs, and procedures that are designed to continually assure the quality and safety of your operations.
To use the collected IEP information to make operating judgments about specific programs, improve program effectiveness, modify quality and safety controls/thresholds, and make informed decisions about future program development.
Within your IEP, suppose you want to collect information on a new parts supplier for your maintenance shop. You send your maintenance manager to visit the office of the parts supplier. They tour the facility, speak to key personnel, and sample the quality of the parts to be supplied. Does the manager write an ‘evaluation’ or ‘inspection’ report? Could the manager count this event as a supplier ‘audit’ giving recommendations of further ‘review’?
Let’s break down some terminology -
Evaluations:
Are systematic and objective assessments of an ongoing or completed project, programs or policy, its design, implementation success, and results
Initially employs a rigorous methodology of collecting and documenting business processes, their organizational structure, responsibility, authority, controls, interfaces, and process measures[i]
Can be conducted internally or external to an organization with the understanding that an internal evaluator is directly accountable to the organization (function) they are evaluating, whereas an external evaluator is not
Are efforts to focus on the aspects of SMS safety promotion, continuous quality improvement, learning and development, and directing cultural change
Produce findings that are in “non-conformance” to internal company policy, operating objectives, and non-regulatory industry standards
Create targeted reviews and follow-up by managing “opportunities for improvement”
Audits:
Are an assessment of the adequacy of management controls to ensure economical and efficient use of resources, the safeguarding of assets, the reliability of financial and other operating information[ii]
Are a measurement of compliance to regulations, rules, and established policies; the effectiveness of risk management; and the adequacy of organizational structures, business systems and processes.
Are focused on compliance while evaluations are more intricately linked to the organizations learning and development
Inspections:
Focus on the general examination of a specific organizational unit, issue, or practice to ascertain the extent it adheres to normative standards and good practices[iii]
Performed to make further recommendations for improvement or to validate an open corrective action requests generated by a recent IEP event
Performed, potentially as a no-notice event, when there is a perceived risk of a systemic non-compliance or direct threat to personnel
To evaluate, or audit, that is the question.
The differences between auditing and evaluating processes stem from the intellectual disciplines from which they evolved. Auditing grew out of the accounting discipline, while evaluation grew out of organizational behavior and social sciences. An IEP ‘auditor’ looks for particular instances of things going wrong and places great weight on the need for accuracy of administrative records. An auditor is likely to be suspicious of statistical inferences based on aggregate data if they cannot find actual cases confirming the general conclusion.
The IEP ‘evaluator’ generally mistrusts a manager’s summation and views financial records as a macro-performance indicator. By design, evaluators are predisposed to seek their own direct observations and collect a much more generalized view of what happened. They prefer large volumes of data and generally mistrust small numbers of specific events. Evaluators have little confidence that what was found in a single case can lead to a chronic event. Unlike the auditor, the evaluator is never able to observe everything that needs to be observed.[iv] Thus, most internal evaluation programs are set up on a continual review process that refines its scope and reach over time, based upon improved volumes of observation.
For an effective IEP, each discipline must use the techniques of the other to reach a shared solution of shared problems. Circumstances will dictate which method is more appropriate to a particular review. Regulatory compliance or financial statement work is important, and the IEP manager must utilize personnel who are very skilled at establishing compliance baselines. From an SMS perspective, any aviation program that merely passes a compliance audit only means that it meets the minimum regulatory thresholds of safety. To grow a program by integrating best practices, sustainable objectives, and organizational excellence, the IEP team needs equal amounts of adept program evaluators.
[i] Brown, Ann-Murray A. (April 4, 2017). Evaluation, Inspection, Audit: Is there a difference? www.annmurraybrown.com [ii] Ibid. [iii] Ibid. [iv] Harry S. Havens, USGAO – Association of Government Accountants, Audit and Evaluation: Is there a difference, October 16, 1980.