MSAT – Desktop Review and On-Site Evaluation

Sofema Aviation Services (SAS) Considers both Desktop & On-Site Evaluation in the context of EASA MSAT Application for Initial Certification, Continuing Oversight, and Performance-Based Evaluation.

Introduction

This document establishes a structured approach for desktop review and on-site evaluation of an organisation’s Management System (MS), ensuring alignment with EASA MSAT methodology and ICAO Annex 19 principles. It is intended for Subject Matter Experts performing high-level, performance-oriented assessments that go beyond prescriptive compliance and focus on management system maturity and effectiveness.

Assessment Objectives

• Validate that the MS processes are Present and Suitable during initial certification/implementation.
• Confirm that processes are operating effectively during ongoing oversight.
• Provide evidence-based insights into the performance, suitability, and scalability of the MS in relation to the organisation’s size, complexity, and operational risk profile.
• Support continuous improvement by identifying strengths, gaps, and opportunities.

Desktop Review

The desktop review is a preparatory and verification stage performed before or in conjunction with an on-site evaluation.

Scope

• Regulatory documentation: Review the MS documentation against applicable EASA domain regulations (e.g., Part-CAMO, Part-145, Part-21) and relevant AMCs/GM.
• SMS framework alignment: Verify inclusion of the 12 ICAO Annex 19 SMS elements, adapted to the EU context.
• Evidence of PSOE maturity:
  • Present – Documented evidence that processes exist.
  • Suitable – Evidence that design fits organisation size, complexity, and risk.
• Cross-domain integration: Where multiple approvals exist, confirm MS integration or alignment.
• Resource adequacy: Evidence of appropriate allocation of human, financial, and technical resources.

Key Document Types

• Safety policy, safety objectives, and just culture policy.
• MS manual(s), expositions, and process descriptions.
• Records of hazard identification, risk assessment, and safety performance monitoring.
• Compliance monitoring programme and outcomes.
• Training and competence records.
• Minutes of Safety Review Board or equivalent.

Desktop Review Deliverables

• Preliminary Assessment Matrix with PSOE ratings for each MS element.
• Evidence Log referencing documentation reviewed.
• Pre-site Evaluation Brief highlighting:
  • Potential areas requiring on-site validation.
  • Key personnel to be interviewed.
  • High-risk or underdeveloped processes.

On-Site Evaluation

The on-site evaluation validates the practical application, operation, and effectiveness of the MS.

Scope

• Confirm that processes identified as Present and Suitable are also Operating and/or Effective.
• Validate organisational safety culture through observation and interviews.
• Assess operational interfaces and communication channels.
• Verify that safety objectives are measurable, monitored, and aligned with the organisation’s risk profile and the State Safety Programme (SSP).

Methods

• Structured interviews with:
  • Accountable Manager.
  • Safety Manager.
  • Senior management with safety accountability.
  • Operational staff and contracted personnel.
• Workplace observation:
  • Application of safety reporting.
  • Risk assessment in operational contexts.
  • Evidence of safety promotion and communication.
• Process tracing:
  • Select a safety issue or objective and follow through from identification → risk assessment → mitigation → monitoring → review.
• Verification sampling:
  • Randomly review records to confirm consistency between documented processes and actual practice.

On-Site Deliverables

• Assessment Notes: Evidence-based observations, interview summaries, operational findings.
• Updated PSOE Ratings: Adjust based on observed maturity and operational application.
• Immediate Risk Notifications: If any safety-critical deficiencies are found.

Evaluation Criteria

Assessments should:

• Avoid checklist-driven “tick-box” approaches.
• Focus on performance orientation and maturity progression.
• Consider suitability and scalability in an operational context.
• Address integration of compliance monitoring and safety risk management.
• Ensure alignment with performance-based oversight principles.

Reporting – The Final Assessment Report Should:

• Provide an overall MS maturity profile using PSOE ratings.
• Highlight effective processes and good practices.
• Identify gaps with recommendations for improvement.
• Include findings (rule-based non-compliance) and observations (opportunities for improvement).
• Assign timeframes for corrective actions that are proportionate to the risk.

Competence of Assessors

SMEs conducting these reviews must:

• Have deep knowledge of ICAO Annex 19, EASA MS requirements, and applicable domain rules.
• Be trained in the use of the EASA MSAT.
• Understand risk-based and performance-based oversight principles.
• Apply objective, evidence-driven assessment techniques.

Continuous Improvement

• Use results to inform oversight planning cycles.
• Consider extending oversight cycles only when Operating and Effective maturity is consistently demonstrated.
• Feed lessons learned back into assessor training and organisational safety promotion.

Next Steps

Sofema Aviation Services (SAS) offers a 2-day training as a Classroom or Webinar – Using The EASA Management System Assessment Tool (EASA MSAT) -2 Days.  For comments & questions, please email [email protected]. 

Tags:

Risk Assessment, Hazard Identification, Safety Performance Monitoring, Initial Certification, on-site evaluation, EASA MSAT, Continuing Oversight, Performance-Based Evaluation, Regulatory documentation, SMS framework alignment, PSOE maturity, allocation of human, technical resources, organisational safety culture, Immediate Risk Notifications