header background

Assessment in the Fellowship Training Program

Home / About Us / Governance / College Policies / Assessment in the Fellowship Training Program
Subject: Assessment in the Fellowship Training Program
Approval Dates: October 2018
Approved By: Education and Training Committee
Review Dates: August 2018
Review By: Education and Training Committee

PURPOSE

AIMS

Consistent with the College’s adoption of an outcomes-based approach to Curriculum design, the Assessment Policy aims

SCOPE

This policy applies to College processes for both formative programmatic observation and feedback in the workplace (assessment for learning); and summative examination of performance by examination processes (assessment of learning). 

CONTEXT

This Assessment Policy builds on previous Assessment Frameworks described for the Fellowship Training Program in 2012 and 2014 and has been renewed in 2018 to accommodate the transitioning of the FTP’s structure. The FTP has evolved from a progression model of learning culminating in an exit oral examination; to an integrated model of teaching, learning and aligned assessment activities in four domains.  Each of the Domains incorporates one or more of the College’s eight role competencies:

PRINCIPLES

The principles of the Assessment Policy are that:

FITNESS-FOR-PURPOSE

Each Domain has a program of formative assessment activities and summative tasks which are aligned to the overall intended learning outcomes of those domains.

The Health System Science Domain is addressed by candidate participation in learning and assessment in a recognised university master’s degree (while working in an accredited training post). In this activity the candidates are involved in programmatic learning – they are taught by university academics with the experience to facilitate their learning and they experience assessment in the relevant topics on a regular and frequent basis throughout each course.

The other three Domains’ intended learning outcomes are covered by RACMA – guided assessment activities. Learning and training in the Medical Management Practice domain, the Personal and Professional Leadership Development domain and the Research Training domain occurs on the job (as a registrar in medical administration or as a Candidate in a substantive medical leadership position).

This training is supported by College-organised enhancement of University learning (workshops and distance tutorials); the commitment of RACMA-acknowledged supervisors in the workplace who formatively assess the Candidates in a schedule of regular and frequent face-to-face discussions; and College level examination of Candidates at the Oral Examination and the oral and written presentations of research.

BLUEPRINT FOR ASSESSMENT METHODS

The blueprint for alignment of assessment methods with intended learning outcomes has been developed in terms of Domain groupings of role competencies and is outlined in Table 1.

Table 1 Formative and summative activities aligned with domain learning objectives

      Domains Formative activities Summative activities
Masters study RACMA workshops Logbook entries Workplace observation and feedback In-training performance reports   Oral Examination Oral and written research tasks Masters study coverage
Health system science (HSS) Specialist knowledge acquisition        
Research training in health service research (RT) Skill development in evidence-based practice      
Personal and professional leadership development (PPLD)  Mastery of reflective practice
   
Medical management practice (MMP) Skill development     ✔   ✔   ✔   ✔      

DOMAIN-BASED LEARNING AND ASSESSMENT SCHEDULE

The schedule for learning and assessment has been allocated across three full-time-equivalent years of training by Domain and is summarised in Table 2.

Table 2 Schedule of assessment activities

DomainsHealth system science LearningHealth system scienceAssessmentby universitiesMedical Management Practice Learning and formative assessment in workplaceMMP workplace summativeassessment by Training Progress CommitteeMMP College formative and summative assessments by examinationsBoard of Censors Personal and Professional Leadership Development learning and formative assessment in workplacePPLDworkplace summative assessment by Training Progress committeeResearch training learningResearch training College assessments and examinations
Pre-requisite criteria for entry to the FTPMeet criteria for accredited University program Recognition of Prior Learning 
3 years medical practice, registration,RPLE  
Interviews and references for aptitude and RPLE 
Exemptions for prior learning and experience
 1st year of supervised practice Master’s study Jurisdictional tutorials  Master’s assignments(external)Annual training plan Participation Workshop 1 Learning sets  Workplace observation and feedback Training log in ITP Report   MMP ITP Reports satisfactory   Participation jurisdictional practice examinationsLearning sets Interact webinars and E-modules Indigenous health module PPLD discussions   PPLD Report satisfactoryHealth Services Research webinars Health Service Research assignment    
2nd year of supervised practice Master’s study Jurisdictional tutorials   Master’s assignments(external) Annual training plan Learning sets  Ministerial briefing Workplace observation and feedback Training log in ITP report  MMP ITP Reports satisfactory  College Trial ExaminationLearning sets Interact and E-modules Participation Workshop 2 PPLD discussions  Reflective writing   PPLD Report satisfactoryProposal endorsement   Research conduct under supervision  Oral presentation of research progress
3rd year of supervised practice andthereafterMaster’s study Jurisdictional tutorials   Master’s assignments(external)         Annual training plan Participation Workshop 3 Workplace observation and feedback Oral Exam learning sets  Training log in ITP Report  MMP ITP forms satisfactory    College Oral Examination  Learning sets Interact and E-modules PPLD discussions  PPLD Report satisfactory   Research completion  Report writing  Written research paper     
Eligibility for Fellowship
 Completion of Master’s degree
Minimum 3 years MMP ITP forms satisfactory   College Oral Examination success
Minimum 3 yearsPPLD ITP forms satisfactory 

Research based written paper satisfactory 

TRAINING OF ASSESSORS

The College is committed to orientation and training of any Fellow or Non-Fellow who is involved in assessing the performance or the progress of Candidates’ in the Fellowship Training Program.  Supervisor and Censor orientation is provided in written form and in conversations with the Dean and the Censor-in-Chief; and a program of Faculty webinars and workshops is organised on an annual basis.

New Censors are expected to observe oral presentations (Research Progress and the Oral Examination) prior to examining; and are initially paired with experienced Censors when they begin marking oral or written examination material. Special webinar sessions are conducted by the Censor-in-Chief or Dean in the event of changes in processes for assessment e.g. assessment for Recognition of Prior Learning and Experience, and assessment of Specialist International Medical Administrators seeking comparability.

SEPARATION OF MEASUREMENT MOMENTS

The Assessment Policy’s call for separation of formative assessment measurement from summative examination is enacted at various levels.

IN-TRAINING PERFORMANCE REPORT

In the Fellowship Training Program, the key form of learning is workplace practice under supervision. The goal of assessment is the self- or supervisor-generated feedback that enhances information for reflection and improvement in skill development. Progress in workplace learning is identified in logging of educational opportunities and logging of discussions concerning observed management tasks, presentations of medical management case studies and discussions of reflections on incidents or professional topics.

Rubrics for assessment of these tasks have been developed for consistency in identifying the stages that have been demonstrated and guiding topics for further learning. Although use of these rubrics is not mandatory, there is an expectation that face-to-face discussions with supervisors (day-to-day report person, training supervisor, mentor, preceptor, research overseer, executive coach) will occur regularly and frequently.

The Board of RACMA agreed in 2012 that the Leadership Program within the FTP will not be summatively assessed. The reason for this was that proposed personal and professional leadership development adopts an underlying constructivist approach and there is research evidence that summative assessment of these aspects of learning may negate growth. Hence the summative statement in the Personal and Professional Leadership Development Domain relates only to participation in activities, not to an assessment of progress.

An In-Training Performance Report (ITPR) is required every six-months. There are two parts to the In-Training Performance Report. The first part is the logging of educational activities and observed management tasks and the second part is the completion of a rubric of opinion on performance level against the intended learning outcomes of the Domains. There is a final question in the ITPR seeking the supervisor/preceptor/executive coach global assessment of the level to which learning expectations have been met in the MMP Domain and the PPLD Domain.

TRAINING PROGRESS COMMITTEE

The In-Training Progress Report is reviewed by the Training Progress Committee, a group made up of Jurisdictional Co-ordinators of Training. This Committee’s function has been the monitoring of Candidate compliance with completion of required tasks for presentation at the Oral Examination. In 2018 it will be transitioning to summatively assessing Candidate Progress as Satisfactory in both the MMP and PPLD Domains by considering the information in the In-Training Progress Reports.

ORAL EXAMINATION

The status of the Pre-Fellowship Oral Examination is transitioning from that of an Exit Examination to that of a component of the Medical Management Practice Domain. The eligibility requirements are changing. Those Candidates who are expecting to sit the MMPD Oral Examination in 2020 will be required to:

They will not be required to have completed the assignments of the other domains before being allowed to sit the Oral Examination.

The format for the Oral Examination continues to be one of open-book preparation for presentation of four scenarios and discussion with two Censors. Customised rubrics are pre-prepared provided for scenario discussion and these are linked to marks.

ORAL AND WRITTEN PRESENTATION OF RESEARCH PROGRESS

There is an expectation that Candidates will receive regular formative feedback from their research supervisors and that their research progress will be marked summatively by different Fellows who have been trained in the use of rubrics which have been developed for consistency purposes.

BUSINESS RULES

Business rules for the preparation, standard-setting, timetabling, conduct and review of assessment tasks are renewed annually to ensure that Candidates, Supervisors, Jurisdictional Co-ordinators of Training and Censors are all aware of the expectations. Generally, a score of 60% in a summative assignment is considered a ‘passing’ score. Supplementary information may be required for borderline situations. Lack of success in a summative assessment will prompt re-submission, or re-presentation. There may be limitations on the number of times some activities may be attended – e.g if Candidates are unsuccessful at three Oral Examinations, they must re-apply for candidacy.

These rules also include reference to processes for special consideration and reconsideration of decisions made by College Office-holders.

MONITORING AND EVALUATION OF ASSESSMENT IN THE FELLOWSHIP TRAINING PROGRAM

Monitoring and evaluation of the Fellowship Training Program occurs at the ‘macro’- program level, ‘meso’- domain, and ‘micro’ – task level; and is considered in terms of demonstration of meeting standards and acting on the opinions of stakeholders.

The College follows a cyclical improvement approach for monitoring and evaluation at each level: observations are made and recognised, reasoning takes place following analysis, responses (changes) occur and these are reviewed prior to renewing or confirming policies and processes. See Figure 1.

Figure 1 Cyclical evaluation and monitoring

Monitoring of implementation of the Assessment Policy takes place on an annual basis and takes the form of reporting to the Education and Training Committee on the outcomes of survey responses from Candidates, New Fellows and Supervisors; and relevant reports from then Training Progress Committee and the Board of Censors. 

Never miss a RACMA moment