|Subject:||Assessment in the Fellowship Training Program|
|Approval Dates:||October 2018|
|Approved By:||Education and Training Committee|
|Review Dates:||August 2018|
|Review By:||Education and Training Committee|
Consistent with the College’s adoption of an outcomes-based approach to Curriculum design, the Assessment Policy aims
This policy applies to College processes for both formative programmatic observation and feedback in the workplace (assessment for learning); and summative examination of performance by examination processes (assessment of learning).
This Assessment Policy builds on previous Assessment Frameworks described for the Fellowship Training Program in 2012 and 2014 and has been renewed in 2018 to accommodate the transitioning of the FTP’s structure. The FTP has evolved from a progression model of learning culminating in an exit oral examination; to an integrated model of teaching, learning and aligned assessment activities in four domains. Each of the Domains incorporates one or more of the College’s eight role competencies:
The principles of the Assessment Policy are that:
Each Domain has a program of formative assessment activities and summative tasks which are aligned to the overall intended learning outcomes of those domains.
The Health System Science Domain is addressed by candidate participation in learning and assessment in a recognised university master’s degree (while working in an accredited training post). In this activity the candidates are involved in programmatic learning – they are taught by university academics with the experience to facilitate their learning and they experience assessment in the relevant topics on a regular and frequent basis throughout each course.
The other three Domains’ intended learning outcomes are covered by RACMA – guided assessment activities. Learning and training in the Medical Management Practice domain, the Personal and Professional Leadership Development domain and the Research Training domain occurs on the job (as a registrar in medical administration or as a Candidate in a substantive medical leadership position).
This training is supported by College-organised enhancement of University learning (workshops and distance tutorials); the commitment of RACMA-acknowledged supervisors in the workplace who formatively assess the Candidates in a schedule of regular and frequent face-to-face discussions; and College level examination of Candidates at the Oral Examination and the oral and written presentations of research.
BLUEPRINT FOR ASSESSMENT METHODS
The blueprint for alignment of assessment methods with intended learning outcomes has been developed in terms of Domain groupings of role competencies and is outlined in Table 1.
Table 1 Formative and summative activities aligned with domain learning objectives
|Domains||Formative activities||Summative activities|
|Masters study RACMA workshops Logbook entries||Workplace observation and feedback||In-training performance reports||Oral Examination||Oral and written research tasks||Masters study coverage|
|Health system science (HSS) Specialist knowledge acquisition||✔||✔|
|Research training in health service research (RT) Skill development in evidence-based practice||✔||✔||✔|
|Personal and professional leadership development (PPLD) Mastery of reflective practice|
|Medical management practice (MMP) Skill development||✔||✔||✔||✔|
DOMAIN-BASED LEARNING AND ASSESSMENT SCHEDULE
The schedule for learning and assessment has been allocated across three full-time-equivalent years of training by Domain and is summarised in Table 2.
Table 2 Schedule of assessment activities
|Domains||Health system science Learning||Health system scienceAssessmentby universities||Medical Management Practice Learning and formative assessment in workplace||MMP workplace summativeassessment by Training Progress Committee||MMP College formative and summative assessments by examinationsBoard of Censors||Personal and Professional Leadership Development learning and formative assessment in workplace||PPLDworkplace summative assessment by Training Progress committee||Research training learning||Research training College assessments and examinations|
|Pre-requisite criteria for entry to the FTP||Meet criteria for accredited University program||Recognition of Prior Learning||3 years medical practice, registration,RPLE||Interviews and references for aptitude and RPLE||Exemptions for prior learning and experience|
|1st year of supervised practice||Master’s study Jurisdictional tutorials||Master’s assignments(external)||Annual training plan Participation Workshop 1 Learning sets Workplace observation and feedback Training log in ITP Report||MMP ITP Reports satisfactory||Participation jurisdictional practice examinations||Learning sets Interact webinars and E-modules Indigenous health module PPLD discussions||PPLD Report satisfactory||Health Services Research webinars Health Service Research assignment|
|2nd year of supervised practice||Master’s study Jurisdictional tutorials||Master’s assignments(external)||Annual training plan Learning sets Ministerial briefing Workplace observation and feedback Training log in ITP report||MMP ITP Reports satisfactory||College Trial Examination||Learning sets Interact and E-modules Participation Workshop 2 PPLD discussions Reflective writing||PPLD Report satisfactory||Proposal endorsement Research conduct under supervision||Oral presentation of research progress|
|3rd year of supervised practice andthereafter||Master’s study Jurisdictional tutorials||Master’s assignments(external)||Annual training plan Participation Workshop 3 Workplace observation and feedback Oral Exam learning sets Training log in ITP Report||MMP ITP forms satisfactory||College Oral Examination||Learning sets Interact and E-modules PPLD discussions||PPLD Report satisfactory||Research completion Report writing||Written research paper|
|Eligibility for Fellowship||Completion of Master’s degree||Minimum 3 years MMP ITP forms satisfactory||College Oral Examination success||Minimum 3 yearsPPLD ITP forms satisfactory||Research based written paper satisfactory|
TRAINING OF ASSESSORS
The College is committed to orientation and training of any Fellow or Non-Fellow who is involved in assessing the performance or the progress of Candidates’ in the Fellowship Training Program. Supervisor and Censor orientation is provided in written form and in conversations with the Dean and the Censor-in-Chief; and a program of Faculty webinars and workshops is organised on an annual basis.
New Censors are expected to observe oral presentations (Research Progress and the Oral Examination) prior to examining; and are initially paired with experienced Censors when they begin marking oral or written examination material. Special webinar sessions are conducted by the Censor-in-Chief or Dean in the event of changes in processes for assessment e.g. assessment for Recognition of Prior Learning and Experience, and assessment of Specialist International Medical Administrators seeking comparability.
SEPARATION OF MEASUREMENT MOMENTS
The Assessment Policy’s call for separation of formative assessment measurement from summative examination is enacted at various levels.
IN-TRAINING PERFORMANCE REPORT
In the Fellowship Training Program, the key form of learning is workplace practice under supervision. The goal of assessment is the self- or supervisor-generated feedback that enhances information for reflection and improvement in skill development. Progress in workplace learning is identified in logging of educational opportunities and logging of discussions concerning observed management tasks, presentations of medical management case studies and discussions of reflections on incidents or professional topics.
Rubrics for assessment of these tasks have been developed for consistency in identifying the stages that have been demonstrated and guiding topics for further learning. Although use of these rubrics is not mandatory, there is an expectation that face-to-face discussions with supervisors (day-to-day report person, training supervisor, mentor, preceptor, research overseer, executive coach) will occur regularly and frequently.
The Board of RACMA agreed in 2012 that the Leadership Program within the FTP will not be summatively assessed. The reason for this was that proposed personal and professional leadership development adopts an underlying constructivist approach and there is research evidence that summative assessment of these aspects of learning may negate growth. Hence the summative statement in the Personal and Professional Leadership Development Domain relates only to participation in activities, not to an assessment of progress.
An In-Training Performance Report (ITPR) is required every six-months. There are two parts to the In-Training Performance Report. The first part is the logging of educational activities and observed management tasks and the second part is the completion of a rubric of opinion on performance level against the intended learning outcomes of the Domains. There is a final question in the ITPR seeking the supervisor/preceptor/executive coach global assessment of the level to which learning expectations have been met in the MMP Domain and the PPLD Domain.
TRAINING PROGRESS COMMITTEE
The In-Training Progress Report is reviewed by the Training Progress Committee, a group made up of Jurisdictional Co-ordinators of Training. This Committee’s function has been the monitoring of Candidate compliance with completion of required tasks for presentation at the Oral Examination. In 2018 it will be transitioning to summatively assessing Candidate Progress as Satisfactory in both the MMP and PPLD Domains by considering the information in the In-Training Progress Reports.
The status of the Pre-Fellowship Oral Examination is transitioning from that of an Exit Examination to that of a component of the Medical Management Practice Domain. The eligibility requirements are changing. Those Candidates who are expecting to sit the MMPD Oral Examination in 2020 will be required to:
They will not be required to have completed the assignments of the other domains before being allowed to sit the Oral Examination.
The format for the Oral Examination continues to be one of open-book preparation for presentation of four scenarios and discussion with two Censors. Customised rubrics are pre-prepared provided for scenario discussion and these are linked to marks.
ORAL AND WRITTEN PRESENTATION OF RESEARCH PROGRESS
There is an expectation that Candidates will receive regular formative feedback from their research supervisors and that their research progress will be marked summatively by different Fellows who have been trained in the use of rubrics which have been developed for consistency purposes.
Business rules for the preparation, standard-setting, timetabling, conduct and review of assessment tasks are renewed annually to ensure that Candidates, Supervisors, Jurisdictional Co-ordinators of Training and Censors are all aware of the expectations. Generally, a score of 60% in a summative assignment is considered a ‘passing’ score. Supplementary information may be required for borderline situations. Lack of success in a summative assessment will prompt re-submission, or re-presentation. There may be limitations on the number of times some activities may be attended – e.g if Candidates are unsuccessful at three Oral Examinations, they must re-apply for candidacy.
These rules also include reference to processes for special consideration and reconsideration of decisions made by College Office-holders.
MONITORING AND EVALUATION OF ASSESSMENT IN THE FELLOWSHIP TRAINING PROGRAM
Monitoring and evaluation of the Fellowship Training Program occurs at the ‘macro’- program level, ‘meso’- domain, and ‘micro’ – task level; and is considered in terms of demonstration of meeting standards and acting on the opinions of stakeholders.
The College follows a cyclical improvement approach for monitoring and evaluation at each level: observations are made and recognised, reasoning takes place following analysis, responses (changes) occur and these are reviewed prior to renewing or confirming policies and processes. See Figure 1.
Figure 1 Cyclical evaluation and monitoring
Monitoring of implementation of the Assessment Policy takes place on an annual basis and takes the form of reporting to the Education and Training Committee on the outcomes of survey responses from Candidates, New Fellows and Supervisors; and relevant reports from then Training Progress Committee and the Board of Censors.