Regulation on the Conduct of the Oral Examination
Reviewed: February 2019
Next Review Date: July 2020, by the Board of Censors
Background to policy changes
Across 2015-2018 the Fellowship Training Program (FTP) of the College has been transitioning from a progression model of learning to an integrated one. The Pre-Fellowship Oral Examination has been an Exit Examination since the inception of the College and it is retaining that status until 2020 when it will be held as a summative component of one of the FTP’s Domains.
The Board of Censors has also recently agreed that the format of the Oral Examination will change from two stations with a choice of scenarios to be addressed and two stations with no choice; to four stations with no choice; in 2020. This will require the adoption of this format in the Trial Examination to be held at the end of 2019.
A key change has also occurred in the scheduling of the Oral Examination. From 2019, the Oral Examination will be held in the middle of the Calendar training year, brought forward from the end of the year.
The Board of Censors recognises that the scheduling change may have inadvertently disadvantaged candidates who enrolled in the Fellowship Training Program prior to 2018. It has agreed to the scheduling of a second Oral Examination in 2019, in October of this year, to accommodate Candidates in that cohort who will otherwise not be eligible to sit until mid-2020. It should be noted that this accommodation applies only to those candidates specifically disadvantaged:
Candidates and Supervisors have been made aware of the policy changes as they have been developed. These business rules have been articulated for 2019 and 2020 to clarify the transitional processes associated with the conduct of the Oral Examinations in those years.
Fellowship Training has four learning and assessment domains:
The two key summative activities in the Medical Management Practice Domain are satisfactory performance in specified years of supervised medical management practice and satisfactory performance in the College Oral Examination.
The Oral Examination is designed as an assessment tool that aligns with and assesses the intended learning objectives of the graduate outcomes of Medical Expert, Medical Manager and Communicator in the Medical Management Practice Domain.
The standard to be met is that the candidate is able to verbally describe and discuss the knowledge, skills and attitudes required for health management reasoning and action as an independent registered Specialist Medical Administrator in Australia and New Zealand.
The Censor-in-Chief is responsible for the conduct of the College Oral Examinations (the mandatory formative Trial Oral Examination and the summative College Oral Examination) and is supported in this activity by panels of the Board of Censors and staff in the College Office.
The Terms of Reference of the Board of Censors and the role of the Censor-in-Chief are available on the College website. The Board of Censors has been refining eligibility criteria, standard setting, formatting and moderation processes for the College Oral Examinations on a continuing basis and this regulation is updated to reflect recent changes in policy.
The purpose of this document is to outline the standard rules and procedures for the conduct of the Oral Examinations of the Fellowship Training Program of the College in 2019 and 2020.
This document details the business rules associated with eligibility to attempt the examination, pre- examination standard setting, formatting of the examination, post examination moderation of results and notification of outcomes.
The College conducts mandatory Trial Oral Examinations and Jurisdictions conduct optional Practice Examinations which utilise the principles and format of the College Oral Examination, with the exception that immediate feedback to the candidates is offered at the time of these examinations.
Eligibility to sit the ‘Pre-Fellowship’ Oral Examination in August 2019
Candidates will be eligible, and expected, to sit the Oral Examination in August 2019, if at the time of the examination:
Note: Candidates who are unsuccessful at the December 2018 Oral Examination will not be entitled to sit again at the August 2019 Examination. They will be entitled to sit again at the October 2019 examination.
Eligibility to sit the ‘Pre-Fellowship’ Oral Examination in October 2019
Candidates will be eligible to sit the Oral Examination in October 2019, if at the time of the examination:
Eligibility to sit the MMPP Oral Examination from 2020
Candidates will be eligible to sit the Oral Examination in 2020 if at the time of the examination:
Eligibility limitations policy
Full time equivalency of time in supervised medical management practice is calculated as minimum 47 weeks in Australia and 46 weeks in New Zealand. This time includes up to two weeks of professional development. It does not include annual leave, maternity leave or long service leave.
The years of experience in supervised medical management practice, and years of satisfactory performance required to have been completed prior to sitting the Oral Examination are calculated on this basis.
The years of experience in supervised medical management practice is calculated inclusive of experience prior to entry to Candidacy.
Candidates are expected to be continuing in medical management practice at the time that they sit the Oral Examination. The amount of time is to be commensurate with their status as either a full-time candidate (and hence should have been working full time in medical management in the previous 12 months) or part-time (either in a fractional appointment or based on approved leave from the training program).
Candidates who have not been working in medical management are to provide details of why they do not meet this criterion to the College Office no later than when they nominate for the examination, using the processes for seeking Special Consideration (available as Policy on College website).
Leave, and exemptions from eligibility, may be granted at the discretion of the relevant Officer (Censor-in-Chief or Dean) in relation to illness, disability, family reasons, special learning programs outside supervised medical management practice and minor variations from time in practice.
Candidates who are unsuccessful at the Oral Examination are not allowed to sit again until approximately 12 calendar months have elapsed.
Candidates are entitled to sit the Oral Examination three times before their candidacy compliance will be considered to have been breached. They may apply again for training status. They will be subject to rules concerning recognition of prior learning and experience that apply at that time and, if successful in their applications, they will be expected to participate in the conditions of a renewed pathway of learning.
Booking of the Oral Examination
The Oral Examination is held at least annually. From 2019 the Oral Examination will be held in the middle of the calendar year. The College Trial Examination will be held in the later part of the calendar year, for Candidates presenting in the following year. Dates are notified in College correspondence (website and Annual Calendars) as soon as the venues are confirmed.
The preferred venue is the AMC National Test Centre in Melbourne.
The examination sessions will be booked as 4-5-hour events over the required number of days, based on the capacity of the examination venue, the availability of censors and the numbers of candidates. A candidate will be allocated to attend one examination session as notified by the College. Every attempt will be made to accommodate distance travelled and circumstances of those Candidates who have been endorsed as having Special Consideration. The examination for each candidate will be conducted on one day. Sufficient sessions will be provided to allow all eligible candidates who nominate, to be assessed.
Applications close 10 weeks before the date of the examination. Candidates will have completed forms that address the eligibility criteria.
Censors are identified by preliminary calls for availability for the relevant dates, once they have been confirmed.
Format of the Oral Examination
The Oral Examination is an open-book examination encouraging Candidates to analyse and critique responses to scenarios for verbal discussion with examiners.
The Oral Examination consists of four (4) interview stations and the candidates present at four stations.
Each station process is 40 minutes in length consisting of 20 minutes for preparation of responses and 20 minutes for interview:
After the Candidate has left the room the Censors assess the Candidate’s performance using a marking rubric against the knowledge, skills and attitudes required by the scenario. The censor pair will assess independently for five minutes and will each award a score out of 15.
When the censor pair have recorded their independent scores, they will conduct a discussion about their respective scores. Based on this discussion one or both censors may amend their scores. There is no requirement for absolute consensus, however each censor is to be prepared to explain the score during the moderation meeting with all examining censors, if needed. A final summed score out of 30 is recorded by the Censor pair.
Note: The Trial Oral Examination in October 2019 will be set with four stations with one-only scenarios to match the expectations for the Oral Examination in 2020.
After all candidates in the session have completed four stations, all examining censors conduct a moderation session to review the results of candidates (with names hidden). This session is led by the Censor appointed by the Censor in Chief as the Moderating Censor on the day.
The Panel of examining Censors:
This score allows for a (pre-agreed) standard error of measurement of 4 marks out of 120 (3%) below 72/120 (60%).
Censors examining a fifth station will not have assessed the Candidate in any other station that day and will have no conflicts of interest.
Candidates who are clearly successful or unsuccessful will be allowed to leave the Examination space (without being told their outcomes). Candidates who are neither successful, nor unsuccessful are invited to remain in the Examination area to present for a fifth station.
All Candidates undertaking a fifth station will use the same pre-prepared ‘fifth station’ scenario.
After completion of the station, the Candidate leaves the Examination area.
The Moderating Censor is informed of the summed score and a declaration is made on the outcome of the Examination for that Candidate as Successful or Unsuccessful.
The Censor-in-Chief (or delegate) compiles results from the Oral Examination and presents them to a specifically convened Education and Training Committee meeting, usually the working day after the completed examination set. Candidates are notified of their Oral Examination outcomes after endorsement of their results by the Education and Training Committee (usually within 48 hours of the completion of the Examination). After that, notification of the outcome is made to Candidates’ Supervisors, Preceptors and the relevant Jurisdictional Co-ordinators of Training (unless Candidates have opted out of this notification).
Standard setting and Moderation of Examination outcomes
In the RACMA Assessment Policy, moderation is described as the process that ensures the consistency of marking of summative assessment tasks in terms of:
Calibration takes place in the pre-event preparatory activities of an examination and involves:
Oral Examination Topics
Candidates will be assessed on their ability to convey to the Censors that they have the requisite knowledge, skills and attitudes/approach to satisfactorily deal with the examination scenarios. The topics for assessment in this format are outlined in the RACMA Medical Leadership and Management Curriculum in the learning objectives of the role competencies of Medical Expert, Medical Manager and Communicator. The scenario settings include critical care, acute care and sub-acute care; public and private hospitals; and government and non-government health services.
Panel for Examination Questions
The Lead Censor for preparation of Examination Questions co-ordinates the drafting, evaluation, calibration and finalisation of questions for the Oral Examinations, in consultation with a panel of Censors nominated for examination question development.
The Panel is responsible for preparation of the marking rubric for each scenario – which is based on the template published on the RACMA website each year.
The Panel members ensure that the standard for the writing of the questions and the wording in the rubrics is consistent, by internal circulation; or by consultation amongst Fellows (new Fellows or Supervisors) who have ‘tested’ the validity and reliability of the rubrics, during the preparation period.
The Censor for Examination Questions will send the final drafts of the Oral Examination (six questions for each day plus one ‘5th station’ for each day) to the Censor-in-Chief, Dean of Education and an independent Censor, during the pre-event period, either as the questions are developed or at completion, to enable them to assess and verify the details for consistency.
The selected scenarios are sent to the nominated Censor Panellists one-two weeks before the Oral Examination for their information and review. The Censor Panels meet the day before the Oral Examination to collectively familiarise themselves and calibrate the expected responses.
That calibration meeting is also an opportunity to identify to the Censor-in-Chief any previously undeclared conflicts of interest.
Previous Examination Questions
Examples of previous station scenarios are available to Candidates, Preceptors and Supervisors on the RACMA Website. These examples allow Candidates to familiarise themselves with the scope, format, style and acceptable responses in examination questions. Candidates practising past examination questions with their Preceptors and Supervisors will be able to identify where they may have training or knowledge gaps. Many candidates prepare by participating in Practice Oral Examinations learning sets in their jurisdictions.
Pairing of Censors for the Oral Examination
The Censor-in-Chief will oversee the pairing of Censors at the RACMA Oral Examination as follows:
Conflict of interest
All Censors must declare, to the best of their knowledge, any interests with Candidates and/or paired Censors that may prejudice/bias the Censor in the assessment of candidate performance – examples of these could include:
Planning for adequate numbers of Censors on the day will also take into consideration the need to ensure that Candidates are not marked or observed by a Censor more than once on the day.
Note: The panel of participating Censors will be advised to Candidates prior to the examinations. Candidates are also able to declare Censors with whom they believe there may be a potential or perceived conflict which introduces bias.
Candidates will be advised when registering to sit the Oral Examinations that there are cameras in the examination rooms, streaming to the Control Room at the AMC National Test Centre. Permission is sought from Candidates and Censors for videoing in the case of the College Trial Examinations. The final Oral Examinations are not video recorded.
Candidates will be advised when registering to sit the Oral Examinations that there may be authorised Observers in the examination rooms or in the Control Room.
The Observer/s may be:
Format of the College Trial Examination
The format of the College Trial Examination is consistent with the format of the Oral Examination.
The College Trial Examination is an open-book examination encouraging Candidates to analyse and critique responses to scenarios for verbal discussion with examiners.
The College Trial Examination consists of four (4) interview stations and the candidates present at four stations.
Each station process is 40 minutes in length consisting of 20 minutes for preparation of responses and 20 minutes for interview:
After the Candidate has left the room, he/she will wait outside for the Censors to prepare their feedback. The Censors assess the Candidate’s performance using a marking rubric against the knowledge, skills and attitudes required by the scenario. The censor pair will assess independently for five minutes and will each award a score out of 15.
When the censor pair have recorded their independent scores, they will conduct a discussion about their respective scores. Based on this discussion one or both censors may amend their scores. There is no requirement for absolute consensus. A final summed score out of 30 is recorded by the Censor pair.
Feedback is provided by the Censor pair to the candidate after each of the stations at the Trial Oral Examination, when the candidate is handed their marking sheets.
Candidates are encouraged to discuss their performance and feedback on the Trial Examinations with their Supervisor and Preceptor/Coach to address deficiencies that had been identified by the Censors across the four exam stations.
Unless the candidates have opted out, notification of the overall outcome of the Trial Oral Examination is made by the College Office to Candidates’ Supervisors, Preceptors and the relevant Jurisdictional Co-ordinators of Training.
For those Candidates who are unsuccessful at the Oral Examination, the College will provide a formal verbal feedback session within 3 – 4 weeks of the Oral Examination involving the Censor-in-Chief on the day, the Candidate, the Candidate’s Preceptor and/or Supervisor (if agreeable to the Candidate) and a member of the Board of Censors who was involved in the examination of that Candidate – particularly in relation to the questions in which a Candidate was unsuccessful.
Candidates who wish to participate in a verbal feedback session are required to contact the College within 1 week of the examination to request this session.
In College Trial or Jurisdictional Practice Examinations, feedback is given immediately following the scenario interview process. Completed marking rubrics are given to the Candidates.
Bernard Nicholson Prize (Meritorious award)
The Bernard Nicholson Prize is awarded to the Candidate with an outstanding performance in the College Oral Examination. The recommended awardee is determined at the moderation meeting of the Board of Censors held directly after the set of examination sessions. The awarding of this prize is discretionary and the College may decline to make the award if, in the opinion of the Board of Censors, no Candidate has achieved an outstanding result during the examination.
On the recommendation of the Board of Censors, the Education and Training Committee will endorse the award of the Bernard Nicholson Prize to the Board of RACMA for final approval.
Note: ‘The Bernard Nicholson Prize was donated by and is awarded in memory of Dr Bernard Nicholson. Dr Nicholson was a past President of the College and played a major role in its establishment.’
Candidates can request to have a decision by a College Officer/Committee, or in this case, the Board of Censors, reconsidered and reviewed and can follow the College Appeals process in accordance with the Policy for Reconsideration, Review and Appeal of Decisions of the College Officers and Committees.
Code of Conduct
All Censors and Candidates sitting the examination are expected to act at all times ethically, responsibly and in the best interest of the College. All Censors and Candidates will adhere to the RACMA Officer’s Code of Conduct.
Censor Peer Review processes
In accordance with their participation in the College Continuing Education Program, Censors may elect to participate in formal peer review during their activities as an examiner at the College Trial Oral Examination, when it takes place at the AMC National Test Centre. It is expected that Censors will have participated in Censor Peer Review at least once in every three-year term to retain their status as Censors.
Examination peer review process
On the day of the College Trial Examinations there will be times when identified senior Censors will be present in the Control Room viewing the scenario discussions and recording commentary on a peer review form which records observations on the performance of the two censors being observed.
The peer reviewers will provide their commentary to the Censor-in-Chief. The Censor-in-Chief will review the forms and may discuss the commentary with the observing peer review Censors. The forms will be provided by the Censor-in-Chief to individual Censors at the end of the day for reflection and feedback as a part of their own CPD processes.
Examination score peer comparison process
The spreadsheet of scores of Candidates, by Censor and by Station for the Trial Examination Session will be de-identified and coded and sent following the Examination to participating Censors for their perusal, comparison and reflection. The opportunity to further discuss comparisons will be offered by the Censor-in-Chief.
Significant variations across scores and peer review outcomes will routinely be analysed by the Censor-in-Chief and reported as components of evaluation of the Examination process.