Regulation on the Conduct of Oral Examinations
Reviewed: September 2020
Next Review Date: September 2022
The Fellowship Training Program has four learning and assessment domains:
The two key summative assessment activities in the Medical Management Practice (MMP) Domain are satisfactory performance in specified years of supervised medical management practice and satisfactory performance in the College MMP Oral Examination.
The MMP Oral Examination is designed as an assessment tool that aligns with and assesses the intended learning objectives of the role competencies of Medical Expert, Medical Manager and Communicator in the Medical Management Practice Domain. Learning objectives from other domains may also be assessed as they apply to the Medical Management Practice Domain.
The standard to be met is that the candidate is able to verbally describe and discuss the knowledge, skills and attitudes required for health management reasoning and action as an independent registered Specialist Medical Administrator in Australia and New Zealand.
The Censor in Chief is responsible for the conduct of the Fellowship Training Program Oral Examinations (the mandatory formative College Trial Examination and the summative MMP Oral Examination) and is supported in this activity by panels of the Board of Censors and staff in the College Office.
The Terms of Reference of the Board of Censors and the role of the Censor in Chief are available on the College website. The Board of Censors has been refining eligibility criteria, standard setting, formatting and moderation processes for the Fellowship Training Program Oral Examinations on a continuing basis and this regulation is updated to reflect recent changes.
The purpose of this Regulation is to outline the standard rules and procedures for the conduct of the Oral Examinations for the MMP Domain of the Fellowship Training Program from 2020.
This regulation details the business rules associated with eligibility to sit the MMP Oral Examination, pre- examination standard setting, formatting of the examination, post examination moderation of results and notification of outcomes.
The College conducts mandatory Trial Oral Examinations and Jurisdictions may conduct optional Practice Examinations which utilise the principles and format of the MMP Oral Examination, with the exception that feedback is offered to the Candidates as part of these examination sessions.
Candidates will be eligible to sit the MMP Oral Examination if at the time of application for the examination:
A full-time equivalent year in supervised medical management practice is calculated as a minimum of 47 weeks in Australia and 46 weeks in New Zealand. This time includes up to two weeks of professional development leave. It does not include annual leave, parental leave or long service leave.
The years of experience in supervised medical management practice, and years of satisfactory performance required to have been completed prior to sitting the MMP Oral Examination are calculated on this basis.
The years of experience in supervised medical management practice are calculated inclusive of time approved through Recognition of Prior Learning and Experience processes.
Candidates are expected to be continuing in medical management practice at the time that they sit the MMP Oral Examination. The time fraction in which they are working should be that which has been approved for their training posts.
Candidates who have not been working in medical management practice are to provide details of why they do not meet this criterion. Details should be provided to the College Office at the time of application for the examination, in accordance with the Policy and Process for Special Consideration for the Oral Examinations.
Applications for leave, and exemptions from eligibility, in relation to illness, disability, family reasons, special learning programs outside supervised medical management practice and minor variations from time in practice, are made to the College office and addressed by the Censor in Chief.
Candidates who are unsuccessful at the MMP Oral Examination are not allowed to sit again until an examination scheduled for the following calendar year.
Candidates are entitled to sit the MMP Oral Examination three times before their candidacy compliance will be considered to have been breached. They may apply again for Candidacy in the Training Program, or apply for membership status as an Associate Fellow of RACMA. If they apply for another Candidacy, they will be assessed for recognition of prior learning and experience according to the conditions that apply at that time and be subject to rules concerning recency of practice for applications for further attempts at the MMP Oral Examination.
The MMP Oral Examination is held at least annually. The College Trial Examination will be held at least annually. Candidates must complete and submit the application form that address the eligibility criteria by the required date.
All Candidates will be notified of examination dates as soon as details are confirmed.
A Candidate will be expected to attend the examination session allocated by the College. Every attempt will be made to accommodate distance travelled and circumstances of those Candidates who have been approved for Special Consideration. The examination for each Candidate will be conducted on one day. Sufficient examination sessions will be provided to allow all eligible Candidates who apply, to be assessed.
The MMP Oral Examination is an open-book examination encouraging Candidates to analyse and critique responses to set scenarios, for verbal discussion with examiners.
The MMP Oral Examination consists of four (4) interview stations and the Candidates present at all four stations.
Each station process is 40 minutes in length consisting of 20 minutes for preparation of responses and 20 minutes for interview:
After the Candidate has left the room the Censors complete the assessment of the Candidate’s performance using a marking rubric against the knowledge, skills and attitudes required by the scenario. The Censor pair will assess independently for five minutes and will each award a score out of 15
When the Censor pair have recorded their independent scores, they will conduct a discussion about their respective scores. Based on this discussion one or both Censors may amend their scores. There is no requirement for absolute consensus, however each Censor is to be prepared to explain the score during the moderation meeting with all examining Censors. A final summed score out of 30 is recorded by the Censor pair.
After all Candidates in the session have completed four stations, all examining Censors conduct a moderation session to review the scores awarded for each of the four stations and to each individual de-identified Candidate. This session is led by the Moderating Censor appointed by the Censor in Chief.
The Panel of examining Censors:
Censors examining a fifth supplementary station will not have assessed the Candidate in any other station that day and will have no conflicts of interest.
Candidates who are clearly successful or unsuccessful will be asked to leave the Examination space (without being told their outcomes). Candidates who are eligible for a fifth, supplementary station are invited to remain in the Examination space to present for a fifth station.
All Candidates undertaking a fifth station on any one day will use the same pre-prepared ‘fifth station’ scenario.
After completion of the supplementary station, Candidates will be asked to leave the Examination space (without being told their outcomes).
The Moderating Censor is informed of the summed scores for the fifth supplementary station and makes a declaration to the Censor in Chief on the outcome of the Examination for all Candidates as either Successful or Unsuccessful.
Results from the Oral Examination are compiled and presented by the Censor in Chief (or delegate) to a specifically convened Education and Training Committee meeting conducted as soon as practicable during the week following completion of the examination. Candidates are notified of their Oral Examination outcomes after endorsement of their results by the Education and Training Committee. After that, notification of the outcome is made to Candidates’ Supervisors, Preceptors and the relevant Jurisdictional Coordinators of Training.
Calibration takes place in the pre-event preparatory activities of an examination and involves:
Moderation is the process that ensures the consistency of marking of summative assessment tasks in terms of:
Candidates will be assessed on their ability to convey to the Censors that they have the requisite knowledge, skills and attitudes/approach to satisfactorily deal with the examination scenarios.
The topics for assessment in this format are outlined in the RACMA Medical Leadership and Management Curriculum in the learning objectives of the role competencies of Medical Expert, Medical Manager and Communicator in the Medical Management Practice Domain. Learning objectives from other domains may also be assessed as they apply to the Medical Management Practice Domain.
The scenario settings include critical care, acute care and sub-acute care; community practice, public and private hospitals; and government and non-government health services.
The Lead Censor for preparation of Examination Questions coordinates the drafting, evaluation, calibration and finalisation of questions for the MMP Oral Examination, in consultation with a Panel of Censors nominated for examination question development.
The Panel is responsible for preparation of a customised marking rubric for each scenario – which is based on the published RACMA template.
The Panel ensures that the standard for the writing of the questions and the wording in the rubrics is consistent.
The Lead Censor for Examination Questions will send the final drafts of all questions for the MMP Oral Examination to the Censor in Chief, Dean and an independent Censor (appointed by the Censor in Chief) to enable them to assess and verify the details for consistency.
The selected scenarios are sent to the nominated Censor Panellists who will be examining each scenario, before the Oral Examination Calibration meeting for their information and review. The Censor Panels meet before the Oral Examination to collectively familiarise themselves and calibrate the expected responses.
That calibration meeting is also an opportunity to identify to the Censor in Chief any previously undeclared conflicts of interest.
Examples of previous station scenarios are available to Candidates, Preceptors and Supervisors on the RACMA Website. These examples allow Candidates to familiarise themselves with the scope, format and style of; and acceptable responses in, examination scenarios. Candidates practising past examination questions with their Preceptors and Supervisors will be able to identify where they may have training or knowledge gaps. Many Candidates prepare by participating in Jurisdictional Practice Oral Examinations.
The Censor in Chief will oversee the pairing of Censors at the MMP Oral Examination as follows:
New Censors, who have completed the required induction and training prior to being eligible to examine will be paired with a senior and experienced Censor.
Note: The panel of participating Censors will be advised to Candidates prior to the examination. Candidates are also able to declare Censors with whom they believe there may be a potential or perceived conflict which introduces bias. In doing so the Candidate must list the reason behind their declaration.
Candidates will be advised when registering to sit the Fellowship Training Program Oral Examinations that examination stations are monitored either by cameras in the examination rooms (when the examinations are conducted “face to face” streaming to the Control Room at the Examination Centre) or by remote monitoring (when conducted by Zoom or similar technology).
Permission is sought from Candidates and Censors for videoing in the case of the College Trial Examinations.
The MMP Oral Examination is not audio or video recorded.
Candidates will be advised when applying to sit the MMP Oral Examination that there may be authorised Observers in the examination rooms or in the Control Room via remote technology used for the examination.
The Observer may be:
For those Candidates who are unsuccessful at the MMP Oral Examination, the College will provide a formal verbal feedback session of the MMP Oral Examination. The session involves the Censor in Chief (or delegate), the Candidate, and a member of the Board of Censors who was involved in the examination of that Candidate – particularly in relation to the questions in which a Candidate was unsuccessful. It is strongly recommended that Candidates involve their Preceptors and/or Supervisors in the verbal feedback session.
Candidates who wish to participate in a verbal feedback session are required to contact the College by the date nominated in their examination outcome notification.
In College Trial Examinations or Jurisdictional Practice Examinations, feedback is provided to the Candidate by the Censor pair for each of the stations. The Candidate is provided with verbal feedback and their marking sheets.
The Bernard Nicholson Prize is awarded to the Candidate with an outstanding performance in the MMP Oral Examination. The recommended winner is determined at the meeting of the Board of Censors held after the completion of the examination. The awarding of this prize is discretionary and the College may decline to make the award if, in the opinion of the Board of Censors, no Candidate has achieved an outstanding result during the examination.
On the recommendation of the Board of Censors, the Education and Training Committee will endorse the award of the Bernard Nicholson Prize to the Board of RACMA for final approval.
Note: ‘The Bernard Nicholson Prize was donated by and is awarded in memory of Dr Bernard Nicholson. Dr Nicholson was a past President of the College and played a major role in its establishment.’
Candidates can request to have a decision by a College Officer/Committee, or in this case, the Board of Censors, reconsidered and reviewed and can follow the College Appeals process in accordance with the Policy for Reconsideration, Review and Appeal of Decisions of the College Officers and Committees.
The format of the College Trial Examination is consistent with the format of the MMP Oral Examination.
The College Trial Examination is an open-book examination encouraging Candidates to analyse and critique responses to scenarios for verbal discussion with Censors.
The College Trial Examination consists of four (4) interview stations and the Candidates present at four stations.
Each station process is 40 minutes in length consisting of 20 minutes for preparation of responses and 20 minutes for interview:
After the Candidate has left the room, they will wait outside for the Censors to prepare their feedback. The Censors assess the Candidate’s performance using a marking rubric against the knowledge, skills and attitudes required by the scenario. The Censor pair will assess independently for five minutes and will each award a score out of 15.
When the Censor pair have recorded their independent scores, they will conduct a discussion about their respective scores. Based on this discussion one or both Censors may amend their scores. There is no requirement for absolute consensus. A final summed score out of 30 is recorded by the Censor pair.
Feedback is provided by the Censor pair to the Candidate for each of the stations at the Trial Oral Examination. The Candidate is provided with verbal feedback and their marking sheets.
Candidates are encouraged to discuss their performance and feedback on the Trial Oral Examination with their Supervisor and Preceptor/Executive Coach to address deficiencies that had been identified by the Censors across the four examination stations.
Notification of the outcome of the Trial Oral Examination is made by the College Office to Supervisors, Preceptors and the relevant Jurisdictional Co-ordinators of Training.
All Censors and Candidates sitting the examination are expected to act at all times ethically, responsibly and in the best interest of the College. All Censors will adhere to the RACMA Officers’ Code of Conduct.
Candidates are expected to comply with the Terms and Conditions for Participation in RACMA Oral Examinations as defined in the Candidate Examination Agreement.
In accordance with their participation in the College Continuing Education Program, Censors may elect to participate in formal peer review during their activities as an examiner at the College Trial and MMP Oral Examinations. It is expected that Censors will have participated in Censor Peer Review at least once in every three-year term to retain their status as Censors.
At College Trial or MMP Oral Examinations there will be times when identified senior Censors will view the scenario discussions of Censors from the Control Room/Centre. The observing Censor will record observations and commentary of the Censor performance on a peer review form.
The peer reviewers will provide their commentary to the Censor in Chief. The Censor in Chief will review the forms and may discuss the commentary with the observing peer review Censors. The forms will be provided by the Censor in Chief to individual Censors at the end of the day for reflection and feedback as a part of their own CPD processes.
Significant variations across scores and peer review outcomes will routinely be analysed by the Censor in Chief and reported as components of evaluation of the Examination process.