Regulation on the Conduct of Oral Examinations
Reviewed: September 2020
Next Review Date: September 2022
1. BACKGROUND
The Fellowship Training Program has four learning and assessment domains:
- Health System Science;
- Medical Management Practice;
- Research Training; and
- Personal and Professional Leadership Development.
The two key summative assessment activities in the Medical Management Practice (MMP) Domain are satisfactory performance in specified years of supervised medical management practice and satisfactory performance in the College MMP Oral Examination.
The MMP Oral Examination is designed as an assessment tool that aligns with and assesses the intended learning objectives of the role competencies of Medical Expert, Medical Manager and Communicator in the Medical Management Practice Domain. Learning objectives from other domains may also be assessed as they apply to the Medical Management Practice Domain.
The standard to be met is that the candidate is able to verbally describe and discuss the knowledge, skills and attitudes required for health management reasoning and action as an independent registered Specialist Medical Administrator in Australia and New Zealand.
The Censor in Chief is responsible for the conduct of the Fellowship Training Program Oral Examinations (the mandatory formative College Trial Examination and the summative MMP Oral Examination) and is supported in this activity by panels of the Board of Censors and staff in the College Office.
The Terms of Reference of the Board of Censors and the role of the Censor in Chief are available on the College website. The Board of Censors has been refining eligibility criteria, standard setting, formatting and moderation processes for the Fellowship Training Program Oral Examinations on a continuing basis and this regulation is updated to reflect recent changes.
2. PURPOSE
The purpose of this Regulation is to outline the standard rules and procedures for the conduct of the Oral Examinations for the MMP Domain of the Fellowship Training Program from 2020.
3. SCOPE
This regulation details the business rules associated with eligibility to sit the MMP Oral Examination, pre- examination standard setting, formatting of the examination, post examination moderation of results and notification of outcomes.
The College conducts mandatory Trial Oral Examinations and Jurisdictions may conduct optional Practice Examinations which utilise the principles and format of the MMP Oral Examination, with the exception that feedback is offered to the Candidates as part of these examination sessions.
4. BODY OF REGULATION
Eligibility to sit the MMP Oral Examination from 2020
Candidates will be eligible to sit the MMP Oral Examination if at the time of application for the examination:
- They have been credited with satisfactory performance in a minimum of two full-time equivalent years of programmed supervised medical management practice in accredited positions;
- They are currently working in an approved program of supervised medical management practice (unless exempted from this criterion for authorised reasons);
- They have participated in a formatively assessed College Trial Examination within the prior two years;
- They have not been unsuccessful at the MMP Oral Examination more than twice in the period of candidacy; and
- They are in financial good standing with their RACMA membership status and have paid all relevant College and training fees.
Eligibility limitations policy
A full-time equivalent year in supervised medical management practice is calculated as a minimum of 47 weeks in Australia and 46 weeks in New Zealand. This time includes up to two weeks of professional development leave. It does not include annual leave, parental leave or long service leave.
The years of experience in supervised medical management practice, and years of satisfactory performance required to have been completed prior to sitting the MMP Oral Examination are calculated on this basis.
The years of experience in supervised medical management practice are calculated inclusive of time approved through Recognition of Prior Learning and Experience processes.
Candidates are expected to be continuing in medical management practice at the time that they sit the MMP Oral Examination. The time fraction in which they are working should be that which has been approved for their training posts.
Candidates who have not been working in medical management practice are to provide details of why they do not meet this criterion. Details should be provided to the College Office at the time of application for the examination, in accordance with the Policy and Process for Special Consideration for the Oral Examinations.
Applications for leave, and exemptions from eligibility, in relation to illness, disability, family reasons, special learning programs outside supervised medical management practice and minor variations from time in practice, are made to the College office and addressed by the Censor in Chief.
Candidates who are unsuccessful at the MMP Oral Examination are not allowed to sit again until an examination scheduled for the following calendar year.
Candidates are entitled to sit the MMP Oral Examination three times before their candidacy compliance will be considered to have been breached. They may apply again for Candidacy in the Training Program, or apply for membership status as an Associate Fellow of RACMA. If they apply for another Candidacy, they will be assessed for recognition of prior learning and experience according to the conditions that apply at that time and be subject to rules concerning recency of practice for applications for further attempts at the MMP Oral Examination.
Applying to sit the MMP Oral Examination
The MMP Oral Examination is held at least annually. The College Trial Examination will be held at least annually. Candidates must complete and submit the application form that address the eligibility criteria by the required date.
All Candidates will be notified of examination dates as soon as details are confirmed.
A Candidate will be expected to attend the examination session allocated by the College. Every attempt will be made to accommodate distance travelled and circumstances of those Candidates who have been approved for Special Consideration. The examination for each Candidate will be conducted on one day. Sufficient examination sessions will be provided to allow all eligible Candidates who apply, to be assessed.
Format of the MMP Oral Examination
The MMP Oral Examination is an open-book examination encouraging Candidates to analyse and critique responses to set scenarios, for verbal discussion with examiners.
The MMP Oral Examination consists of four (4) interview stations and the Candidates present at all four stations.
Each station process is 40 minutes in length consisting of 20 minutes for preparation of responses and 20 minutes for interview:
- In the 20 minutes for Candidate preparation, Candidates will be presented with one scenario.
- Candidates then have 20 minutes to present the prepared response and answer questions from a panel of two Censors. The 20 minutes is split evenly with 10 minutes for the Candidate’s prepared response and 10 minutes to address questions from the Censor panel. The questions from the Censor panel will explore the Candidate’s understanding of issues that are relevant to the scenario and matters which arise from the Candidate’s response.
After the Candidate has left the room the Censors complete the assessment of the Candidate’s performance using a marking rubric against the knowledge, skills and attitudes required by the scenario. The Censor pair will assess independently for five minutes and will each award a score out of 15
When the Censor pair have recorded their independent scores, they will conduct a discussion about their respective scores. Based on this discussion one or both Censors may amend their scores. There is no requirement for absolute consensus, however each Censor is to be prepared to explain the score during the moderation meeting with all examining Censors. A final summed score out of 30 is recorded by the Censor pair.
Outcomes
After all Candidates in the session have completed four stations, all examining Censors conduct a moderation session to review the scores awarded for each of the four stations and to each individual de-identified Candidate. This session is led by the Moderating Censor appointed by the Censor in Chief.
The Panel of examining Censors:
- Identifies any scenarios that appear to have consistently anomalous scores (very high, very low) or other discrepancy;
- Agrees on how to moderate the outcomes from each station in which there may be anomalies;
- Determines the outcome for each Candidate according to the following algorithm:
- A Candidate has been successful if the summed score from both Censors is at or above 18/30 for each of the four stations
- A Candidate has been unsuccessful:
- if the summed score from both Censors is below 15/30 for one or more stations; OR
- if the summed score from both Censors is below 18/30 for two or more stations.
- A Candidate who has been considered neither successful or unsuccessful may be offered a fifth, supplementary station on the day. A Candidate will be offered a supplementary station:
- if the summed score from both Censors is at or above 18/30 for three stations AND
- if the summed score from both Censors for the fourth station is at or above 15/30.
- A Candidate who achieves a summed score from both Censors of at or above 18/30 for the supplementary station will then be considered to have achieved a successful outcome.
- If a Candidate declines to sit the supplementary station, that Candidate will then be considered to have been unsuccessful at the Examination.
Censors examining a fifth supplementary station will not have assessed the Candidate in any other station that day and will have no conflicts of interest.
Candidates who are clearly successful or unsuccessful will be asked to leave the Examination space (without being told their outcomes). Candidates who are eligible for a fifth, supplementary station are invited to remain in the Examination space to present for a fifth station.
All Candidates undertaking a fifth station on any one day will use the same pre-prepared ‘fifth station’ scenario.
After completion of the supplementary station, Candidates will be asked to leave the Examination space (without being told their outcomes).
The Moderating Censor is informed of the summed scores for the fifth supplementary station and makes a declaration to the Censor in Chief on the outcome of the Examination for all Candidates as either Successful or Unsuccessful.
Examination Results
Results from the Oral Examination are compiled and presented by the Censor in Chief (or delegate) to a specifically convened Education and Training Committee meeting conducted as soon as practicable during the week following completion of the examination. Candidates are notified of their Oral Examination outcomes after endorsement of their results by the Education and Training Committee. After that, notification of the outcome is made to Candidates’ Supervisors, Preceptors and the relevant Jurisdictional Coordinators of Training.
Standard setting and Moderation of Examination outcomes
Calibration takes place in the pre-event preparatory activities of an examination and involves:
- Design;
- Standard setting and consistency;
- Delivery; and
- Agreement on outcomes.
Moderation is the process that ensures the consistency of marking of summative assessment tasks in terms of:
- Alignment of assessment tasks with intended learning outcomes;
- Appropriateness of assessment content in terms of curriculum coverage;
- Standardisation of level of challenge; and
- Fairness to Candidates of processes.
Oral Examination Topics
Candidates will be assessed on their ability to convey to the Censors that they have the requisite knowledge, skills and attitudes/approach to satisfactorily deal with the examination scenarios.
The topics for assessment in this format are outlined in the RACMA Medical Leadership and Management Curriculum in the learning objectives of the role competencies of Medical Expert, Medical Manager and Communicator in the Medical Management Practice Domain. Learning objectives from other domains may also be assessed as they apply to the Medical Management Practice Domain.
The scenario settings include critical care, acute care and sub-acute care; community practice, public and private hospitals; and government and non-government health services.
Panel for Examination Questions
The Lead Censor for preparation of Examination Questions coordinates the drafting, evaluation, calibration and finalisation of questions for the MMP Oral Examination, in consultation with a Panel of Censors nominated for examination question development.
The Panel is responsible for preparation of a customised marking rubric for each scenario – which is based on the published RACMA template.
The Panel ensures that the standard for the writing of the questions and the wording in the rubrics is consistent.
The Lead Censor for Examination Questions will send the final drafts of all questions for the MMP Oral Examination to the Censor in Chief, Dean and an independent Censor (appointed by the Censor in Chief) to enable them to assess and verify the details for consistency.
The selected scenarios are sent to the nominated Censor Panellists who will be examining each scenario, before the Oral Examination Calibration meeting for their information and review. The Censor Panels meet before the Oral Examination to collectively familiarise themselves and calibrate the expected responses.
That calibration meeting is also an opportunity to identify to the Censor in Chief any previously undeclared conflicts of interest.
Previous Examination Questions
Examples of previous station scenarios are available to Candidates, Preceptors and Supervisors on the RACMA Website. These examples allow Candidates to familiarise themselves with the scope, format and style of; and acceptable responses in, examination scenarios. Candidates practising past examination questions with their Preceptors and Supervisors will be able to identify where they may have training or knowledge gaps. Many Candidates prepare by participating in Jurisdictional Practice Oral Examinations.
Pairing of Censors for the MMP Oral Examination
The Censor in Chief will oversee the pairing of Censors at the MMP Oral Examination as follows:
- At least one of the examining Censors within the pair must have private or public hospital medical administration knowledge or experience.
- Questions specifically relating to a hospital setting will involve Censors with recent hospital experience.
New Censors, who have completed the required induction and training prior to being eligible to examine will be paired with a senior and experienced Censor.
Conflict of interest
- All Censors must declare, to the best of their knowledge, any interests with Candidates and/or paired Censors that may prejudice/bias the Censor in the assessment of Candidate performance – examples of these could include:
- personal or family relationships;
- previous knowledge of Candidates which may introduce perceived bias (positive or negative, such as being on job interview panel, being a referee or previously working together, direct report etc.); and
- having been Candidates’ Preceptor/Executive Coach or Supervisor.
- Planning for adequate numbers of Censors on the day will also take into consideration the need to ensure that Candidates are not marked or observed by a Censor more than once on the day.
Note: The panel of participating Censors will be advised to Candidates prior to the examination. Candidates are also able to declare Censors with whom they believe there may be a potential or perceived conflict which introduces bias. In doing so the Candidate must list the reason behind their declaration.
Observers
Candidates will be advised when registering to sit the Fellowship Training Program Oral Examinations that examination stations are monitored either by cameras in the examination rooms (when the examinations are conducted “face to face” streaming to the Control Room at the Examination Centre) or by remote monitoring (when conducted by Zoom or similar technology).
Permission is sought from Candidates and Censors for videoing in the case of the College Trial Examinations.
The MMP Oral Examination is not audio or video recorded.
Candidates will be advised when applying to sit the MMP Oral Examination that there may be authorised Observers in the examination rooms or in the Control Room via remote technology used for the examination.
The Observer may be:
- Censors / Censors-in-training who will observe other Censors’ behaviour, technique etc.;
- an invited delegate from another College or a regulatory body to observe RACMA processes;
- Censor in Chief;
- College Office Staff, or
- other person approved by the Censor in Chief for the purpose of Examination integrity or training.
Post-examination Feedback
For those Candidates who are unsuccessful at the MMP Oral Examination, the College will provide a formal verbal feedback session of the MMP Oral Examination. The session involves the Censor in Chief (or delegate), the Candidate, and a member of the Board of Censors who was involved in the examination of that Candidate – particularly in relation to the questions in which a Candidate was unsuccessful. It is strongly recommended that Candidates involve their Preceptors and/or Supervisors in the verbal feedback session.
Candidates who wish to participate in a verbal feedback session are required to contact the College by the date nominated in their examination outcome notification.
In College Trial Examinations or Jurisdictional Practice Examinations, feedback is provided to the Candidate by the Censor pair for each of the stations. The Candidate is provided with verbal feedback and their marking sheets.
Bernard Nicholson Prize (Meritorious award)
The Bernard Nicholson Prize is awarded to the Candidate with an outstanding performance in the MMP Oral Examination. The recommended winner is determined at the meeting of the Board of Censors held after the completion of the examination. The awarding of this prize is discretionary and the College may decline to make the award if, in the opinion of the Board of Censors, no Candidate has achieved an outstanding result during the examination.
On the recommendation of the Board of Censors, the Education and Training Committee will endorse the award of the Bernard Nicholson Prize to the Board of RACMA for final approval.
Note: ‘The Bernard Nicholson Prize was donated by and is awarded in memory of Dr Bernard Nicholson. Dr Nicholson was a past President of the College and played a major role in its establishment.’
Appeals
Candidates can request to have a decision by a College Officer/Committee, or in this case, the Board of Censors, reconsidered and reviewed and can follow the College Appeals process in accordance with the Policy for Reconsideration, Review and Appeal of Decisions of the College Officers and Committees.
Format of the College Trial Examination
The format of the College Trial Examination is consistent with the format of the MMP Oral Examination.
The College Trial Examination is an open-book examination encouraging Candidates to analyse and critique responses to scenarios for verbal discussion with Censors.
The College Trial Examination consists of four (4) interview stations and the Candidates present at four stations.
Each station process is 40 minutes in length consisting of 20 minutes for preparation of responses and 20 minutes for interview:
- In the 20 minutes for Candidate preparation, Candidates will be presented with one scenario.
- Candidates then have 20 minutes to present the prepared response and answer questions from a panel of two Censors. The 20 minutes is split evenly with 10 minutes for the Candidate’s prepared response and 10 minutes to address questions for the Censor panel. The questions from the Censor panel will explore the Candidate’s understanding of issues that are relevant to the scenario and matters which arise from the Candidate’s response.
After the Candidate has left the room, they will wait outside for the Censors to prepare their feedback. The Censors assess the Candidate’s performance using a marking rubric against the knowledge, skills and attitudes required by the scenario. The Censor pair will assess independently for five minutes and will each award a score out of 15.
When the Censor pair have recorded their independent scores, they will conduct a discussion about their respective scores. Based on this discussion one or both Censors may amend their scores. There is no requirement for absolute consensus. A final summed score out of 30 is recorded by the Censor pair.
Feedback is provided by the Censor pair to the Candidate for each of the stations at the Trial Oral Examination. The Candidate is provided with verbal feedback and their marking sheets.
Candidates are encouraged to discuss their performance and feedback on the Trial Oral Examination with their Supervisor and Preceptor/Executive Coach to address deficiencies that had been identified by the Censors across the four examination stations.
Notification of the outcome of the Trial Oral Examination is made by the College Office to Supervisors, Preceptors and the relevant Jurisdictional Co-ordinators of Training.
Code of Conduct
All Censors and Candidates sitting the examination are expected to act at all times ethically, responsibly and in the best interest of the College. All Censors will adhere to the RACMA Officers’ Code of Conduct.
Candidates are expected to comply with the Terms and Conditions for Participation in RACMA Oral Examinations as defined in the Candidate Examination Agreement.
Censor Peer Review processes
In accordance with their participation in the College Continuing Education Program, Censors may elect to participate in formal peer review during their activities as an examiner at the College Trial and MMP Oral Examinations. It is expected that Censors will have participated in Censor Peer Review at least once in every three-year term to retain their status as Censors.
Examination peer review process
At College Trial or MMP Oral Examinations there will be times when identified senior Censors will view the scenario discussions of Censors from the Control Room/Centre. The observing Censor will record observations and commentary of the Censor performance on a peer review form.
The peer reviewers will provide their commentary to the Censor in Chief. The Censor in Chief will review the forms and may discuss the commentary with the observing peer review Censors. The forms will be provided by the Censor in Chief to individual Censors at the end of the day for reflection and feedback as a part of their own CPD processes.
Examination score peer comparison process
Significant variations across scores and peer review outcomes will routinely be analysed by the Censor in Chief and reported as components of evaluation of the Examination process.
5. ASSOCIATED DOCUMENTS
- Appointment and Training of Censors Policy
- Assessment in the Fellowship Training Program Policy
- College Censor Position Description
- Conflicts of Interest and Declaration of Interests Policy
- In Training Performance Report
- Jurisdictional Coordinator of Training Position Description
- RACMA Medical Leadership & Management Curriculum
- RACMA Privacy Policy
- Recognition of Prior Learning and Experience (RPLE) Policy
- Reconsideration, Review and Appeal of Decisions of the College Committees and Officers Policy
- Requesting an Extension in the Fellowship Training Program Regulation
- Research Training Program Handbook
- Special Consideration for the Oral Examinations Policy
- Terms and Conditions for Participation in RACMA Oral Examinations
- Terms of Reference for the Board of Censors
- Terms of Reference for the Training Progress Committee