Sandra MonteiroMcMaster University, Touchstone Institute

Plenary IV: Research on Impacts and Outcomes

Perspectives on Successful Transitions for Internationally Educated Health Professionals

Sandra Monteiro, PhDMarch 2, 2017

Impact and Outcomes

Measuring Impact – typically means measuring change

Measuring Outcomes – typically means measuring learning, skill, competence

2/22

What makes a transition program

successful?

3/22

What are the outcomes of a transition program?Customizing program evaluationSandra MonteiroDavid Rojas

4/22

Agenda

• Describe the pre-residency education program

• Who should define success?– Examples of studies on predicting and

measuring outcomes• What are we doing to measure

“successful” outcomes? But also “failed” outcomes?– Innovative way to think about program

evaluation

5/22

Pre-Residency Program at Touchstone Institute

Provide international medical graduates with an exceptional learning experience on medical training and practice in Canada with a focus on:

– Patient-Centred Communication

– Collaboration– Professionalism– Health Advocacy– (Medical Expert)

6/22

Our Curriculum

7/22

Defining Success Touchstone Institute?

– Integrate best and defensible practices in education and assessment

– Able to address stakeholders’ needs– Provide international medical graduates with an exceptional

learning experience

Medical Residents?– Preparation for a successful residency– Prefer not to attend

Public? Residency Programs? Regulators? – Residents are ready to provide high quality care

8/22

Markers of success in education

Focused feedback

Goal directed teaching

Opportunities for deliberate practice9/22

Markers of success in education Focused feedback

• Information on their strengths • Opportunities to improve • Not specific to specialty

Goal directed teaching• Whose goals should we evaluate?

Opportunities for deliberate practice• Small Group Simulations • Not everyone is equally engaged

10/22

Markers of success in residency

From 2005 to 2009, … the CMG pass rate was 95%, while the IMG pass rate was 76%

Can the PRP do anything to address this? What factors influence these outcomes?

*Thomson, G., & Cohl, K. (2011). IMG SELECTION. Independent review of access to postgraduate programs by international medical graduates in Ontario. Volume 1: Findings and Recommendations; Volume 2: Analysis and background.

11/22

Factors that influence outcomes

Selection and admissions

12/22

13/22

• Medical Council Evaluating Exam Scores– Simulated office orals– Short answer management problems

• Demographics of Medical Residents– Country of birth (Human Development Index)– First language– Professional experience; internship

13/22

• First language other than English related to positive outcome on exam

• Prior professional experience and country of birth, slight benefit for outside Canada

Studies of outcomes have an impact• The development of a multi-institution data

sharing, processing, and governance agreement between Ontario PGME Programs and the CFPC and RCSPC

14/22

Factors that influence outcomes

Challenges in social integration

15/22

Program Evaluation in Health Professions Education

18/22

• Identify what works• Often focused on satisfaction ratings

Program Evaluation in Health Professions Education

Outcome based

• Rigid model• Focus only

on one outcome

Theory based

• Rigid model• Acknowledges

complexity but is unable to capture it

Holistic program evaluation

•Flexible model•Understand emergence•Acknowledges and aims to capture complexity

18/22

Holistic evaluation

Parker, K. (2013). A better hammer in a better toolbox: Considerations for the future of program evaluation

Acknowledges and aims to capture complexity through emergent properties

How do we define emergence?

Emergent properties are those unexpected behaviors exhibited by the system that cannot be explained by elements, or the interaction of the components in the system.

Emergence from Design

•Director•Coordinator

Emergence from Expectations

•Students•Instructors

Stakeholders

20/22

Example of unintended outcomeNot engaging with simulated sessions

Trainee dissatisfaction with PRP

• Not enough specialty content; doesn’t apply

• Frustration with transition program, seen as a hurdle and a burden to overcome

• Lack of communication to trainees

21/22

Future Directions

• Communicate with all stakeholders to better define goals of the PRP and areas for improvement

• Address intended but failed outcomes and unintended negative outcomes

22/22

Thank you

2017 Perspectives SymposiumMarch 2, 2017

#Perspectives2017