Transforming Physician Education With Adaptive Learning

Speaker: Matthew O'Rourke

Amount of medical knowledge is increasing exponentially. While physicians don’t need to know everything, part of the problem is that they don’t know what they need to know. He started with a medical journal, it evolved and became more multi-media and interactive. However, to maintain relevance they needed to go beyond that format completely. Their needs assessment showed:

What MDs want:

  • Progressive learning

  • Make the most of my time

  • Make it rewarding for me

  • Learn it my own way

What MDs need:

  • To demonstrate clinical knowledge in training

  • To demonstrate clinical knowledge in exams

  • To maintain and grow clinical knowledge in practice

Current learning science:

  • Spaced repetition

  • Metacognition (do you know what you don’t know?)

  • Bite-sized learning

  • ILPs (individualized learning plans – a trend now in medical education)

  • Adaptive learning

  • Physicians are frustrated by the piecemeal approach of occasional required courses, the need for both continual and just-in-time learning was great.

Adaptive learning (as opposed to traditional):

  • Adapts to individual

  • Assesses first, then offers targeted content

  • Takes into consideration what learner already knows and focuses on knowledge gaps

  • Is self-directed, with follow-up on what the learner has trouble with

They created NEJM Knowledge Plus, a tool that either provides questions or lets admins create them. It is used mostly for CMEs and also preparation in taking the Board exams.

There is a self-assessment dashboard for the learner to see. The majority of their users not only access on phone, but don’t even use a desktop. They use the term “probes” instead of questions, since traditional multiple-choice questions are one way to do this, but not the only way.

When learner answers the question they have to also answer their confidence level in their answer - “I know it”, “I think so”, “not sure”, or “no clue”. So if they get it right, but without confidence they will see this kind of question again to make sure it wasn’t a lucky guess. Repeated questions are spaced out to help with mastery. It also “primes the pump” for the learner to think more about their response.

The Ebbinghaus Forgetting Curve shows how forgetting time shortens the more one is exposed to the knowledge.

For the content creators, the hardest part of the process is to provide the feedback on the questions – why the correct answer is correct, and why the other answers are incorrect. They are finding this also leads to a good click through rate on the additional resources.

They also have a “challenge us” with each question – a way for learner to give feedback to them to help them keep on top of information and keep it current. This also helps learner to more deeply think about the question to articulate why they think it might be inaccurate or unclear.

Metacognition

They show the learner how their confidence level aligned with the correct/incorrect answer. Doctors who have a high degree of confidence and are often wrong are the most dangerous, and need to have this self-awareness. Other than system error, over-confidence is one of the biggest causes of medical error.

Beyond Gamification: Think Like a Game Designer to Create Engaging, Meaningful Instruction

Speaker: Dr. Karl Kapp

Dr. Kapp began the presenting by asking the audience for five things they think about as an instructional designer when they are designing a course. Their answers included, branding, how much time the course would take, measures of success, outcomes, and platform to be used.

I was not surprised to observe that in a very large ballroom of designers, nobody mentioned motivation.

He presented a scenario and tried to use pollev.com to let audience pick teams and vote. Unfortunately it seemed he had locked his questions and so had to get answers by show of hands. Ideally Pollev lets audience answer questions by browser or app and display their answers immediately.

According to Dr. Kapp, research now shows that people are wired to enjoy learning, but more facts are retained through stories than bulleted lists.

Learners are motivated by challenge, curiosity, control, fantasy, cooperation, competition, and recognition. If something is too easy nobody cares, figuring out they don’t need to pay attention. Learners want some sense of control over their learning as well.

Kapp referred to the unholy trificate of gamification – points, badges, and leaderboards. These tools have their role, but sometimes they are not enough or the right approach for the audience. Perhaps for instance engineers would rather solve problems. Malone’s theory of intrinsically motivating instruction shows that the impact of external motivation can decrease over time.

Fantasy helps learners to apply old knowledge to understand new things and help them remember the content. Emotionally people can connect with the experiences in the fantasy and no have to bring with it “real world” concerns or fears.

In games, failing is allowed, it’s acceptable, and it’s part of the processes. Continual success would bore the learner. Research indicates that our brains grow when we make a mistake because it is a time of struggle. However, some research says you learn more from the failure of colleagues than your own failures. With your own failures you make excuses and justifications, there is distance with observing and learning from failures of others. That’s why it can be educational to see a character in a story fail.

We as designers need to point out to learners what they don’t know, and create cognitive dissonance that they want to have solved. (paraphrasing Knowles). To do so he like to send out a survey pre-class to get people discussing and wanting to know the answers before they start.

Curiosity and mystery should be drawn into the design – gaps between known and unknown that needs to be resolved. The following enhance mystery:

  • Novelty

  • Complexity

  • Inconsistency

  • Surprise

  • Incomplete information

  • Inability to predict the future

Videogames don’t start off with a lot of instructions – they throw you into the action right away, providing complexity before mastery.

Quizzing learners with embedded question a very good way to build in spaced repetition and increase retention and recall. Most gamified platforms use this.

Learners shouldn’t be able to do things so quickly. Too little challenge leads to boredom. Start with challenge, and build on the challenge level as they master. But too much challenge can make learners anxious. How do you put learners into flow state? Learners have to know the task can be achieved. Present clear goals, give control over actions (autonomy) and allow them to concentrate.

Choices allow learners to be motivated, give them a sense they can master content. When given control over the learning, learners invested more time and retained more. Sometimes it helps to let learners choose their path in the beginning to make it seem more applicable to their circumstances - especially if you have some difference in your learners.

On a side note, people who study in-person presentations have noticed a “magic T” formed of the learners in front rows and aisles who tend to learn more. But it’s self-selection, not that the teacher focuses there most.

At the end of the presentation Dr. Kapp had the audience summarize key takeaways of the presentation. Their responses were:

  • Competition

  • Curiosity

  • Choice

  • Non-linear

  • Fantasy

  • Mastery

  • Learning from failure

  • Fun

Dr Kapp pointed out that none of these were on the audiences list in the beginning.

Other key takeaways:

  • Military is in the forefront of using gamificaiton in training – war games, strategy games.

  • Medical industry was one of first industries to do so.

  • He is inspired by Choose Your Own Adventure books and cheap spy novels.

  • He also tries to talk to clients and see what they like (for example some might like Survivor type shows, which would be a good thing to adapt to games).

Microlearning: Making a Tangible Difference in Healthcare Training and Patient Outcomes

Speaker: Rich Lanchantin of Qstream

 “Qstream is a continuous learning program for knowledge retention and critical thinking skills that is respectful of time”.

 What is microlearning?

  • Breaking information into small segments to get the attention of busy employees?

  • Information delivered “just in time”?

  • Series of short videos and presentations organized on a web portal?

  • Scenario-based challenges delivered over spaced intervals to assure retention and behavior change?

For Lanchantin they are all correct, but Qstream focuses on the scenario-based. They use the “spacing effect” (i.e. spacing content out to improve retention) and the “testing effect” – scenario based Q & A which best reinforces learning. It also helps improve critical thinking.

Merely chunking information into smaller segments in an unsystematic way has proven ineffective.

Qstream sends a notification via email or mobile device that the recipient has a question today. The activity only takes 5-10 minutes to do, which learners really appreciate. They recommend rich, relateable scenarios with a question and complete explanation. Then the learners sees the leaderboard to show how they are doing, which is very motivational. Learners can be put into teams to compete against each other.

Qstream claims approximately 92% engagement, even in cases where the activity is not required. Engagement is defined as going through the program at least once.

They sell Qstream to many of the major hospitals who put in their own scenarios. Different departments use it, as well as for residency education. Also used by pharma and biotech – in particular sales reps, as there are thousands of pharmaceutical sales reps who need to be kept update. Also used by clinical trial monitors, and medical device companies.

They get frequent feedback that short time span is really appreciated.

Users can comment on the questions. This not only makes it social, but provides immediate feedback for the designer, in case they want to refine or improve the questions.

It is customizable, but typically the users will go through a sequence of getting a question every other day for about 8 weeks. They will usually see the same question more than once to reinforce the learning. It adds up to about 2 hours of content, 5-10 minutes at a time.