Evaluation System
One of the important pivots we made during the development, is to define ourselves, the Real Empathy system, as an initiator and an exam in the whole training process. Therefore, providing the final report to students for review is as important as the training experience itself.
How to evaluate the empathy level of people? It’s hard to quantitatively measure by humans, even impossible by machines. Therefore, we consulted clients and University of Pittsburgh’s standardized program to learn how they evaluate their medical students’ empathetic behaviors.
First, we reviewed the checklists from the SP program for student performance evaluation. In the following, we are going to list several discoveries or reflections from studying the references.
- Different Scenario Setting
Usually, our clients use Bad-News-Deliver-like scenarios to teach students to show empathy in the conversation. Considering the error disclosure case is a heavy doctor-leading conversation type, it’s very different from the current scenario we use in Real Empathy, a patient-oriented conversation flow.
The Difference means we cannot just copy-paste the sheet for our evaluation system, and we need to tailor it. But it also implies the potential of this project that we expand the field of scenarios and compensate for a shortage of current cases which makes students learn to active-listening as a passive role in the setting.
- The Process of Experience:
Generally speaking, every scenario is under a similar empathy-learning structure. According to the checklist, we realized our experience design is almost correct. In sequence, Setting Up Interview(Greeting), Cognizing, Understanding, Communicating, and Intention to help / Summary. For each step, there are some guidelines and milestones.
- In the stage of Setting Up Interview(Greeting), healthcare providers need to focus on building connections with the patient. Taking time to finish the self-introduction is important, but also be aware of managing time constraints and interruption.
- The Cognizing stage is critical to obtain the patient’s invitation regarding how they would like to receive/discuss the information. Careful listening is required.
- Understanding and Communicating are the keys to empathy. Giving enough time for the patient to express themselves. Observe any emotion on the patient and identify the reason. Repeat/rephrase what the patient said to make sure you understand them correctly. Let the patient know that you are actively listening by making acknowledgment or connection statements.
- Last but not least, Intention to help / Summary. If the patient indicates they are ready, discuss or provide a further plan to help.
- Common mistakes should avoid:
- Avoid using jargon. Using jargon is not helping to understand and make patients feel distant. Therefore, we didn’t train/tune our models to learn/identify abbreviations of medical terms.
- Avoid excessive blunt speech. Directly to the point may cause uncomforted. Try to decorate or soften your words.
- Avoid overwhelming information. Overloaded information is distracting and confusing. Try to give information in small chunks.
Also, we did some studies on empathy scales in practical use. We learned there is a well-known and world-wide using evaluation tool called the Jefferson Scale of Empathy(JSE). It’s long-term research on evaluating the empathy level of medical students/healthcare providers by using a survey with over 80 questions. We want to dig deeper and find some insight to leverage our project with this tool. However, JSE is an exclusive service for healthcare institutions or medical schools, and there are not too many open-sourced papers or research on the internet.
Considering JSE is using self-evaluated scale questions to determine individual empathy level, we are thinking about what if we could obtain or generate some subjective data to support the empathy scale. We are not equipped with the ability to analyze the statistical data, but we can record down those raw data and provide them for future research. Now, we are able to store the timestamp of each response, the empathy level changes with timeline, the word count of response, the detected speech input, and its confidence. Furthermore, we could probably monitor the data of eye contact or tune analysis in the future when the technology improves.