Visual Overview
Abstract
Background and objectives Despite the increasing prevalence of childhood kidney disease worldwide, there is a shortage of clinicians trained to provide peritoneal dialysis (PD). E-learning technologies may provide a solution to improve knowledge in PD. We describe the development of a virtual PD simulator and report the first 22 months of online usage.
Design, setting, participants, & measurements The PD simulator was developed and released on OPENPediatrics in January of 2016. A prospective study of international, multidisciplinary healthcare providers was conducted from January of 2016 through October of 2017. User action data were analyzed with descriptive statistics and linear regression. Paired t tests compared user pre- and post-test scores. User satisfaction was assessed by survey.
Results The simulator was accessed by 1066 users in 70 countries. Users spent a median of 35 minutes (interquartile range [IQR] 14–84) in the simulator. Users who completed the structured learning curriculum (n=300) spent a median of 85 minutes (IQR 46–95), and those who completed the entire simulator (n=63) spent a median of 122 minutes (IQR 69–195). Users who completed the simulator were more likely to scroll through text and access the simulator in multiple sessions. The 300 users that completed testing showed statistically significant increases in the post- versus pretest scores, with a mean increase of 36.4 of 100 points, SD 19.9 (95% confidence interval, 34.1 to 38.6, P<0.001). Eighty-seven percent (20 of 23) of survey respondents felt the simulator was relevant to their clinical practice, and 78% (18 of 23) would recommend it to others.
Conclusions This is the first reported virtual PD simulator. Increased test scores were observed between pre- and post-tests by clinicians who completed testing, across disciplines, training levels, and resource settings.
Podcast This article contains a podcast at https://www.asn-online.org/media/podcast/CJASN/2018_05_02_CJASNPodcast_18_6_O.mp3
- Child
- Curriculum
- Global Health
- Health Personnel
- Humans
- Kidney Diseases
- Knowledge
- Learning
- Linear Models
- Medical Education
- Nephrology Education
- Nursing Education
- pediatric nephrology
- peritoneal dialysis
- Personal Satisfaction
- Prevalence
- Prospective Studies
- Serious Gaming
- Surveys And Questionnaires
- Virtual Simulation
Introduction
In less well resourced countries, peritoneal dialysis (PD) remains the predominant form of both acute and chronic pediatric dialysis (1). Although the prevalence of childhood kidney disease in the United States and the world is increasing (2,3), a global dearth of clinicians trained to provide PD for children with ESKD is described, and medical trainees often have limited exposure to PD delivery (2–7). Furthermore, interest in nephrology as a career choice has declined in recent years, potentially related to this lack of exposure (4–7). Thus, the American Society of Nephrology created a Task Force on Increasing Interest in Nephrology Careers, which recommended the creation of novel educational tools to make nephrology more attractive to trainees (7,8).
E-learning technologies have been shown to be effective in teaching medical education topics, including nephrology (7,9). Virtual games and simulators appeal to Millennial learners; incorporate interactivity, competition, and adult learning theory principles; and provide scalable, convenient methods to practice skills, such as PD administration, in a safe, contextualized environment with directed feedback and individualized control of pace and timing of learning (7,9–13). A literature and website search identified an online nephrology game and a hemodialysis simulator (8,14), but no publications or sites describing virtual simulators or games for PD.
In a report by the Commission on Education of Health Professionals for the 21st Century, Frenk et al. (15) called for a revolution in medical education strategies to parallel the rapidly growing volume of information, and emphasized technologic advancements to promote mutual educational resource sharing across professional and geographic groups. Given the importance of PD worldwide, along with its high use in regions where training is less readily available, our goal was to develop a globally accessible, stand-alone educational resource to teach pediatric PD. We developed and released the PD simulator on OPENPediatrics (https://www.openpediatrics.org), our free-of-charge education website offering resources for clinicians caring for sick children, with registered users in approximately 2500 hospitals in 159 countries (16). We hypothesized that using the simulator would increase clinician knowledge of PD. Here, we describe the virtual PD simulator, and report usage patterns and knowledge gains by clinicians in the first 22 months of release.
Materials and Methods
This project and its component analyses were approved for exemption by the Boston Children’s Hospital Institutional Review Board. A team of subject matter experts, medical animators, and an application developer developed the PD simulator in 11 months. The subject matter experts included three pediatric nephrologists (two from the United States and one from South Africa), a pediatric intensivist, a pediatric resident, and a medical student. The simulator was designed by nephrologists who manage PD in different settings with a focus on general concepts needed to perform PD and topics relevant across resource settings and practice types, avoiding topics specific to certain settings and protocols. In biweekly meetings that drew from the subject matter experts’ varied experiences and areas of expertise, we continually re-evaluated and discussed the simulator’s globally relevant topics and the most generalizable ways to teach and test them (13). The PD simulator was developed using a previously published iterative development process that utilized prototyping and formal usability testing, adapted from best practices in education, software design, and quality improvement (13). An iterative, external peer-review process was conducted by nurses, residents, and critical care and nephrology fellows and attendings, across institutions and resource settings (13). Their feedback was used to refine content and structure (13).
The PD simulator was finalized and released on OPENPediatrics in January of 2016 (https://www.openpediatrics.org/assets/simulator/peritoneal-dialysis-simulator) (13). Its release was announced in presentations and an OPENPediatrics booth at academic conferences, banners on the website, social media accounts (Facebook and Twitter), and monthly Email newsletters to users of OPENPediatrics.
The simulator is available free-of-charge and accessible as a stand-alone resource, or as part of a structured learning curriculum including pre- and post-testing. The pre- and post-test questions included 15 case-based multiple-choice questions to assess critical thinking, patient management capabilities, and ability to integrate the knowledge obtained from the simulator into new clinical cases and challenges. The questions addressed the following categories of knowledge: basic concepts and definitions (calculation of ultrafiltration, properties of peritoneum as a membrane, properties of dialyzable compounds, selection of patients for PD, symptoms of catheter obstruction, indications for anticoagulation), selecting an initial prescription (selection of potassium amount in dialysate, choosing cycle number), changing a prescription (fluid overload, hyperkalemia), and troubleshooting complications (inadequate drainage from catheter, peritonitis, PD catheter leakage, catheter obstruction).
Virtual Peritoneal Dialysis Simulator
The PD simulator was developed for the learner to practice PD in multiple clinical scenarios over varying pediatric ages and disease states (acute kidney failure, chronic kidney failure, sepsis, peritonitis, congenital malformations), and promotes a systematic approach to setting and adjusting the prescription, mechanical complications, electrolyte abnormalities, and patient-specific management issues. The learner virtually diagnoses conditions, manipulates the PD set-up (moving the fill and drain bags, clamping and unclamping catheters), adjusts the PD prescription (cycle time, fill volume, and concentrations of electrolytes and dextrose in the dialysate), and administers medications (sodium supplements, intraperitoneal antibiotics, and fibrinolytics) in response to clinical assessment, fluid balance, and laboratory results. The simulator incorporates three levels of increasing challenge that the learner must complete sequentially.
A knowledge guide teaches principles of PD through text, embedded videos, and interactivity (Figure 1A), and is organized into the following sections:
Screenshots of the various components of the peritoneal dialysis simulator. (A) The knowledge guide. (B) The tactics. (C) The case studies. (D) Learner-controlled feedback.
Definitions (dialysis, osmosis, diffusion, convection, membrane properties)
Concepts (indications, contraindications, ultrafiltration, clearance, infection control)
Prescription (sodium, potassium, dextrose, heparin, antibiotics, fill time, drain time, dwell time, cycle number, fill volume)
Administration (fill phase, dwell phase, drain phase, patient monitoring)
Complications and trouble-shooting (peritonitis, catheter obstruction, hyponatremia, hypokalemia, hyperkalemia, peritoneal catheter leak, poor ultrafiltration, impaired filling or drainage)
The tactics are short, case-based problems for the learner to identify and address (Figure 1B). The learner is scored on actions in three categories: (1) PD prescription, (2) PD circuit, and (3) labs and medications, receiving feedback specific to each action (Figure 1D). The tactics include:
Electrolyte abnormalities (hyperkalemia, hypokalemia, hyponatremia)
Mechanical complications (catheter obstruction, clamped catheter, inadequate fill bag height, inadequate drain bag height)
Patient-specific management issues (dehydration, peritoneal catheter leak, peritonitis, poor ultrafiltration)
In the case simulations, the learner cares for a simulated patient in a more realistic and less directed manner (Figure 1C). A variety of simulated patients progress through a hospitalization, with learners tasked to administer, monitor, and adjust PD while recognizing and addressing multiple complications. Learners are again scored for actions in the three categories (Figure 1D). Learners win medals (bronze, silver, or gold) for achieving passing scores on each attempt.
Data Collection and Analysis
From January of 2016 to October of 2017, embedded site analytics collected individual user actions within the simulator (including time spent using simulator components, scrolling of text in the knowledge guide, answer choices in tactics, actions in cases, scores in tactics and cases, and time elapsed between each click). All anonymous and registered users that used the simulation during the included time period were included in analyses. For users with registered accounts on the OPENPediatrics website, demographic information (profession, sex, location of practice) and performance scores from pre- and post-tests, when applicable, were also collected. Only registered users were able to use the structured curriculum and participate in pre- and post-tests. Individual question responses and overall percent scores for the tests were collected for this subset of users. Paired t tests were used to compare pre- and post-test scores. We conducted question-level analyses by calculating the average number of users who answered each question incorrectly on the pretest, but subsequently answered it correctly on the post-test. We used linear regression to examine whether first post-test score was related to overall time spent in the simulator.
A small number of user actions (161of 307,659 actions, 0.05% of total) were logged with extremely long durations, which we hypothesize occurred when users were inactive within the simulator without logging out or closing the Web browser. These episodes occurred when the OPENPediatrics website did not automatically log a user out of the system after 15 minutes of inactivity, as it currently does. Thus, to reduce the risk of overestimating time spent in the simulator, the duration for these actions was reduced to 15 minutes. To collect satisfaction data, we sent a survey via Survey Monkey (Palo Alto, CA) to those who used the simulator in the first year of release, between January and December of 2016. Data were analyzed using Microsoft Excel and Stata/SE 13.1.
Results
The simulator was accessed by 1066 users in 70 countries from January of 2016 to October of 2017. Nine hundred and thirty-three users had registered accounts on OPENPediatrics and 133 were anonymous users. Table 1 presents user demographic information. Of the registered users, there were more nurses (n=531, 57%) than physicians (n=340, 36%) or other healthcare professionals (n=24, 3%), with nursing students (n=422, 45%) being the largest proportion of users. One South Korean nursing school utilized the PD simulator as part of their training, making up a large proportion of users. East Asia (n=345, 37%), North America (n=177, 19%), and Europe and Central Asia (n=78, 8%) had the largest numbers of users.
Demographic information of registered users of the peritoneal dialysis simulator (n=933) including occupation type, sex, and region
Table 2 presents the numbers of users who started and completed each section of the PD simulator, the median time spent per user in each section, and the median time per user to complete each section. Because the simulator requires the user to complete activities linearly, and the user must complete one section to proceed to the next, we observed a steady decline in number of users completing each section. The knowledge guide section demonstrated the highest degree of user retention with 72% (658 of 909) of users completing the section. The overall median time spent in the simulator by all users of the PD simulator (including those that did not complete the simulator) was 35 minutes (interquartile range [IQR] 14–84). Users who completed the entire PD simulator spent a median of 122 minutes (IQR 69–195). Users who participated in the structured learning curriculum spent a median of 85 minutes in the simulator (IQR 46–95), again including those that did not complete the simulator.
The total number of users that started and completed each section of the peritoneal dialysis simulator, and median number of minutes spent by users in each section
Figure 2 displays total simulator use times for all users, users completing the structured learning curriculum, and users who completed the entire simulator. Those who accessed the simulator as part of the structured learning curriculum or who completed the simulator had a much higher percentage of users that spent >30 minutes using the simulator. Eighty-one percent of users that used the simulator as part of the structured learning curriculum spent between 30 and 120 minutes in the simulator. Table 3 displays total simulator use times for subgroups of users with a single session versus multiple sessions in the simulator. Those who used the simulator for multiple sessions spent a longer total time in the simulator. The majority of users used the simulator in a single session (78%, 835 of 1066). Forty-one percent (26 of 63) of users who completed the PD simulator did so in a single session. The users who completed the simulator had an average of 3.5 sessions using the simulator as compared with 1.4 sessions on average for all users.
Number of minutes spent using the peritoneal dialysis simulator as a percentage of users, organized according to all users, guided learning pathway users, and users who completed the simulator. PD, peritoneal dialysis.
Median time spent in simulator, with mean pre- and post-test scores for users completing the structured curriculum (n=300)
In order to approximate whether users were actually reading text in the knowledge guide versus simply clicking through the sections, the knowledge guide required scrolling in order to read the entire text for some sections. An average of 11% (118 of 1066) of users scrolled through greater than half of those sections, suggesting that these users read the text and may have been more engaged with the simulator. Twenty-five percent of users that scrolled through greater than half of the sections completed the PD simulator (30 of 118), as compared with 3% of users that did not scroll through greater than half of the sections (33 of 948), supporting scrolling as a marker of engagement.
Table 3 presents changes in pre- and post-test scores. The pretest scores showed an expected increase in score with increasing training level. There was a statistically significant mean increase of 36.4 percent, SD 19.9 (95% confidence interval, 34.1 to 38.6, P<0.001), between pre- and post-test scores for the 300 users that completed the structured curriculum. Linear regression was performed to examine whether first post-test score varied in relation to overall time spent in the simulator, and there did not appear to be any significant linear relationship between the two variables (β=−0.04 [−0.11 to 0.02], r2=0.01, P=0.23).
Table 4 presents the average improvement in each question category for users who answered a question wrong in the pretest and correctly in the post-test (n=106). The greatest improvements were observed in selecting an initial prescription (63%) and changing a prescription (50%).
Improvement from pre- to post-test in knowledge categories of the structured curriculum users (n=106) that answered a question wrong in the pretest
Only 23 of 509 (5%) users responded to our satisfaction survey. Seventy percent (16 of 23) were satisfied or very satisfied with the simulator. Eighty-seven percent (20 of 23) felt the simulator was relevant to their clinical practice, 74% (17 of 23) reported it was up-to-date and reflects current best practice and evidence, and 78% (18 of 23) would recommend the simulator to others. Some reasons cited for not completing the simulator in its entirety included: the length of time required, technical issues, lack of relevance to practice (i.e., scenarios were too simplistic), and the interface being difficult to use.
Discussion
To our knowledge, the PD simulator is the first reported virtual simulator designed to teach PD concepts and management practices to healthcare workers and students globally. The PD simulator was designed by providers who practice across resource settings, and global users across all disciplines, training levels, and resource settings utilized the simulator and demonstrated significant increases between pre- and post-test scores. Interest, engagement, and motivation for self-directed learning with the simulator were supported by the time spent in the simulator and by scrolling actions, both of which were higher for those that accessed the simulator as part of the structured curriculum, as well as for those that completed the simulator. The majority of survey respondents were satisfied with the simulator and would recommend it to others.
As was shown by William and Huang with their nephology case–based modules (8), the PD simulator demonstrated significant knowledge gains through pre- and post-testing. Our results were observed across multiple healthcare disciplines, training levels, and cultures in different countries and regions, demonstrating the effectiveness of this globally applicable, interdisciplinary training tool. Hosting this resource freely online allowed users from 70 countries to share the same high-quality resource, fulfilling the recommendations by the Commission on Education of Health Professionals for the 21st Century (15).
We observed a low overall completion rate for the simulator (7%, 63 of 909 users). Few published studies describe virtual simulators or serious games that are released for free global access rather than as part of a controlled study (7,9–13). One study that evaluated a serious game designed to teach sepsis management (Septris) released for free access described a completion rate of 13% (17). Our completion rate is lower, which may be attributable to the longer median length of time required to complete our simulator (94.2 minutes, IQR 62.8–155.6) compared with Septris (20 minutes) (17). Also, the low completion rate may be related to the PD simulator’s linear design. This design, which included pretraining (training on game functionality before introducing content), mastery learning (having learners master one concept before trialing a second), and hints, was on the basis of adult learning theory and incorporated published best practices from the serious gaming literature (10–13,17–23).
The tactics are highly interactive, brief cases surrounding single clinical problems, more similar to the design of Septris than the relatively long, open-ended cases, and the completion rates for our tactics section were high (33%, or 301 of 909 of total users, and 54%, or 301 of 557 of users who started the section) (13,17). The tactics offer the engaging, case-based, and user-led learning that the cases do, but they require a relatively short time-on-task to complete, have clear-cut and narrow learning goals, and offer gratification and feedback that are more quickly received by the learner (10–13). Thus, they may strike a balance that optimizes user engagement, especially because the most cited reason for not completing the simulator was the time required to do so. Because we anticipate that more users may complete more of the simulator if the tactics were available earlier, we are exploring opportunities that allow users to demonstrate performance in tactics and then provide additional learning with knowledge guide sections specifically tailored to performance gaps for more customized learning.
Additionally, there were likely differing motivations for using the simulator. Those that used the simulator as part of the structured learning curriculum spent more time in it, potentially reflecting self-directed motivation to gain knowledge of PD or use of the simulator as a mandatory educational activity for a course or program. Attending physicians made up nearly 20% of users, but spent a median of only 24 minutes in the simulator (IQR 4–64), possibly reflecting their motivation to assess the simulator’s potential as a teaching tool for trainees. A more detailed qualitative evaluation to identify factors contributing to the low overall completion rate will be a focus for improving the PD simulator in the future.
In this study, we present the development and initial results of the PD simulator. Kirkpatrick’s model of program evaluation organizes methods for assessing educational interventions into levels of increasing sophistication (reaction to training, gain of skills or knowledge, application of learning in the workplace, and improvement in patient outcomes) (22–24). Like many studies evaluating serious games and virtual simulators for medical education, we evaluated the PD simulator through the second stage: gain of skills or knowledge (23,24). Future studies will focus on studying the effect of the PD simulator on practical clinical skills and applications, as well as assessing cost-effectiveness.
Conclusion
Across the world, there remain great disparities in both numbers of institutions and funds allocated to train healthcare providers. Freely sharing online resources may help equalize global health professional education. Here, we demonstrate that a virtual PD simulator was able to increase knowledge about PD concepts and management in an engaging, relevant, and efficient manner on a global scale to interdisciplinary healthcare professionals.
Disclosures
None.
Acknowledgments
We wish to thank Genuine for their creativity and guidance during development, especially Rob Brecher and Paul Devlin. We are thankful to Brittanie Marques and Sarah Kim for their beautiful animations and illustrations, the numerous reviewers and testers who have helped improve the learner experience, and the Boston Children’s Hospital Simulator Program for allowing us to demonstrate peritoneal dialysis in action in the simulation suite.
The authors also wish to thank the Technology and Innovation Development Office of Boston Children’s Hospital and the Department of Anesthesiology, Critical Care and Pain Medicine, Boston Children’s Hospital for their generous financial support for the development of the peritoneal dialysis simulator.
Footnotes
Published online ahead of print. Publication date available at www.cjasn.org.
- Received September 19, 2017.
- Accepted February 20, 2018.
- Copyright © 2018 by the American Society of Nephrology
References
In this issue
Jump to section
More in this TOC Section
Original Articles
Maintenance Dialysis
Cited By...
- No citing articles found.
Podcast