August 2006 VIEW on Medical Education Postgraduate Institute for Medicine Outcomes measurement Measuring Educational Outcomes in CME Jan Schultz, RN, MSN Director, Educational Outcomes and Accreditation Michael R. Lemon, MBA President Commercial supporters want their grant dollars to optimize educational impact, including maximizing the quality and reach of educational activities. In today’s hyper-regulatory climate, the CME community is seeking to firmly demonstrate that physician participation in CME makes a positive difference in the quality of patient care. There are many challenges in this task. Levels of Outcome Achievement D.E. Moore outlines six levels of educational outcomes measurement: • Population Health • Patient Health • Performance • Learning • Satisfaction • Participation All CME activities are measured at the bottom two levels: whether participants liked the menu, venue, and speaker presentation style and how many attended. So, those providers that are reluctant to jump into educational outcomes measurement (EOM) already have a toe in the water. But as a community of CME professionals, we must begin to move up this ladder to demonstrate the educational impact of CME offerings. Gaining Access to Data Determining whether actual patient health outcomes change as a result of a CME intervention requires access to individual medical records. In this day of HIPAA regulations, gaining access to medical records by anyone outside the direct care provider chain for that patient can be challenging. But some aggregate data can be accessed if a CME provider partners with a healthcare provider or payer organization. But determining whether learning has occurred in a CME activity only requires access to the learner, which is already available through his or her participation in the activity. Making EOM Results Meaningful Research design is important; a poorly designed study makes the results and effort meaningless. It’s important to remember that EOM research is grounded in the social sciences, where randomized controlled trials are not necessarily the same gold standard that they are in pharmaceutical research. Sample sizes may be small, and cause and effect is not the common goal, but rather inference, association, or influence. Of the many things that can influence a physician’s change in knowledge or behavior, demonstrating that the physician’s participation in a CME activity is associated, more often than not, with the change can be determined through several methodologies. There are many choices: pre- and postsurveys, posing clinical assertions that ask for degree of agreement, case vignettes, chart reviews, collecting adverse event data, and so on. Some of these methodologies are precluded to those outside the direct patient-care arena. Each also differs in cost and time to execute; higher levels of measurement tend to be more expensive and involved. Commercial Support Increasingly, commercial supporters are not only willing to pay for outcomes research, but are demanding that it be included in grant submissions. They want to see if their investment in education is making a difference, and they want to be able to show regulators that they are focusing on physician knowledge and patient health in their CME funding. But outcomes measurement is not yet a universal expectation or requirement. Choosing the right level of measurement and a scientifically defensible methodology will assist in soliciting grant funding. Supporters want their grant dollars to optimize educational impact, including maximizing the quality and reach of educational activities. Choosing the appropriate EOM methodology for each activity that balances the cost with the benefit is clearly part of this equation. Paralysis of the Perfect With so many possible approaches to EOM, it’s easy to understand why some providers may be fearful of making the wrong choice. Waiting for an elegant methodology that measures actual patient or population outcomes could potentially paralyze the CME community. There are cost-effective outcomes methodologies available today that take into account the level of complexity of the CME activity, yet do not dramatically increase the grant funding needed. We must generate defensible data that demonstrate CME activities do make a difference in physician knowledge and performance. We also must demonstrate that analyses are used to determine what topics, formats, and audiences are addressed in the future. EOM is not only a process at the conclusion of a CME activity; done correctly, EOM is a critical component of the needs-assessment process. To continue the “toe in the water” analogy, the CME community may believe that the water is a little deep, but taking the plunge into EOM is well worth the effort. Postgraduate Institute for Medicine, Englewood, Colo., which is accredited by ACCME and ACPE and is an approved provider of continuing nursing education, focuses exclusively on partnering with nonaccredited organizations to jointly sponsor CME/CE activities. For more information, visit pimed.com.
