Our local utility company offers free “energy audits." An auditor will check the robustness of your heating, ventilation, and air conditioning (HVAC) systems and examine your home’s physical envelope — walls, floors, windows, doors — to assess it for leaks. When you live in New England (cold winters, old houses), an energy audit is a welcome service. It’s not that I need anyone to tell me my100-year-old windows are leaky — just stand in front of them and feel the breeze — but when an auditor can add up the money flowing out those windows, I am more motivated to address the problem. What if you could have an “energy audit" of your clinical trial? Why not conduct an assessment to quantify the effort (and therefore, money) we’re wasting through leaky oversight, murky processes, and redundant work? The clinical study lead or line manager, like the New England homeowner, might be moved to remediate these issues when faced with a dollar figure that represents potential cost savings. Conducting a Clinical Energy Audit A clinical energy audit might start with a look at the efficiency and robustness of the study’s infrastructure: the protocol, plans, and processes that define the time, quality, and resource targets for the study. These are, essentially, the HVAC system for a clinical trial. A weak infrastructure may not have immediate consequences but can cause problems in the long run that necessitate rework, cost overruns, and change orders. First, the auditor reviews the following components: Tasks and accountabilities. The typical contract has a high-level scope of work. This is natural; contracts are finalized early, and it doesn’t make sense to focus intently on a downstream process like protocol deviations when the protocol is still being hammered out. If the team never gets around to defining accountabilities fully, though, they may find that multiple people — or none — are performing certain tasks. The interstices between the sponsor and vendors are particularly fertile ground for these disconnects. An energy audit can evaluate the protocol and supporting documents such as statistical analysis plans, monitoring plans, data management plans, contracts, and training materials for clarity, consistency, and completeness with regard to tasks and accountabilities. One team, for example, found that they and their contract research organization (CRO) held different views of what would happen to the CRO-held trial master file (TMF) documents after study closeout. The contract specified that the file would be transferred post-lock, but the team had never defined the process for resolving issues. Thus, the sponsor was expecting to have CRO personnel on hand for several weeks post- transfer to locate missing documents; the CRO thought the sponsor’s acknowledgment of the transfer would mark the end of their association. This sort of contractual ambiguity can be resolved without additional charges if expectations are set early but can result in a change order if it’s perceived as a late-breaking demand. Timelines. A clinical energy auditor looks at the content of the timeline, but also whether it is used as conversational currency in team discussions. Successful study teams talk productively about time, focusing on durations and dependencies rather than milestone dates. When one task slips, the study lead initiates a conversation about subsequent critical path tasks with a goal of shortening durations to avoid further slippage. The absence of those conversations is an indicator of future cost overruns, as the team either throws resources at every problem to make the original timeline or lets additional milestones slip, both of which cost dearly. It’s also a good indicator of team burnout and cross-functional resentment. For example, one study team focused intently on the first patient in date without working out the critical path leading up to it. After protracted vendor contract negotiations, they realized the interactive web response system (IWRS) required for randomization would not be ready on time. The team paid a premium to the vendor and conducted user acceptance testing (UAT) over a weekend to meet their timelines, benefiting neither their budget nor their mood. Budget. Few industries run multi-million-dollar projects with as little discussion of the budget as we see in pharmaceutical clinical development. A savvy energy auditor looks for clearly defined budget tracking, reporting, and forecasting processes that provide information to support decision-making. No effective team makes decisions solely on cost, but a team that ignores costs is bound to be surprised by them. One biotech company, for example, considered adding sites to bolster stagnant enrollment numbers but rejected the measure because they did not want to seek a budget increase. At study’s end they were surprised to find that they had overrun both the budget and their timelines. Their calculation of the cost to add sites failed to take into account the cost of not adding them: longer enrollment increased vendor project management fees, technology fees, and other costs. A healthy infrastructure is all about cost avoidance. An energy audit can also evaluate in-progress tasks for “leaks"— inefficiencies and redundancies with quantifiable costs. The tasks that are vaguely defined, targeted and budgeted are obvious candidates for this phase of the audit. Some common “leaky windows" are discussed below: Meetings. A study team meeting is a surprisingly expensive undertaking. One “virtual" team estimated their weekly study team meetings at $6,000 an hour. Assuming fifty meetings a year over two years, that brings the total cost to $600,000. It would be shortsighted to cancel meetings just to save money: a team needs to meet. But what are you getting for that price? A close look at this team’s minutes revealed that meetings were little more than status updates with many extraneous invitees. Important discussions and key decisions were typically deferred to other meetings. Post-audit, the team found it could cut these general updates to a monthly half-hour while the key issues were handled in the smaller side meetings. Adopting the good meeting practice of publishing the meeting objective in the invitation helped both functional and ad-hoc meetings become even more productive. Data collection and cleaning. There are two expensive mistakes in CRF design. The first is collecting too much data. Every data point has associated costs, including CRF development time, edit check specification and development, user acceptance testing (UAT), site fees, lab processing, source data verification, data management review, and medical review. By quantifying these costs, an audit can help the team decide whether the benefit of collecting a data point outweighs the cost. The second mistake is failing to collect data required for a meaningful analysis — hence, the fear that drives mistake No. 1. Critical datapoints may be omitted, but more often they are poorly defined, leading to different interpretations of the data. One study relied on study drug compliance data for a key endpoint, but the CRF that collected study drug termination did not differentiate clearly between drug that was stopped due to early stopping rules and drug stopped for other reasons. Sites responded inconsistently, resulting in extra cleaning and analysis time. Monitoring visits. We’ve all heard that reducing source-data verification (SDV) will reduce the number of monitoring visits and thus reduce costs. But reduced SDV has potential for cost avoidance even if the number of monitoring visits remains the same. SDV alone has been shown to add little value: it takes a lot of time and results in few significant data changes. Other monitoring activities arguably add considerable value, such as training site staff, reviewing drug accountability, ensuring protocol and GCP compliance, and verifying adherence to data capture conventions. Requiring 100% SDV encourages site monitors to give these other activities short shrift. Site monitors are often resistant to risk-based monitoring because they conflate SDV with those other value-added activities. We need to reframe monitoring activities so that they focus on less on traceability and more on logic, consistency, and compliance. The site monitors for the study with the confusing drug termination CRF, for example, were able to verify responses on that CRF against source without noticing that sites were completing the forms inconsistently. It’s not enough to free monitors from checking all those SDV boxes; we need to give them tools for looking at data differently. An auditor, using a simple algorithm that calculates SDV time per datapoint, can give a sponsor an idea of how much on-site time is spent on value-added tasks and may provide an incentive to reduce the SDV requirement. Trial Master File. TMFs for outsourced studies can suffer from duplication. One study, for example, was maintaining many document types in two different sponsor sites and three different CRO regional locations, in both electronic and paper formats. At the end of the study, the CRO shipped its files back to the sponsor, now the proud possessor of five copies of the same document. Each time a document is received, tracked, or scanned, there is a cost. In addition, a team dealing with a large volume of duplicative documents may fail to identify missing items, resulting in potential inspection findings with their own associated costs. By specifying who holds each document type and what is transferred to the sponsor at the end, the team can reduce duplication. Project managment. This is perhaps the “leakiest window" of all: almost every vendor contract includes a healthy percentage for project management, yet sponsor study leads tend to oversee vendors through “high-touch" activities that effectively duplicate these expensive services. Examples abound. One sponsor paid its CRO project manager to review monitoring reports and escalate issues, but the sponsor study lead continued to review all reports to “stay in touch" with the study. Another sponsor took the minutes at CRO conference calls because they didn’t like the way the CRO did it. A vendor oversight plan that defines both hands-on and data-driven oversight measures helps sponsors avoid this pitfall, but only if they follow it. It’s no surprise that the inefficiencies that drive up costs are the same ones that lower quality and lengthen time. These elements are inextricably linked. By emphasizing the budgetary consequences of inefficiency, a clinical energy audit can give managers the information and impetus they need to change behaviors to improve cost, quality, and speed. This Old House: Conducting a Clinical Trial Energy Audit Denise DeRenzo Lacey, Principal ­Consultant, Halloran Consulting Group Halloran Consulting Group is a boutique consulting firm for the life-sciences industry dedicated to helping companies achieve excellence in regulatory, quality, and clinical development. { For more information, visit hallorancg.com.

Download PDF