Validating Data Is More Than Positive Affirmations
When you get a dump of project data, do you ever find yourself trying to validate its feelings rather than the data itself? “You’re intelligent and useful,” you might say. “You’re strong and focused. You belong in the world, and there are people who need you.” And so on.
It’s hard to say if this really makes the data feel better, but it’s not the kind of data validation you should be doing. I only ask because this is precisely what happens in the case of data dumps from an Electronic Medical Record (EMR) system (sometimes called an Electronic Health Record or EHR). I’ve seen numerous examples of people accepting this information as a source of truth to be defended at all costs; they do constitute legal records, after all. Unfortunately, EMRs don’t offer the kind of reliable data that people think they do, and hospital systems would do well to learn better methods for exploring and validating their outputs.
In this blog, I’ll discuss data validation from the healthcare perspective, especially data from an EMR, though I find these issues exist in any electronic data record.
Let’s start with the boring stuff. According to Informatics.com, “Data validation means checking the accuracy and quality of source data before using, importing, or otherwise processing data. Different types of validation can be performed depending on destination constraints or objectives. Data validation is a form of data cleaning." Some people separate data validation and data quality. According to Atlan, the difference between data validation and data quality is, “Data validation is the process of ensuring that data input meets predefined criteria and standards before it is processed, acting as a crucial checkpoint for accuracy. On the other hand, data quality is a broader concept that encompasses the overall completeness, consistency, and reliability of data within a system.” For our purposes, we’ll use the term data validation to encompass both data validation and data quality; from my perspective, incomplete or unreliable data would not be considered valid.
When we get data from an electronic system, EMR or otherwise, how do we validate it? Let’s start with how data gets into an EMR.
The computer time stamps when an action occurs. For example, a physician enters a note, which timestamps an entry as “Exam Complete.”
A staff member enters data from an Observation. I see the patient enter the operating room at 0735.
A staff member enters data from a previously logged observation. I see the patient enter the exam room at 1312 and note this down (with a pen). Later, that information is transferred to the EMR.
A patient enters the information in a log, which is later entered into the system. I’ve seen emergency departments, urgent care clinics, etc., where patients write their name and time of arrival on a log, which is later entered into the system.
Now, let’s look at some common problems. The first of these is data definition. Do all the staff understand and agree on the definitions around these entries? I worked on a project in a Radiology department where the time stamp “Exam Start” to some techs referred to the time when the patient entered the exam room, and to other techs, it meant the time when the patient was put on the x-ray table. Since this was a case of the time being entered by observation from the Radiology Tech, there were some discrepancies around the “Exam Start” data point. Sometimes, a data dictionary defines these terms; other times? Nothing. And if there is a data dictionary, who defines the terms there? Sometimes, IT folks who install the system define these terms, and depending on their level of clinical knowledge, they may not explain it well. It doesn't do much good if defined and not shared and reinforced with the staff.
To have valid data, everyone entering data must have the exact same definition for the data element. When we download data, we need to know both the formal definition and the informal one (the one used by the staff) so we can use the data appropriately. If we download “Exam Start” and “Exam End” in Radiology and we think this is the time a patient is in the Radiology suite, but instead, it is the time a patient gets on the table until they are off the table, we can make terrible decisions regarding our Radiology suite utilization, a critical issue if we’re using this data to decide on scheduling or expansion. If we use these times to drive a simulation model, as described in our book Simulation Solutions: A Practical Guide to Improve Patient Flow & Facility Design in Healthcare Operations, our results will be skewed because patients will spend longer in Radiology than our model shows.
The next common problem is human error, often stemming from their understanding of the importance of the data they’re responsible for. People make mistakes when they enter data, especially clock data. A problem I sometimes see is that data is intended to be entered using a 24-hour clock, but the clock the staff uses is a 12-hour clock, so they enter 0800 at 8:00 p.m. when they should have entered 2000. And sometimes, people just make mistakes. The related problem is data importance. Who cares about the data?
I once worked on an ED project where all the ED-related data entered by the staff was garbage. Why? Was the staff intentionally putting information into the system to confuse things? No, the staff had decided that inputting data wastes time. Nobody cared about the data and having to enter it (the computer defined these fields as required), and so staff had to make an entry before it would move to the next field) was getting in the way of them doing their job to render care to patients.
No one had explained to the staff how we could use this data, had it been accurate, to understand problems and bottlenecks in the ED workflow and improve the patient’s and staff’s experience. So, we took a step back and explained to the staff why the data was important and how we could use it. We showed examples of how we used this data in other EDs to improve workflow, reduce the patient’s avoidable suffering, and improve the staff’s work experience. Armed with this information, the staff began entering data accurately, and we were able to make improvements. If we haven’t used data from a system before, it may be routine for the staff not to take the time to enter information accurately. This is related to Organizational Change Management issues, which I discuss in another article.
The last common problem is clock synchronization. This issue is commonly addressed in Measurement System Analysis (MSA).
Practitioners of Lean Six Sigma should be familiar with Measurement Systems Analysis (MSA), sometimes colloquially called Gage R&R (Gage Repeatability and Reproducibility), and indeed, if you were one of my students as a Green Belt, Black Belt, or Master Black Belt you should be well versed with these tools.
According to Wikipedia, “A measurement system analysis is a thorough assessment of a measurement process and typically includes a specifically designed experiment that seeks to identify the components of variation in that measurement process.” So, when we look at errors in the synchronization data, what we see is of two kinds. The first is the actual error in the data, and the second is the error in our measurement process or system. Errors in the measurement process distort our view of the actual data. If we don’t understand the error introduced by the measurement system, we don’t understand the error in the data.
In a manufacturing setting, much of our measurement uses gages to understand if parts are sized appropriately to fit with other parts. In the hospital setting, a great deal of our measurement involves time. Time waiting to be seen in the ED, time in an operating room, etc. This brings us to the question raised earlier about clock synchronization.
As I was teaching a Green Belt class and discussing how using different clocks to measure the length of a process could be affected by the clocks not being synchronized, I could see a student in the back of the room focusing intently on this discussion. My favorite example of the clock problem comes from my Surgery consulting experience. I would download information from Surgery Information systems (this was in the late 80s and early 90s) and often find that the time of incision (the first cut in the operation) had been logged before the In Room time (the time the patient enters the operating room).
While you may enjoy the mental image of a surgeon kneeling on top of a patient on a gurney and slicing them open as the gurney rolls down a hall, this kind of TV stuff should obviously not be happening in a hospital. Rather, this situation usually arose as a result of using different clocks for when the patient entered the room vs. the incision time.
As I discussed this situation, I could see the student’s eyes getting bigger, and at the break, the student bolted from the room. This student managed an outpatient blood draw area, which both drew blood and did EKGs. Patients arrived at a check-in desk, and their lab slips were stamped with the arrival time. When the phlebotomist completed the blood draw or EKG, they stamped the slip with the completed time. These times were entered into the Lab System, and various metrics were calculated, including throughput time for patients (this was before the prevalence of computerized Laboratory Information Systems, Electronic Medical Records, etc.).
In checking the information, the manager found that the time stamp in the phlebotomist work area was 5 minutes faster than the time stamp at the check-in desk (when the check-desk stamp said 0800 – the work area time stamp read 0805, thus adding 5 minutes to the throughput time). While one may think 5 minutes is not too bad, the entire process could be done in 10 to 15 minutes on a good day, meaning the measurement system was introducing a 33% to 50% error. This underscored to the class the importance of clock synchronization and measurement system analysis.
Let’s revisit and define Gage R&R. According to the MSI Management and Strategic Institute, “Gage R&R is a structured approach that quantifies the variation in measurement systems, distinguishing between two types of variability: repeatability and reproducibility. Repeatability refers to the variation in measurements taken by a single operator (staff, worker, etc.) using the same instrument when measuring the same part or item multiple times. It assesses the consistency of measurements under the same conditions. Reproducibility, conversely, assesses the variation in measurements when different operators (staff, workers, etc.) or instruments (e.g., clocks) are used to measure the same part (or in an ED, for example, process). It evaluates the consistency of measurements across multiple operators or equipment.” Italics added for clarification.
How do we quantify the variation due to reproducibility when looking at times from our EMR? Let’s take our Emergency Department (ED) as an example. Reproducibility involves different “operators” measuring the same thing. So, we set up a tracker. This may be a member of the process improvement team or another staff member to track patients through the ED and record times. Before we start, we must ensure the tracker knows the data elements' definitions and agree on the clock device to use. For example, if the staff uses the clock on the computer monitor, the tracker will use the same clock. This way, we ensure we don’t get any clock synchronization errors. We track 10 to 20 patients and measure the variation in our tracker’s times and the staff/system’s times (most EMRs are a blend of staff-entered and system-recorded times).
A 5% to 10% error is generally acceptable for measurement systems. Of course, this depends on the degree of accuracy we need and the cost and consequences of the error. If the error is between 10% and 30%, we should look at the degree of accuracy required and the cost and consequences of the error, and if we are over 30%, we will need to stop and fix the measurement system. When we see significant variations between the tracker times and the EMR’s times, it is usually due to staff not understanding the data definitions, staff not understanding the importance of accurate measurements, and occasionally clock synchronization issues (e.g., the staff should be using the computer monitor, but some people are using their wrist watches or wall clocks. Many Lean Six Sigma and some statistical software packages have specific sections for Gage R&R/MSA.
Besides MSA, what tests do we use to help us understand problems with data validity? We generally use two tests:
Linear logic test – do element's times appear in the correct order? (e.g., the surgery example above)
Reasonable interval test – does the interval match benchmarks?
If we are preparing to build a simulation model or start another process improvement project, we will download a month of data (sometimes more rarely less) and apply these two tests. Just because we pass these tests does not mean that the measurement system is accurate or the data is valid; however, if we see significant errors, we may postpone the tracking phase and start with data definitions, agreement on clocks to be used, etc. and then let these changes permeate the measurements before doing the tracking to check that our measurement system is accurate.
Since we mentioned simulation earlier, let’s distinguish between simulation model validation and data validation. One of the key steps of simulation model building is model verification, validation, and accreditation (all discussed in Simulation Solutions: A Practical Guide to Improve Patient Flow & Facility Design in Healthcare Operations, available on Amazon). Data validation is an essential step in simulation model validation, but having valid data does not validate the simulation model by itself.
In summary:
1. Simply staying positive about our data does not validate it.
2. Just because information comes from an electronic system, such as an EMR, does not mean it is accurate. This is especially true with time-related data.
3. With time-related data, if we use MSA plus the linear logic test and the reasonable interval test, we can validate the data, discover errors, and fix the measurement system.