How to Evaluate a Protocol

By Sheila Wheeler, RN, MS

Winter, 2004

Whether you are evaluating electronic or paper based protocols the same basic steps should be followed.  Begin the process by obtaining paper copies of the following items: the table of contents, three representative protocols (abdominal pain; nausea and vomiting; or respiratory problems), and the documentation form. 

Ask the vendor: How Were They Developed And Tested?  Were the protocols developed by a single RN, MD, or a Nurse Taskforce?  When it comes to protocol development, four heads are better than one.  A taskforce working collaboratively as a group is superior to the single developer working alone or taskforce members working independently. 

Who is best suited to develop protocols, RNs or MDs?  Nurses are best suited and more economical as protocol developers, while physicians are best suited to consult, review, and approve them. 

What were the qualifications and experience of the taskforce members?  The best candidates for protocol developers are expert level nurses or nurse practitioners with expertise in pediatric and adolescent medicine, adult, geriatric, and OB/GYN medicine as well as experience in telephone triage.  Physicians who are familiar with and supportive of nurse telephone triage are the best consultants and reviewers. 

What is the optimum development time?  For a complete set of protocols (50 to 100) a development time of less than six months is suspiciously fast.  Usually, one to two years is more realistic. 

How long have the protocols been in use and where?  A history of several years or more in large, busy group practice or HMO is more meaningful than a few months in a small office. 

How Are They Designed To Be Used?  There are two distinct philosophical approaches to protocols and decision-making in telephone triage.  One philosophy is that the nurse's judgment is primary and that protocols are a decision support tool.  A competing philosophy is that the protocols do not require trained professionals and thus are the ultimate decision making tool.  Decision support has been defined as an expert system designed to remind experienced decision makers of alternative options or issues to consider.  Decision-making systems are defined as expert systems that allow non-experts to make decisions beyond their training and experience.  Ask whether the protocols require an autonomous professional or whether the phone interaction is mainly "protocol driven."

Assess the protocols: Table of Contents/Title System: One quick and easy way to assess a set of protocols is to simply review the table of contents which is essentially the author's "search engine" to find the right "tool" in the system.  Remain alert to titles that are confusing, inconsistent, or redundant.  For example, is lay language mixed with medical terminology?  Is there a consistent, intuitive approach to titling?  Are the protocols listed consistently by system, site, or symptoms, or are they a confusing mix of all three?  Poorly organized titling results in a search engine that is not user friendly and slows the interaction. 

What is the scope of the collection?  Having too many or too few protocols decreases the "user friendliness" of the product.  Too many protocols results in excessive page turning or screen changes, which can be confusing and time consuming.  Too few protocols leads to the "out of protocol experience," where the nurse is left without adequate protocol coverage and must "fly blind."  The optimum number of protocols is between 50 and 100, which covers about 1000 common diagnoses, including common emergencies.

Documentation Form: The protocols should have an accompanying documentation form.  It should contain the steps of the nursing process - assessment, impression, treatment, and self-evaluation.  In addition to demographic information, there should be ample space for the assessment and treatment sections.  Labeled spaces for standard information to be collected, trigger words for key questions, well-developed problem/client history section, and words delineating the nursing process will facilitate complete, accurate data collection.  Forms that have too much blank space or are too busy are not user-friendly. 

Test Driving and Training: Once you have narrowed the selection down to two or three designs, have staff test them out, using scenarios.  See how effective they are to use within the "real world" time of 10 minutes per call.  Finally, for best results, all protocols (paper-based included) require training in their correct and safe use.  Training affords the staff the opportunity to fully grasp the design, achieve buy-in, shorten the learning curve, and facilitate compliance.

Sheila Wheeler, RN, MS, is an expert in the field of telephone triage.  Ms. Wheeler is the author of Telephone Triage: Theory, Practice, and Protocol Development (Delmar Publishers: 1993) and Telephone Triage Protocols for Pediatric and Adult Populations (Aspen Publishers: 1998).  She is the founder and chairperson of the first national telephone triage conference, "Telephone Triage: Essentials for Expert Practice," held annually.  [See our industry calendar for the dates of the next conference.]