Over the past 35 years I have been involved with designing, implementing, and evaluating 30 or more legal delivery systems ranging from staff attorney to pro bono to hotline to brief services units to reduced fee panels. Each has its pluses and minuses and each presents its own unique challenges in terms of measuring "outcomes." We incubated many of these delivery systems at Legal Counsel for the Elderly and, with varying degrees of success and failure, we tried to help other programs replicate these systems across the country. In doing so, we learned something about human nature and change, the curiosities of "measurement," the culture of legal services programs, and how quickly emotions can overcome common sense in discussions about how best to serve our clients and how best to expend program resources.
The delivery system that has perhaps caused the most stir around the country is the hotline. And, after more than 25 years overseeing a hotline, I am acutely aware of both its strengths and limits as a delivery system when compared to other methods of providing services. Because it was so controversial, the hotline went through an inordinate amount of scrutiny, evaluation, outcome studies, client satisfaction reviews, and other attempts to measure whether hotlines were worthwhile (see, for example, the plethora of "Outcomes Studies" posted on the Legal Hotlines website under "Reports").
In recent years, there has been a heightened discussion of outcomes as they pertain to hotlines, full-service, and other methods of service delivery. I think that hotlines do provide a good example of some of the complexities of measuring outcomes and the need for common sense in terms of one’s approach.
To ensure that everyone is on the same page, let me explain what I think a legal hotline does and does not do. A hotline provides to callers relatively immediate access to an attorney who can elicit facts from the caller on the phone, analyze those facts, usually determine the issues, and make a preliminary assessment of what needs to be done. In maybe half of the calls, the caller simply needs some information and advice which the hotline attorney can provide.
In my view, providing thorough, accurate, and complete advice in terms the caller can understand is a very useful service. If you have ever had a legal problem in which a knowledgeable attorney provided you with a few minutes of reliable information or advice in a timely manner, you probably experienced a peace of mind or anxiety reduction which you can even recall to this day. I certainly remember the enormous relief on one client’s face when I informed him that there is no debtor’s prison and that he would not go to jail for failing to pay a credit card bill. That peace of mind is in my view worth something—and sometimes a great deal. Legal services programs provide that service, whether through hotlines or other mechanisms, on a daily basis—a valuable service, indeed, but not one easily measured. One might log "client obtained legal advice" in the outcome field, having recorded the substance of the conversation in the case notes. To the purist, this is not an "outcome" but rather it is an "output." Thus, there is arguably no measurable "outcome" for such a case. I should also note that a large percentage of legal services programs without hotlines log closed cases as "advice and counsel," so such cases, regardless of the delivery system, are difficult to measure in terms of outcomes (For example, according to the Legal Services Corporation Annual Report of 2010, 61.5% of all cases reported to that entity were closed for the reason of "counsel and advice.").
As one author correctly points out (David Godfrey, Legal Hotline Quarterly, Summer 2007), however, it may be not just what you say but how you say it. The example he cites is the "drafting of a power of attorney." Rather than phrasing an activity as an output such as "drafting a power of attorney document" one might record an outcome such as "getting bills paid," "promoting independence," or "averting guardianship proceedings." Maybe that is just semantics, or maybe it is translating our work into language a funder can appreciate.
Some of those hotline "cases" ultimately will have an outcome if the caller follows the advice of going to small claims court and suing the repairman for the $500 caller paid but from whom she did not receive the services. Here, again, the "outcome" is not realized until a judgment has been entered and the caller has collected. To figure out whether this happened may take not one telephone call but several. A hotline receiving thousands of calls has to make the resource judgment call as to whether it makes sense to make multiple calls on this "case" to discern the ultimate outcome or simply provide the advice and move on to the next case. Compelling arguments can be made for taking either step. Many a program, strapped for resources, would opt for the one telephone call and then move on to the next caller. Depending on the ability and level of motivation on the part of the caller this may or may not be a wise choice.
Some hotlines, like ours, have made the decision to separate out "brief services" so that the hotline attorneys do not review documents, call third parties, or draft letters on callers’ behalf. This allows the hotline to be truly "hot" and not engage in the time-consuming activity of what is labeled in the trade as "brief services." The hotline attorneys mainly screen for conflicts, develop the facts, identify the issues, and then provide information and advice, or schedule the caller for an appointment with the "brief services unit" or other staff attorney. There is value to the program in conducting a conflicts check, developing the facts, identifying issues, and scheduling an appointment with staff. These steps are, however, in the jargon of program measurement, outputs and not outcomes. For those programs where the hotline also performs "brief services" there is a greater likelihood of achieving a measurable outcome, especially in the context of completion of a form or phone call to a third party to resolve a simple dispute.
An additional function that the hotline performs is that of screening out "cases" which are lacking in legal merit or are time-barred or both. An informed hotline attorney who tells a caller diplomatically, fully, accurately, but firmly, that the client does not "have a case" or that there is no legal remedy performs a valuable service. Too often, program staff will take the easy way out and refer such individuals to some other program or lawyer though it is apparent that legal representation will not help. A hotline attorney who performs his or her job well in this regard just might prevent this caller from calling every attorney within a 50 mile radius to explain the problem. There is programmatic and societal value in this function, a sort of "tough love" for those few individuals who clog intake systems with frivolous matters and endless rants. But again, there is no measurable "outcome" that we can easily assign to such a situation.
One methodology that we and some other hotlines use is volunteer follow-up to check on callers to ensure that they acted on the information and/or advice that the hotline gave. This is a very good addition to the hotline and our experience has been that we have been able to greatly increase the chances that the caller will follow through. Typically, that follow-through might involve ensuring that an appointment was kept or some other referral was pursued. Thus, even if the caller has done so, or says that they will after getting the call from the volunteer, it again is really only another "output" measure, not an "outcome." It could take a number of calls over the course of a number of months if the program wants to determine if a client got the benefit, judgment, or other result that was being sought. So, while adding a volunteer follow-up component is a good idea, it does not guarantee that the ultimate "outcome" will be captured by the program. A caller who has been instructed to go to court and file documents pro se will often also rely on friends, neighbors, relatives, court personnel, kiosks, and other sources of support. In light of that additional help, is it appropriate to attribute the outcome only to the hotline that originally advised the client to go to court?
Of course there is always the client satisfaction survey that might be used to help answer the hotline outcome conundrum. A survey might ask whether the question or problem was resolved, how much was resolved, and what difference the attorney’s involvement has made (choices might be: "a lot," "some," "a little," or "none"). There are a number of complexities here as well. First, the return rate of client satisfaction surveys typically is not high and therefore the accuracy of the data comes into question. Second, client satisfaction surveys are subjective and thus not a definitive source of information on the true outcome of a case. Third, for those situations in which the hotline attorney correctly tells the caller that there is no case to be made, a negative client satisfaction survey could result despite the attorney’s good advice. Thus, the results of client satisfaction surveys have to be taken with more than a grain of salt and should not be considered the end-all and be-all of outcome measures for the function. Nevertheless, survey results often do reveal many positive outcomes, give feedback about how well a system is working, and provide testimonials regarding "success stories" (see below).
Finally, there is the issue of the client and the nature of the problem. One well-respected study of hotline outcomes (Dr. Jessica Pearson and Lanae Davis, The Hotlines Outcomes Assessment Study, Final Report—Phase III, 2002) found, not surprisingly, that better educated (eighth grade or higher), English-speaking callers were more likely to obtain more favorable outcomes as a result of the hotline advice. That study also found that certain types of cases (housing and consumer) were more inclined to achieving favorable outcomes via a hotline service than were other types of legal matters. These findings raise two issues: (1) whether or not to follow up on a hotline call may require an assessment by the hotline attorney on both the caller’s ability to grasp the information and on the likelihood of the client following through without a follow-up call from a volunteer; and (2) while the instance of a granting agency on achieving "outcomes" may be valuable in prompting the hotline to find cases where outcomes are easily obtainable, it seems inappropriate to favor certain types of cases over others simply because of this strict definition of success. We should not, in the words of Thoreau, become "the tools of our tools" by letting the ease of measurement dictate what cases we work on.
Conclusion
Conclusion
Conclusion
We should infuse a good deal of common sense into the discussion of outcome measures. A hotline is, in many instances, simply one stage in a continuum of services that a client needs. It can provide valuable service in determining the merits of a case. It can provide valuable service in identifying issues, adverse parties, and next steps. This is useful to whoever ultimately handles the case. It can provide a good deal of peace of mind for callers anxious about a whole gamut of legal issues. It can provide guidance about simple matters, which, especially with follow-up, might well result in favorable outcomes. To conclude that a hotline cannot always produce tangible outcomes for all or most of its callers, however, does not mean that the hotline is not worthwhile. It may mean it is simply a short, meaningful service that adds value but often is not the end-point of the service delivery. We cannot fully evaluate a hotline by strict outcome measures any more effectively than we can judge a restaurant by its maitre d’ or a college by its freshman week.
What can we conclude about outcome measures and legal hotlines?
Here are my recommendations:
- Where possible, hotline attorneys should record outcomes they can justify based on their calls with clients (there are many templates available).
- Consider carefully how you express the “outcome” (such as with the power of attorney example described above).
- Where possible and where feasible, hotlines should utilize volunteers to follow up on hotline callers to try to ensure that the callers took the advice and acted on it. Again, where possible, the volunteer should record any “outcomes” that are achieved.
- Utilize a short, easy-to-complete client satisfaction survey, asking about how much of the caller’s question or problem was resolved. This should be mailed out perhaps a month or two after the call (close enough in time that the caller remembers, but long enough for there to be a reasonable chance for the problem to be resolved).
- Do not re-order a program’s priorities based on cases where outcomes are easily measurable—this is the hotline equivalent of the tail wagging the dog. That said, you may find that tweaking hotline procedures on certain cases (e.g., sending written materials) might enhance hotline outcomes.
- Appreciate the value added in many cases where the hotline simply develops cases for others to work on because the case cannot be resolved through advice and counsel.
- Appreciate that a hotline can effectively weed out frivolous claims; ultimately, this is a benefit to both the client and program resources.
- Tell the story and tell it well. Capturing and compiling client success stories from the hotline or client surveys is certainly one significant part of the hotline outcome puzzle.
- Implement and review your hotline system in accordance with the standards adopted by the BA (see Standards For the Operation of a Telephone Hotline Providing Legal Advice and Information, Adopted by the American Bar Association, August 2001). Setting standards, implementing them, and periodically reviewing your operation may well be the best way to ensure that your system is producing quality outcomes for clients served by a hotline or any other legal delivery system.