The views expressed herein have not been approved by the House of Delegates or the Board of Governors of the American Bar Association, and accordingly, should not be construed as representing the policy of the American Bar Association.
Parent representation providers can benefit from data collection and analysis. What kinds of data should be collected, and how might you use them? What benefits and pitfalls does data collection have?
At the New York State Office of Indigent Legal Services, we are mandated to “monitor, study, and make efforts to improve” parent representation in each of the state’s family courts, so we have been thinking about meaningful data collection carefully.1 In this article we take a broad view of data collection and discuss why you should collect data, how to do it, and what to be wary of along the way.
Federal Government Context
New federal rules create opportunities for parent representation providers to have input into state data collection and analysis.2 In 2012, the Administration for Children and Families (ACF) said it would only continue to offer Court Improvement Program (CIP) funding to states that implemented something called “continuous quality improvement” (CQI). CQI involves tracking and regularly reporting data "to identify, inform, monitor and improve progress toward outcomes in an ongoing fashion.”3 State courts must identify intended outcomes and measureable objectives for child welfare cases, describe what data will be collected, how it will be obtained, and who will measure and present it.4 The data are then used to identify needs and develop interventions and activities to meet the stated outcomes and monitor how well the interventions and activities are working.5
A key component of CQI is ongoing collection and analysis of data about the quality of legal representation for parents, children, and child welfare agencies. State courts receiving Court Improvement Program funds must collect data “to measure and track the progress of interventions and activities, including . . . quality indicators of hearings and legal representation.”6 A feedback process involving relevant stakeholders must be in place to use data “to identify, inform, and implement midcourse adjustments and modifications to improve CIP interventions and activities.”7 Beyond that, the details are left to states. This means that states should be talking about measuring parent representation quality, which may open up opportunities that did not previously exist for providers to contribute information about the importance of their work on child welfare processes and outcomes.
Why Data?
There can be advantages to tracking data on your work beyond simple compliance with federal mandates. At their best, data offer credibility. Data can help you convince a funder or budgeting agency that you are effective. Equally, data might be useful to show how overwhelmed you are. Data can help you test whether changes you make have the desired effect, and can show you areas that you could examine in your practice to serve clients better. Data can also, of course, be used by your enemies to allege you are doing a poor job. Either way, the one thing you can be sure of is that the more sound data used to support an argument the more credible it will appear.
The data that work best are those that represent pieces of sound, objective information—defined and collected by you—about what you do. They should provide evidence that will help you to explain what you do and why it’s important. While the data you gather might not always tell you what you want to hear, or everything you want to know, if you collect it carefully you should at least be sure it won’t lie to you.
What Data?
What data should you collect? The more you think about the kinds of data you’d like to have, the longer the list will get. The truth is some data are more useful than others. First, ACF provides a long list to get you started (see Figure 1).8 Second, research on performance measurement can help guide your thinking.9
Think about the kinds of questions your data will have to answer—questions like:
- What resources do you need?
- What do you do for your clients?
- Are your clients better off after you help them?10
These are the kinds of questions parent representatives face often from funders, the media, and others who want to know more about the work. Addressing them gives you a chance to think about this as a marketing exercise. What information would you like to have to answer these questions that helps persuade people your program is worthwhile?
Try using those three questions as a brainstorming exercise:
What resources do you need? People need to understand the situation you face when you try and do your job, so think about data that reflect that.
- What are your caseloads?
- What resources or funding do you have?
- What kind of attorney and/or support staff do you have?
- These kinds of data can show the constraints that define what you are able to do for each client.
What do you do for your clients? People want to know what services you provide, how often, and to how many people—and they want to know whether you’re doing a good job. Think about counting how often you contact clients or meet with them, by phone and in person. Alternatively, or in addition, you could count the other work you do—time spent in court, preparing for court, or conferencing with other key players in the case. These kinds of data tell people how you spend your time, and show the types and quality of services you offer
Are your clients better off? You might not always be able to help every client, but you certainly want to be tracking what happens to them in order to respond to questions like this. Think about recording simple things that you are in a position to observe:
- Is your client separated from his or her children?
- Are parental rights terminated?
- If families are reunified, can you tell how many and how quickly?
When it comes to showing people what is going on in your program, remember that a single well-chosen statistic can save you a thousand words. More importantly, some statistics speak louder than others. Being able to show that “On average our clients have their families reunified within five months” is profoundly important and can be used to answer many questions. A statement like “We conflicted out of 15% of our cases last year” may be important for you to know, but it doesn’t speak to what people want to know about your program in the same way. Think about the kinds of questions you are often asked, starting with the three mentioned above. Consider what you would most like to track and report on a regular basis, and focus your data collection activities accordingly.
How Do You Get Data?
Actually recording the data you need is technical and there are multiple solutions. At its most basic, you need some system to record systematically information about every case you work on, and it must be computerized. Your case files are valuable information repositories, but that information is trapped and useless for systematic analysis until it is in electronic form. You could invest in a case management system produced by a commercial company, but you could also start with something as simple as a spreadsheet.
Next, think about how the data get into the system. A lot of case management products today have multiple methods for data entry that extend far beyond sitting at your computer punching in numbers. Some are accessible via mobile devices, allowing you to record data as you talk to your client, uploading it to the system immediately, saving you the inconvenience of copying it out again later.11 Some integrate the tasks you need to perform (such as note taking on a computer during a client meeting) with the data collection process (creating a record in the system that you met the client, and took notes). Recording data is rarely painless, but it can be simplified through clever labor-saving devices that complement the record-keeping you already do in your case files.
Last, start monitoring and reporting what your data say. Some case management systems will produce dashboard type readouts showing basic information you select, such as the time your cases are taking to dispose, or how many are disposing favorably to your clients. Others are not so sophisticated, leaving you to generate charts and tables on your own. Your decisions earlier in the process about what you want to track will constrain what you can report on, but with wise choices you should be looking at a range of indicators that give you a clear idea of how things are going.
Where Are We With Data?
Recently, we conducted a national survey of parent representation providers about what data they collect. We received 51 responses from providers ranging from single attorneys to offices with staffs of over 200. We followed up with four in-depth interviews of providers that were collecting a lot of data.12 We wanted to know where the parent representation bar was with data collection, and the approach of successful programs.
What we found was that providers tend to have data that are much more suited to answering questions about what resources they need than anything else (see Figure 2). 94% of respondents said they collected data on their caseloads, for example. When it came to tracking “What do you do for your clients?” the numbers recording client contacts or work done investigating or preparing a case were only in the low sixties. When it came to data that might show “Are your clients better off?” the picture varied: 67% recorded whether a client’s children had been removed, while many fewer were collecting data on longer-term issues such as re-entry to the system (28%) or client satisfaction (8%).
We also learned, however, that most managers of parent representation organizations are seriously interested in improving their data collection. The main obstacles to improving data collection, of course, were the time and resource constraints parent representation providers already face, and also the fear that data could be used for good as well as ill against the provider.
In our follow-up interviews with providers that have successfully improved data collection we learned several strategies for expanding data collection. In the words of one interviewee, the key is to “keep it simple, non-redundant, and doable.” More specifically, we learned:
- Focus on a small number of highly important measures rather than a large number of peripheral ones. Consult staff on what these should be. This reduces the inconvenience to staff, increases their buy-in, and improves the quality of data you collect.
- Try to find low-hanging fruit. Figuring out if abuse allegations ever recur in a family where you’re representing a parent may be hard. But recording whether a child is sent to foster care is less so, especially if the attorney is there to see it happen.
- Do you have any incentives you can offer? Some programs have gotten data collection off the ground by offering cash to lawyers to fill out data sheets, or finding ways to offer or supplement any additional support staff required.
- What leverage do you have? It’s rare that forcing people to enter data has good results in terms of data quality. If compliance is important, some people may need extra encouragement. Assigned counsel systems frequently make mandatory certain data fields as a condition for payment; government contracts often require the same. When people don’t submit needed data, have in place a series of escalating responses beginning with a frank conversation about how important this information is. Punitive measures such as cutting payments should be an absolute last resort as use of them ordinarily represents a failure for all concerned.
- Is someone else already collecting the data? Most states have administrative data systems in their courts or child protective services offices that already record basic information about dates, times, and cases. Think about whether you have the IT capability to interface with that system so you don’t have to enter all the information yourself.
- Share the results. Most importantly, once you’ve collected and analyzed your data, be sure to show the value of the data to those who enter it. This can really help people to understand how valuable their work entering data really is. One interviewee put it this way: “The data said, ‘Look what an amazing job you’ve done!’ It was a big morale booster and it allowed us to work with staff to tighten things on the margins. They aren’t just plugging along without understanding the outcomes any more. They aren’t working in a vacuum.”
Final Thoughts
Let’s be clear about what data offer, and what they don’t. At their best, data can help you grow, reform practices, and show others what a good job you are doing. But they take time to gather, and you might find they don’t answer your every need. There will never be any substitute for you making critical decisions case by case on how to represent your client. In these cases, data will seem irrelevant, or even misleading. And your enemies might try to use your data against you –just like they try to use any other fact that seems solid enough to hang an implication upon.
Ultimately, knowledge and data empower you by giving you information you can control, on issues that you care about, and which you can use however you wish. We need to think about ways to show our value to others whether we like it or not if we intend to keep competing for funding. Apart from that, we should think about how data can help us learn more about what we do, and start to collect it.
Andrew Davies is director of research at the New York State Office of Indigent Legal Services.
Angela Olivia Burton is director of quality enhancement for parent representation at the New York State Office of Indigent Legal Services.
Endnotes
1. See New York Executive Law §832.
2. U.S. Department of Health and Human Services, Administration for Children and Families, Program Instruction, ACYF-CB-PI-12-02 (01/11/2012), 6-8.
3. Ibid., 6, note 3. ACF subsequently issued an Information Memorandum on August 27, 2012 providing guidance to State child welfare agencies about how to establish and maintain CQI systems.
4. Ibid.,7.
5. Ibid., 7-8.
6. Ibid., 7.
7. Ibid., 7 & 8.
8. Ibid., Attachment B, “Indicators of Quality Legal Representation.”
9. We turned in particular to Mark Friedman’s book Trying Hard is Not Good Enough, 2009, for inspiration. We are thankful to Trine Bech for this recommendation.
10. These questions are adapted directly from Friedman, Mark, 2009.
11. See, e.g., ‘Defender Data mobile,’ (accessed 1/12/15).
12. For more details, see https://www.ils.ny.gov/files/Parent Attorney Data Utilization Project - Why Collect data.docx. You can find the survey at https://www.surveymonkey.com/s/ParentCounselDataUtilizationSurvey. It will take about 10 minutes.
Figure 1: Indicators of Quality Legal Representation
from the Administration for Children and Families
Several indicators provide evidence that parties are receiving quality, effective representation that meets the requirements of law in accordance with professional standards of ethics. Examples of indicators include:
- timing of appointment of counsel;
- presence of the attorney in court;
- active attorney participation in hearings (e.g., advocating for necessary services for his or her client, including language assistance services [interpreters] for parties with limited English proficiency, formal and informal discovery requests made);
- timeliness of pleadings;
- the frequency and nature of out-of-court meetings the attorney has with client prior to and between hearings;
- attorney participation in out-of-court meetings (e.g., multidisciplinary case reviews or Family Group Decision Making);
- the frequency and nature of the attorney’s communication with collateral contacts (e.g., foster parents, service providers, case workers, school system, etc.);
- attorney caseload information (e.g., whether the size of the caseload is manageable, current child welfare caseload and diversity of those cases, i.e., does the attorney represent children, parents or both; percentage of child welfare cases that comprise the attorney’s practice);
- attorney participation in all placement decisions;
- attorney monitoring of the implementation of court orders and related follow-up and advocacy;
- any specialized child welfare training, accreditation or access to mentoring the attorney may have;
- nature of the supervision/management structure for the attorney’s practice;
- continuity of counsel throughout life of the case;
- access to other multidisciplinary professionals as partners, team members, or employees such as social workers, investigators, Court Appointed Special Advocates (CASAs) etc.