Standard 4.11 on Organization Evaluation

Standard 4.10 | Table of Contents | Standard 5.1

Standard

A legal aid organization should regularly evaluate the efficiency and effectiveness of its operations and infrastructure in supporting high-quality advocacy, including identifying new or emerging needs. Data, including client feedback, should be regularly compiled and analyzed to support this evaluation.

Commentary

General Considerations 

A legal aid organization should regularly review its operations and infrastructure to determine if it is functioning efficiently and effectively, is producing high-quality legal work, and is accomplishing its objectives on behalf of the clients it serves. The overall goal of these assessments should be to support a forward-looking and judicious evolution that addresses the organization's weaknesses and reinforces its strengths, making the most of available resources while remaining good fiscal stewards. 

The organization has a primary responsibility for the ongoing evaluation of itself and should conduct its own assessments consistent with this Standard and commentary. The organization's funding sources also have an interest in assessing whether their funds are being used effectively and in compliance with the terms of the grant or contract. Evaluations conducted by a funder can be an important source of information for an organization, and the organization should cooperate in such evaluations and take advantage of the insights derived from them

Evaluation Purposes

There are several potential purposes evaluations can serve: 

• To improve the organization's operations and/or infrastructure. A fundamental purpose of evaluation is to examine how effectively the organization, or a specific project or initiative, is functioning in order to make informed decisions about possible adjustments. Assessments should examine whether the organization and its projects are operating according to developed plans, whether they are accomplishing the objectives set out for them, whether staff and others involved have the requisite skills and substantive knowledge for them to succeed, and whether those plans can or should be modified to better accomplish the goal. Data reports and analysis are key to these evaluations. 
• To determine if its delivery approaches are succeeding. To be effective, an organization has a responsibility to learn about new delivery techniques and to experiment with new approaches to helping the communities it serves. In determining its approach to serving its constituents, the organization will make a number of choices regarding the appropriate balance between limited and full representation, assistance to self-represented litigants, community legal education, and other preventative legal services, as well as various models for delivering each.The legal aid organization should regularly assess whether its efforts are accomplishing useful and lasting outcomes for those it assists, and whether it is successfully responding to the most compelling needs of the communities it serves. Again, data reports and analysis, and both the organization's data and available public datasets about the client population should be a part of the analysis.
• To test the success of innovative delivery techniques. An effective organization will engage in innovative approaches to serving its clients, utilizing a number of techniques, including new technologies. The organization should evaluate its innovative efforts to determine if they are accomplishing the intended outcomes and are cost-effective. Such an evaluation is particularly valuable when the organization is considering whether an innovative approach should be made permanent or expanded. The organization should undertake these approaches knowing that some will not accomplish what was hoped for, taking lessons from each technique to determine next steps. See Standard 3.10 for more information on evaluation of technology tools. 
• To inform planning and budgeting. Evaluations can also be important to inform planning and budgeting, and to guide the allocation of the organization's resources. Many strategies for serving clients or for using technology involve significant choices for an organization in terms of deployment of staff and expenditure of funds. Evaluations are an important source of information to the organization about the effectiveness of its allocation of these resources and to guide future choices. As much as possible, organizations should include evaluation budgets in their grant requests, which should include work with experts in evaluation in the design and analysis of data collection for new and pivotal projects. 
• To inform training plans. A number of factors help shape an organization's training plans, including conclusions of an evaluation. Evaluations may identify needs for training in substantive knowledge and practice skills, as well as other areas an organization might not otherwise identify, such as management skills, cultural humility, the ability to effectively deal with diversity, trauma-informed care, and the utilization of technology.
• To demonstrate accountability. External and internal self-evaluations are often the key to demonstrating that a project or initiative has met its goals and accomplished its objectives. This may be required by a funding source or initiated by the organization to support requests for additional funding.
• To obtain and increase funding. Innovative initiatives that attract funding are often based on claims regarding the results that they will achieve. Measuring and demonstrating the actual benefits and results are often key to obtaining future grants from the same or other sources. 
• To help build a constituency. Evaluations can sometimes be instrumental in demonstrating the benefit of a project or initiative to community organizations or key individuals in order to gain their support or collaboration.

Approaches to Self-Evaluation

A variety of techniques may be employed to assess an organization's operations. These range from internal reporting systems that regularly supply management and the governing body with information about the organization's activities, to internally conducted formal evaluations of projects and initiatives, to formal review by outside evaluators. An evaluation may involve a full assessment of every aspect of the organization's operations or may focus on specific projects or components. As much as possible and consistent with client confidentiality, the organization should use client feedback from tools such as online chats, triage portal, text messages, online forms, intake modules, and the case management system to make decisions based on empirical evidence, including those around topics such as hiring, fundraising, office location, outreach, marketing, and similar resource allocation decisions. 

The frequency and scope of an evaluation is a function, in part, of the purpose it is designed to serve. Full evaluations of the organization are costly and need to occur only every several years. Some projects or special initiatives, on the other hand, may call for ongoing monitoring by the organization. Many funding sources periodically review the operation of their recipients. Reports from such evaluations serve as a useful source of information to an organization regarding its effectiveness. Data, including client feedback, should be regularly compiled and analyzed as a part of this process.

Evaluation of internal operations. An organization should regularly conduct internal evaluations as a management tool. This is particularly important with regard to the delivery approaches that the organization adopts to respond to the needs of the communities that it serves. An organization should regularly assess whether its staff, particularly its practitioners, have the necessary skills and substantive knowledge for their roles. It should assess the effectiveness of its delivery approaches involving full representation, limited-scope representation, assistance to self-represented litigants, and community legal education.

Other facets of its operation should also be assessed by the organization, including intake, its effort to integrate the resources of the legal profession and involve members of the bar, its cultural humility, its capacity to serve persons with limited English proficiency, its relations with clients, its accessibility to potential clients, its internal systems and procedures, its board's activities, and its methods for quality assurance. Refer to the various standards relating to all of these aspects of operation for further guidance on factors against which to perform an evaluation. Such evaluation should also center on the user experience, including the client and relevant staff members engaged in those aspects of the operations.

Evaluation of the organization's responsiveness to the legal needs of the communities it serves. Strategies that rely on clients taking action to assist themselves - as is the case with limited-scope representation and assistance to self-represented litigants - call for the organization to assess the extent to which users are able to take advantage of the assistance and its benefits to them. Follow-up with the persons served is an effective way to determine if they understood the information or advice provided, if they acted on it, if the action improved their situation, and if it led to a just result. This can be done with standard tools like secure SMS texting, online chats, chatbots that gather information, and data past the last point of contact with the individual. Similarly, organizations should periodically assess whether their legal education efforts succeeded in conveying the information intended and whether persons who received it were able to act favorably based on the information offered. 

An organization should also periodically assess the effectiveness of its legal strategies that utilize various forms of full representation in a way that is consistent with the jurisdiction's ethics rules. Such assessments should consider both whether the organization is achieving clients' objectives in a significant number of cases and whether it is accomplishing meaningful outcomes for its clients.

The organization should also examine the long-term outcomes of the organization’s legal work for the client community overall, measured against the compelling legal needs identified in the organization's planning process. It should also examine unexpected long-term results and consequences that may have been achieved.

Types of evaluation. There are different types of evaluations that can be beneficial to an organization and that should be utilized where appropriate. Two common examples are: 

• Process evaluations, whichassess the extent to which an organization is operating as it was intended. Where an outcome evaluation asks whether a project or component of an organization was successful, a process evaluation asks, "Why or why not was the project successful"? Evaluators are embedded in a project or component of an organization in a process evaluation that allows for real-time analysis and allows the evaluation to influence the project. In this way, process evaluators are not fully external reviewers, but invested participants, and the process evaluation is done in service of the project and the project's staff.
• Outcome evaluations, in contrast, are retrospective. Where a process evaluation provides real-time analysis, an outcome evaluation seeks to ask, after project conclusion, if a project or component of an organization was successful and whether it should be replicated. An outcome evaluation is done in service of the funder and audience the project intends to help - in the case of legal aid organization evaluations, this is typically the client. 

Evaluation techniques. There are a number of evaluation techniques that an organization can utilize to assess its operation and the effectiveness of its legal work:

• Client satisfaction assessmentscan be useful to measure an organization's treatment of clients. They can provide useful information regarding whether clients are being treated with dignity and respect, are being kept informed and properly consulted regarding the conduct of the representation and are satisfied with the outcome in their cases. Client satisfaction surveys alone do not offer reliable data about the quality or effectiveness of representation and should be paired with more robust outcome evaluations in order to gain a more complete picture of the organization's effectiveness.
• Evaluation of online data tools, like analysis of data gathered by Google Analytics or alternative analytics tools like Matomo, and analysis of data created and generated by SAS platforms, can be executed.
• Data-based evaluation can be performed using the tools and resources the Legal Services Corporation routinely updates and publishes so that legal aid organizations can do more robust data analysis .
• Regular internal reporting can provide data about an organization's operations and productivity, such as budget reports, case counts, outcome tracking, major case activity reports or impact dockets, and data related to requests for service.
• A desk audit using an internally administered checklist to confirm that the organization complies with appropriate legal and contractual requirements can be performed.
• Review of the work of practitioners and other staff, consistent with professional conduct rules, through both supervisory and internal peer reviews to ensure that it is of high quality and meets the organization's standards can be done. 
• A peer review can be performed in which outside persons who are knowledgeable and experienced in legal aid review the organization's operations and the quality and effectiveness of its legal work, so long as such review does not violate the duty of confidentiality. 
• Periodic follow-up interviews of a small sample of clients at intervals following the close of the case can be executed.
• Surveys and interviews of personnel of courts and agencies that work with the organization and with the client population can be done.
• Focus groups of low-income persons, clients, agency personnel, and other stakeholders can be held.
• There can be a review of court files to determine case outcomes.
• Evaluation in the form of robust usability testing involving members of the client population can be performed.
• A review and use of end-user feedback through online tools and surveys to improve services, instructions, and capacity to use those services can be done.