January 01, 2016

Enhancing the Quality of Hearings in New York State: Using Data to Drive Improvement

By Alicia Summers, Christine Sabino Kiesel

The views expressed herein have not been approved by the House of Delegates or the Board of Governors of the American Bar Association, and accordingly, should not be construed as representing the policy of the American Bar Association.

The law requires frequent judicial review to achieve timely permanency for children placed outside of their homes. How do you ensure quality does not get pushed aside for timeliness? Go to the frontlines – observe court hearings, talk to and survey jurists, child welfare clients, attorneys, foster parents and others. Gather data to gauge what is happening in court, what’s working and what’s not. 

This is the approach New York State is taking, with good results. This article shares that approach and offers ideas for other states looking to keep quality at the forefront in child welfare proceedings.

Background

In December 2005, New York enacted ambitious legislation to strengthen the judicial oversight of child welfare cases required by the Adoption and Safe Families Act (ASFA). That legislation created a new Article 10-A of the Family Court Act requiring more frequent permanency hearings for all children who are the subject of petitions alleging abuse or neglect, or for whom an application is sought seeking voluntary placement into foster care. The new law replaced a prior law which mirrored ASFA’s timeframes. 

New York State Family Court Act (FCA) Article 10-A provides that the initial permanency hearing for a child who is not freed for adoption begin generally no later than “…six months from the date which is sixty days after the child was removed from his or her home;”1 and that later permanency hearings “be scheduled for a date certain which shall be no later than six months after the completion of the previous permanency hearing...”2  This mandatory expedited timeframe for holding permanency hearings allows jurists and professionals to attend more often to the critical conversations that are necessary to achieve timely, safe, and appropriate permanency for children. 

A review of New York State’s data on the timeliness of permanency hearings shows the legislative effort to hold more frequent permanency hearings was successful. The data reveal that for the 2012 entry cohort of children, 83% of initial permanency hearings were completed within the statutory timeframes, and 93% of subsequent permanency hearings were completed timely.3

Despite the legislation’s success, requiring permanency hearings in a shorter time frame added pressure to an overburdened court system and jurists’ calendars. It also lacked added resources to support implementation. 

In addition to increasing the number and frequency of permanency hearings Article 10-A aims “to provide children placed out of their homes timely and effective judicial review that promotes permanency, safety and well-being in their lives.”4  To fulfill that purpose, jurists were required to hear these cases more frequently and address children’s safety, permanency, and well-being. The challenge became carving out enough time to provide children with this “effective judicial review” in the face of existing stresses on the system. 

Using Data to Improve Hearing Quality

New York’s Child Welfare Court Improvement Project (CIP), with support from Chapin Hall in 2011, first rolled out the Child Welfare Court Data Metrics. Enhanced funding from the Children’s Bureau allowed CIPs to strengthen dependency court data collection and dissemination. National standards encouraged assessing court performance in child welfare cases. New York already had a robust data collection system for child welfare cases established by the Unified Court System Division of Technology. The Children’s Bureau funds could therefore be used to turn the data into useable information.

Time to Permanency Data

The data revealed that despite the comprehensive statutory scheme promoting more frequent judicial review that promotes safety, permanency and well-being, New York State’s time to permanency for children was far below the federal benchmarks set in Round 2 of the Federal Child and Family Services Review. Based on the Child Welfare Court Data Metrics observed at the end of 2012, for children who entered out-of-home care in 2011, 38% of them achieved permanency in 12 months. For children who entered out-of-home care in 2010, 2% of them achieved permanency by adoption within 24 months.

Since the data on timeliness and permanency were opposed, New York’s CIP wanted to explore this dichotomy. Given the strong statutory goal to move children toward permanency through permanency hearings, CIP in 2012 began to examine how to enhance the quality of permanency hearings throughout the state. 

Quality Improvement Model 

A daunting task at first, the goal to improve permanency hearing quality was attainable by harnessing existing resources within the state, requesting help from our federal technical assistance provider, and dividing the process into distinct tasks with defined purposes and measurable outcomes. The CIP was guided by a Continuous Quality Improvement model that supports work at the state and local levels. The model’s key elements include: 

  • collecting data and information; 
  • analyzing and interpreting the data to determine progress; 
  • applying lessons learned; and 
  • developing measurable outcomes and activities to support those outcomes. 

Logic Model

CIP developed a logic model that will continue to guide the process in the future. An overarching strategy is working with counties to enhance their permanency hearings as a vehicle to improve outcomes for children and families statewide. The key steps contained within the logic model are:

  • identify key indicators of a quality permanency hearing using existing national and state publications as well as local focus groups;
  • develop court observation and review tools;
  • provide for a data collection repository with reporting capability;
  • seek out pilot counties and implement court observation and case file reviews;
  • provide feedback on strengths and areas needing improvement and support the county in developing an action plan for improvement;
  • revise tools based on lessons learned in pilot counties;
  • develop a toolkit for a county “self-review.”

Data-Driven Approach

After exploring the quantitative data from the New York State Child Welfare Data Metrics, New York CIP used a two-stage data-driven approach to assess hearing quality. The CIP began a qualitative data “information gathering stage” with a plan to improve permanency outcomes for New York’s children and youth. Focus groups solicited feedback about permanency hearings from child welfare practitioners. An online survey also gathered input from judges and judicial officers around the state. 

Focus groups. Focus groups were held in four jurisdictions to gain a variety of perspectives. It was assumed the challenges and successes of permanency hearings would differ in rural versus urban locations. The locations included New York City, Westchester County, Oneida County, and Monroe County. CIP conducted role-specific focus groups with parent attorneys, children’s attorneys, agency attorneys, and caseworkers in each of the four jurisdictions. Two more focus groups solicited the views of foster youth and foster parents. A total of 18 focus groups were conducted. Focus group questions asked about general perceptions and role-specific values concerning permanency hearings and what those hearings should (or should not) look like in practice. Sample questions included: 

  • In a perfect world, what is the purpose of the permanency hearing? 
  • Does the presence of youth and children add value to permanency hearings?
  • What are some best practices that occur in permanency hearings? 
  • What are the most important elements to address during the permanency hearing?
  • Do permanency hearings assist in achieving more timely permanency for children? 

Online survey. The online survey supplemented the focus groups and was conceived as a more practical approach to gaining the views and experiences of judges and judicial officers. The focus group questions were copied into an online survey format and sent to judicial officers around the state, with 57 judicial officers responding. 

Key Findings—Focus Groups and Online Survey

Several key findings emerged from the focus groups and online survey: 

  • Responses were diverse, yet similar. The views shared by the professionals and parties involved in the focus groups and online surveys were diverse, yet the commonalities outweighed the differences. In fact, most themes noted were common across rural and urban jurisdictions alike. The results gave the CIP a solid framework of stakeholders’ perceptions of permanency hearings across New York State.

  • Permanency hearings lacked impact. Many felt permanency hearings were too brief and pro forma and were not conducted in a thorough and meaningful way. 

  • Little consensus on whether children and youth should participate in court. Despite efforts by the CIP and others to involve children and youth in court, there was much uncertainty about whether children and youth should attend, whether they add value to proceedings, and if they attend, how they should participate. 

  • Youth voices not heard. Youth did not feel their voices were heard in the legal process, either in court or through representation by an attorney.

  • Focus on well-being needs to be improved. Well-being is broadly defined as any aspect related to the welfare of the child, including educational, physical, behavioral or emotional aspects. Many stakeholders felt there was not enough focus in permanency hearings on child and youth well-being. A focus on the parents’ progress often took precedence over the child’s well-being. The literature clearly shows the negative impact of abuse and neglect, and even foster care, on youth. It is therefore important for the courts to ensure efforts are underway to promote resilience and encourage positive well-being outcomes for youth whenever possible. 

Data Analysis & Next Steps

When data collection was complete, the National Resource Center on Legal and Judicial Issues (NRCLJI) analyzed the data and consulted on next steps. The qualitative data from the focus groups and survey were used to: 

  • design court observation and case file review tools; and
  • compile an executive summary with themes drawn from the data to share with local jurisdictions to support in-depth assessments of local permanency hearing practice. 

Court Observation and Case File Review Tools

The court observation and case file review tools were designed to capture quantitative data about what is occurring in permanency hearings, how cases are progressing through the system, and case outcomes. 

Court observation tool. The court observation tool assesses what is occurring in permanency hearings through real-time observations. It focuses on the court process, including the actions of the court and the interplay among the parties, attorneys, and the court. This gives a holistic picture of this snapshot in time. Observations include identifying:

  • which parties were present; 
  • which permanency hearing was observed (first, second, etc.); 
  • whether the judge or judicial officer engaged the parties before him or her, including foster parents, relatives, children and youth; 
  • what topics were discussed; 
  • what findings were made; and 
  • whether reports were timely received. 

Additionally the CIP developed a measure to identify whether the goal approved at the permanency hearing was appropriate for that child or youth.5 For each jurisdiction wanting this in-depth assessment, two members of the CIP staff observe approximately 20 permanency hearings in a three-day period. 

Case file review tool. The case file review tool focuses on all permanency hearings held during a child’s journey from out-of-home care to permanency. It collects data on length of time between the start and end of a permanency hearing, time between permanency hearings, continuances, changes in permanency goals, and case outcomes. Two members of the CIP staff review approximately 20 recently closed cases, resulting in reviewing over 100 permanency hearings. 

Local Assessments and Action Plans 

Upon request, CIP reviewers assess local jurisdictions’ permanency hearing process in a systematic way using the developed tools. Following a county-level assessment, data from the court observation and case file review are analyzed and reported back to the courts. The CIP then works with local jurisdictions to develop an action plan to implement recommended policy and practice changes, including establishing measures to determine success.

Self-Assessment Tool Kit

A final strategy in CIP’s logic model is developing a tool kit for counties to assess their permanency hearings independently. Counties across the state have expressed a desire to closely examine their permanency hearing process, yet the CIP lacks the human resources to provide local assessments for every county in New York. The CIP will develop this tool kit to address this need. 

Lessons Learned—Information Gathering

Assessing and supporting hearing quality in New York is a long, ongoing process. Throughout the information–gathering stage, several valuable lessons were learned about the types of data collection and how to best use the information. 

Securing the parent’s voice is a challenge. Several dates and times were scheduled with a group of parents, but none of those dates were attended. Several issues arose in the planning of a focus group to capture the parental experience. First, there was a concern for confidentiality when the parents had an active case. 

Second, the CIP did not want to cause an undue burden on parents by asking them do additional tasks beyond their case plan. It might be easier to schedule a focus group with parents if (a) incentives are offered (e.g., gift cards), (b) food is provided, (c) daycare is provided, and (d) the parents have recently closed cases. 

Feedback from child welfare practitioners and parties is critical. Seeking input from child welfare practitioners and parties helps define evaluation questions and refine data collection instruments. It also creates greater buy-in when those involved in the process have contributed. 

Buy-in is crucial to continue the work. Having the same theme emerge from four different and diverse jurisdictions is a good counterargument to “that data isn’t from my court.” It is still not 100% effective, however. The CIP recognized that some jurisdictions may need to be guided by their own jurisdiction’s results. In those situations, a survey tool for local jurisdictions was created.

Developing and translating survey questions requires expertise. When designing questions for a survey or a focus group always seek input on the nature of the questions. Someone experienced in survey design will spot if questions have dual meaning, or do not capture the information you seek. 

Further, focus group questions do not always translate well into survey questions. Focus group questions facilitate dialogue between group members. As such, the questions are often long and require follow-up. In an online survey, consumers get fatigued with too many open-ended questions and no one online to follow-up if the participant has a question. The grey area of a question cannot be explained in this format.

Note taking is vital to capture good focus group data. Ideally, one person facilitates and another person takes notes. Note taking should capture content as well as the quantity and intensity of discussion. For example, it is different if one person identifies a challenge than if the entire group agrees. Contested issues should also be documented to add context to findings.

When an outside consultant reviews your data, always review the results. A consultant may bring expertise to the process of turning the data into useable information. If the consultant was not the focus group facilitator or note taker, review the findings with those who served in those roles to ensure the results match the information conveyed on site. 

Conclusion

Dependency cases are often complex, requiring judicial oversight across multiple hearings throughout the life of the case. While there are some standards as to what these hearings should look like, it is largely up to the judicial officers and system stakeholders to shape what occurs in each hearing. New York has begun an important process, examining the quality of these hearings with the ultimate goal of providing data back to the courts to identify challenges and successes so court practices may be improved. 

Improving the quality of these hearings should lead to better outcomes for children and families involved in the system. Using the methodology shared in this article, New York will be able to examine these outcomes and provide much-needed empirical support for this issue. Other courts can contribute to this growing evidence-base by examining quality of hearings and how it may be related to case outcomes.

 

Alicia Summers, PhD, is the Director of Research and Evaluation at the National Council of Juvenile and Family Court Judges.

Christine Sabino Kiesel, Esq. is the Coordinator of the Child Welfare Court Improvement Project for New York State’s Unified Court System. 

The content of this article represents the thoughts and opinions of the authors only and not that of the New York State Unified Court System.

 

Endnotes

1. Family Court Act Section 1089(a)(2).

2. Family Court Act Section 1089(a)(3).

3. New York State Unified Court System, Universal Case Management System - Family Court - CWCIP Data Metrics. Progress of children followed through 12/31/2014.

4. Family Court Act Section 1086

5. Recognizing the great subjectivity that is embedded in a decision to render a specific goal, the goals that were identified as inappropriate had to meet special criteria: Reunification goal with noncompliant caregivers for more than 24 months; APPLA goal for a child less than 14 years of age, and reunification for more than 12 months where the caregivers have not made progress at alleviating the reasons for removal.

 

 

Tips for States 

Exploring hearing quality can be complex. The following ideas simplify the process. 

Have a plan! Identify your research questions. Figure out what you want to know. This will inform the scope of the project, including what data collection instrument is best suited for your work. Create a logic model and work plan to organize your plan in a linear and concrete way with identified outcome measures. 

Ask frontline child welfare professionals and parties. Professionals and parties involved in child welfare proceedings can help define what constitutes a high-quality permanency hearings. They can provide insight into their perception of what quality means and can help direct the focus of more in-depth work.

Consider a multi-method approach. Data can be collected in many ways. Surveys, focus groups, court observation, and case file review (to name a few) all provide different information about hearing quality. Consider your audience to determine the most effective way to collect data and information.

Ask for help. Research or evaluation expertise is not necessary to get started. However, someone with experience in these areas could help inform the process, review or create appropriate instruments, and help interpret the data. Resources are available to help courts and court improvement programs do this work. Technical assistance providers such as the American Bar Association’s Center on Children and the Law, National Council of Juvenile and Family Court Judges, or the Capacity Building Center for Courts may be able to assist. Creating academic partnerships with university faculty or students may yield additional resources. 

Start small. Assessing hearing quality does not have to be a big process. Use the information gathered to identify a focal point and start there. For example, you may just want to explore how parents or youth are engaged in the process (e.g., do they have an opportunity to be heard, are they asked specific questions, etc.), or what topics are discussed at hearings. 

Pilot test your instruments. Designing a valid and reliable instrument can be a challenge. Test the tool to see how it works before you implement it on a larger scale. This typically includes using the tools to code a hearing or review a file. Coders can determine if there are things that work well or items that are hard to capture. Testing also allows the coders to identify “tricky” items, or items that they have questions about how they should be coded. This provides an opportunity to hone the tool and clarify coding instructions. 

Make the data meaningful. Think about how your data can be used. Data can inform next steps on the project, be used to design instruments for data collection, or be used to solicit funding or interest for more in-depth work. More importantly, providing data back to the court in a meaningful user-friendly way can help courts make data-driven decisions that improve practice. 

Be flexible. Many things may not work out as you envision. Being flexible in your approach can help relieve some stress and lead to good alternatives.

Be patient. Engaging in a meaningful qualitative review takes time, persistence, and patience. Be thoughtful at each stage and focus on achieving better quality hearings.

Share your work. Hearing quality is a complex and nuanced. There is still much to learn in the field about what factors contribute to a quality hearing and how hearing quality may affect case outcomes. Disseminating information about methodology, tools, and findings can inform work in other jurisdictions and move the field forward.