Scenarios: District

Step 1: Assessing the Present State

Notice this section is about Step 1: Assessing the Present State because the group is:

  • Identifying existing assessment data sources
  • Developing questions about how students are performing in mathematics on two different assessments

Plan for Improving Achievement in Mathematics

At a meeting of the district leadership team of a moderately large urban school district, the assistant superintendent, Ms. Sullivan, is discussing a plan to increase data use in the district with the goal of improving student achievement in upper elementary mathematics.

The team's decision to take this action resulted from an analysis of state test data, which showed little change in scores at these grades levels for the past three years. They were concerned that students were not entering middle school with the necessary knowledge and skills to be successful. The leadership team had previously agreed that they would run a pilot project to increase data use and use lessons learned from the pilot to eventually scale up to a district-wide roll out of the data use plan.

Ms. Sullivan suggests that the first task for the pilot project is to identify existing assessment data sources to clarify expectations for their use.  She comments, “I am not sure that schools realize how much data are available or understand how the data can be used. I see this as a first step in increasing data use. Once we have this information clearly communicated and understood, we can start to help people analyze the data and make use of the results to improve mathematics achievement. Doing this will also help identify the kind of professional development that teachers and administrators need to support data use.”

Asking the Right Questions

Ms. Sullivan beings the next meeting: “I’ve been thinking quite a bit about the inquiry questions and it seems to me that we should have a general framework of the kind of questions that should be asked of any kind of data within which we can tailor more specific questions. I’d like to suggest, for example, that we have five areas of major foci.” At this point, Ms. Sullivan refers to a slide in her PowerPoint presentation that lists her proposed major foci: a) Overall, how are students performing in mathematics?, b) What are the trends in student performance in mathematics over the past 3 years?, c) How are subgroups performing currently and over time?, and d) What are our relative strengths and weaknesses in teaching and learning?

Mr. Ramirez, the person responsible for district data use, comments, "I think I see where you are going with this, but if we were to use these questions as the platform for inquiry, how would we make them specific to the state data and the interim (benchmark) data?”  “Well,” adds Mr. Newell, the district’s professional development coordinator, “I can imagine that for the first questions we could ask something like ‘How do our students perform by grade level?’ and ‘How does our students’ performance compare with the state performance?’ Questions like these would help them get deeper into the data and begin to get a sense of the current status of student performance.”  With general agreement that this is the right way to proceed, the group engages in discussion about the questions they would include in each of the major foci.

They develop questions that ask about trends in the data, comparisons with other similar schools’ performance, relative strengths and weaknesses according to content strand data, and evidence from new programs and initiatives. “Okay, I think we can be pretty happy with what we’ve got so far,” says Ms. Sullivan. “ I suggest that we move on to think about the questions that we would want schools to ask about the interim or benchmark assessment data.”  The rest of the group agrees. Mr. Casey introduces the idea of looking at the state data and interim data in relation to one another: “It seems to me that we would want the schools to be examining whether the same content areas arise in both the state and interim data. For example, we could see if the weaknesses that show up on the interim data are the same as the ones shown by the state data. What do you all think?” The others nod in agreement with Mr. Casey. Ms. Zacharius suggests, “Why don’t we start by identifying the questions with the major foci categories for them to focus on and then, when we’ve done that, we can go back and think about points where we’d want them to compare the state results and the interim results.”  The group proceeds with Ms. Zacharius’ suggestion and eventually comes up with questions like 'Are the interim test results consistent with the results of the state test?' and 'How are students performing on the interim tests on those areas identified as weak on the state tests?' to show schools how they can use these two sources of evidence.

At the end of the meeting Mr. Ramirez comments: “I’m a little worried that we don’t have the capacity to display the results of our questions in a format that will be easy for the schools to understand. I’m going to go back to my team with these questions and see what we have and what we can come up with.” “Yes,” says Ms. Sullivan, “it’s going to be really important for the schools to have clear visualizations of the data – I really appreciate that you are going to do this. But there is one more thing that I suggest this group needs to do before you talk to your team: I think we need to help schools interpret the data. I’d like to propose that at our next meeting we focus on some interpretive questions within each of our categories. Maybe you could think of a few, email them to me, and I’ll have them ready for the meeting so that we are not starting from scratch. Thank you, everyone.”

 

Professional Development Resources


Leedy, P. D. & Ormrod, J. E. (2004). Practical research: Planning and design (8th edition). Upper Saddle River, New Jersey: Prentice Hall. 978-0131108950

 Summary:

This book provides a broad overview of basic research methodology and takes a hands-on approach to understanding the processes necessary for conducting quality research and generating meaningful results. Topics covered include an overview of the scientific method and research tools, methods for identifying the research problem and conducting a literature review, research design planning and proposal writing, qualitative, historical and descriptive research approaches, experimental and causal-comparative designs, statistical techniques for quantitative data analysis, and writing and publishing the research report.

How To Use:

Stat personnel could use this resource as a foundation for professional development, breaking into modules based on the topic descriptions above. District-level personnel could use this book to support their knowledge and understanding of research methodology, including helping them identify available data from the district and individual schools to develop research questions.


Step 2: Analysis of the Present State

Notice this section is about Step 2: Analysis of the Present State because the group is:

  • Developing a framework for interpreting test results
  • Helping schools analyze and interpret student performance with information from multiple sources, not just state student achievement data

Interpretive Questions

At the next meeting, Ms. Sullivan provides a list of all the interpretive questions for the data the group members sent to her. When the group has reviewed them, Ms. Zacharius says, “Seems like great minds think alike! There is a lot of overlap here.” “Agreed,” says Mr. Newell, “why don’t we go over them, decide which we want to keep and which ones we want to improve?”  

This task does not take the group long, and they quickly come to agreement on which questions they think are the most appropriate for interpretation of their data. They then decide to revise a few questions. Their final list includes these questions:

  • How well are we serving our students?
  • Are some grades/groups performing substantially better or worse than others in math?
  • What are the relative strengths and weaknesses based on the math strand data?
  • Which students, in particular, need help?
  • Are we improving from year to year? From interim assessment to interim assessment?
  • Does the evidence meet our expectations for improvement?

At the end of the process, Mr. Ramirez raises an issue: “You remember at the last meeting when I said we’d need to think about the kind of reports we can generate to display the data?  Well, what I realize now would be very helpful is if we can go through the questions and identify what kind of analysis each one calls for.” “Sure, we can do that – I think that’s a very good idea,” replies Ms. Sullivan.

Now they tackle identifying the appropriate analysis for each question. They find that relatively simple descriptive analyses—such as mean or median scores, percentage of students performing at each proficiency level by subgroup, and longitudinal analysis by cohort—will provide the needed information. Once they have finished, Mr. Ramirez concludes, “I’m so glad that we did this – this is going to be really helpful to me. I’ll generate these reports and to display the data right away and I’ll bring what I have to the next meeting.”

Professional Development Resources


Casserly, M. (2006). Beating the odds: A city-by-city analysis of student performance and achievement gaps on state assessments. Results from the 2004-2005 school year. Washington, DC: Council of Great City Schools.

 Summary:

This is a useful example of how to examine assessment data at the district and state levels, make comparisons between different groups across time, and report data on student achievement in line with the No Child Left Behind Act. Prepared by the Council of the Great City Schools, this report provides detailed information on how major city school systems are performing on state assessments. Data from 65 city school systems in 38 states are presented. The data are broken out by city, year, and grade level for each state test in mathematics and reading. Where the information is available, the data are also reported by race, language, disability and income. In addition to examining student achievement in math and reading over time, this analysis also measures achievement gaps between cities and states and minority populations and whites.

How To Use:

State personnel could use this resource when considering how to examine and report on assessment data. This article could be used as an example for district personnel to help them understand how to compare groups across time and report on student achievement.


Step 3: Transformation Planning

Notice that this section is about Step 3: Transformation Planning because:

  • The schools are given a tool to assist in making sense of the data and beginning to identify areas for improvement

A Protocol for Analysis

The group generally feels they are making good progress. They have a successful meeting looking at the reports from Mr. Ramirez and his team and think they are close to being able to implement the pilot. Then Ms. Sullivan interjects, “You know, looking at these data displays has given me a big ‘aha!.’ We cannot assume that just because we give the schools the questions and show them the reports to generate from our system that they’ll know how to go about examining these data. We definitely need to think about how we can support them in doing that—analyzing the data appropriately.” All nod in agreement.

Then Mr. Newell has an idea: “I bet we could use something from WestEd’s Developing an Effective School Plan  – I’m almost certain they’ll have a protocol for examining data. Let me go to my office and take a look.”  Mr. Newell quickly returns from his office with the materials. “In Developing an Effective School Plan, there is a protocol for writing data statements and then data summaries.  See what you think,” says Mr. Newell, “these seem like a pretty good way to go.”

After reviewing the WestEd resources, there is a consensus among the group members that this is what they need to guide data interpretation. Ms. Sullivan comments, “I really like the way this protocol takes them through the process. It asks them to present the facts objectively rather than to make evaluative or explanatory comments and stresses the need to represent the data accurately by including relevant numerical data when needed for evidence. I think the examples provided will be useful, too.” “I agree,” adds Ms. Zacharius, “and the data summaries lead very naturally from the statements. If they identify the most important things they have found in the data, this can form the basis of their next steps – either gathering more data from any math assessments they have or beginning to think about steps for improvement.”

With all their materials in place, the planning team now concludes that it is time to begin the pilot with their volunteer schools.

Professional Development Resources


Walberg, H. J. (Ed.) (2007). Handbook on restructuring and substantial school improvement. Lincoln, IL: Center on Innovation & Improvement. Retrieved August 7, 2007 from
http://www.centerii.org/survey/downloads/Restructuring%20Handbook.pdf

 Summary:

The purpose of this book is to provide principles for restructuring and improving schools. The Handbook is divided into three sections. The first section is an overview of restructuring, as the authors explain the meaning of restructuring within the context of the No Child Left Behind Act (NCLB) and present early research findings on restructuring schools. The second section includes six modules addressing restructuring efforts and NCLB. Focus topics include district-wide initiatives for improvement, the role of district leadership in improvement efforts, restructuring options and change processes, restructuring through learning-focused leadership, changing and monitoring instruction, and systems for improved teaching and learning. The third and final section addresses indicators of successful restructuring and provides a checklist of actions for supporting an effective restructuring plan. This section can also be used as an instrument to help schools identify needs and strengths of their own restructuring process.

How To Use:

This handbook can be used by districts to help develop and support district-wide improvement and restructuring efforts.



Bernhardt, V. L. (2007). Translating data into information to improve teaching and learning. Larchmont, NY: Eye on Education.

 Summary:

The purpose of this book is to help education professionals evaluate data elements and tools needed to support data-driven decision making at all levels. The author includes examples to demonstrate how actual school districts use data to make decisions about strategies to promote student educational gains. Topics include (but are not limited to) data discovery and mapping, selecting data tools, creating a culture to use data, reporting and using results, and managing the data warehouse. This book is designed for school and district administrators and teacher leaders who want to use data to improve teaching and learning, and university professors who teach future educators at the undergraduate and graduate levels.

How To Use:

This resource can be used by district administrators to learn more about reporting data analysis results and using findings to determine district improvement goals.


Step 4: Transition to a New State

Note that this section is about Step 4: Transition to a New State because District personnel:

  • Take the improvement plans to the schools for implementation
  • Collect data (evaluations) to monitor the effectiveness of the improvement plans
  • Analyze these evaluations and rework their plans to increase data use at the schools

The Pilot

During the pilot project, Mr. Newell and Ms. Zacharius work directly with administrators and teachers in four schools in the district for four half-day sessions focused on mathematics achievement data from state and interim assessments. After each session, they discussed together how the sessions went and what improvements needed to be made and reported these back to the rest of their team.

In general, Mr. Newell and Ms. Zacharius concluded that the materials their group had developed to support the school’s inquiry process worked well.  The data were easily accessible in the district data system—which helped teachers feel confident about data. The reports were user-friendly, especially when participants used the data statement/data summary protocol to figure out what the data meant.  They found that the whole process enabled teachers to identify common areas of strengths and weakness in student performance across mathematics strands. This, in turn, led to a more in-depth investigation of a number of strands to determine specific areas of students’ strengths and weaknesses. For example, some schools decided they needed to collect more data by examining student work, whereas others had end-of-unit assessment data students had completed and decided to examine those data as well.

In the final debrief session with Ms. Sullivan, Mr. Casey, Mr. Ramirez, Mr. Newell, and Ms. Zacharius reviewed the evaluations, which were overall very positive. It appeared that the participants had learned a lot about using data to understand students' mathematics performance. However, all teachers requested additional assistance in planning for improvement.  Ms. Zacharius said, “We should have known this was coming. At least we know what we need to do next from the district perspective – we also need to think about district professional development we need to implement to support this effort.”

Ms. Sullivan added, “I want to get the math coaches involved in this now so they can help with improvement planning. I really want the improvement plans that we require each year from the schools to have some meaning and relevance, rather than just being documents that gather dust. I suggest we go through the process with the math coaches assigned to each school and then decide how we move forward to scale this effort to the whole district. We’ve certainly got our work cut out for us, but I feel strongly that this is what we need to be doing to improve math at this level, and we can use this at all grade levels in the district for all subject areas.” Ms. Sullivan’s comments are met with agreement from all team members.

Professional Development Resources


Schmoker, M. (1999). Results: The key to continuous school improvement (2nd ed). Alexandria, VA: Association for Supervision and Curriculum Development.

 Summary:

This book provides information to support educators in using data to improve student learning, including setting goals, working collaboratively, and keeping track of student-achievement data from many sources. The author includes information on standards and assessments, effective professional development programs, and several examples of successful practices in schools and districts.

How To Use:

District leaders and administrators can use this book to help monitor the effectiveness of improvement plans and analyze and rework plans as needed. Teachers could use this book as reading for professional development or as guidance when thinking about goal setting.


The DataUse web site is a part of AACC and was created and is regularly updated by CRESST
in partnership with WestEd, and supported by the U.S. Dept. of Education.