Becoming a Skilled Consumer of Research

January 6, 2015
8:30 ? Noon
at the
International Congress for School Effectiveness and Improvement
ICSEI 2015
Cincinnati, Ohio

Presented by REL Midwest

Shazia R. Miller - Managing Director, American Institutes for Research

Jeanne Poduska - Managing Director, American Institutes for Research

Cary Cuccio - Managing Technical Assistant Consultant, American Institutes for Research

 

As educators are increasingly called on to ensure that their decisions, programs, and strategies are ?data-driven,? ?research-based,? and ?evidence-based,? this interactive session is designed to provide tools and skills to help those in educational settings to be informed consumers of research. The specific objectives of the workshop are to: provide practitioners with a strategy for identifying core issues; increase attendees? ability to explain the basic criteria for judging the quality of research and assessing its context; increase attendees? ability to apply the criteria to real-world information and judge quality and relevance of research; and provide attendees with simple tools to support their efforts to use research-based practices.

This workshop is designed to speak specifically to the practical needs of local and state education stakeholders?that is, the need for strategies to tightly frame their questions, find relevant documents, recognize the attributes of strong research, and sort through issues of context.

The educational research community aims to produce high-quality research that supports educators? ability to make research-based decisions. Yet the sheer amount of research that is available?not to mention the wide range of quality?makes it difficult for educational leaders to apply what is learned through research toward improving practice. The volume of literature housed in the Education Resources Information Center database (ERIC), just one possible source of information, now exceeds 1 million articles. How is a busy educational leader to make sense of all of that information? This interactive session is designed to provide tools and skills for those in educational settings to become informed consumers of research.

More and more, district and school leaders are required to use research-based interventions and strategies, as well as cite them, in school improvement plans. Many of the leaders are not yet trained to determine the quality of the research or whether the research is applicable to specific settings. With the strong emphasis on using research-based practices, this workshop will help school and district leaders better able to identify useful, high-quality research.

Materials have been developed to support a comprehensive in-person. The workshop format will consist of presentations, participant activities, and group discussions. The objectives of the workshop are to:

  1. provide practitioners with a strategy for identifying problem causes;

  2. increase participants? ability to identify and evaluate sources of research; and

  3. enhance practitioners? ability to assess the strengths and weaknesses of individual pieces of research.

Workshop presentation materials will be provided to the trainees to use during the sessions, and additional materials will be organized as a useful resource guide to consult in the future.

The staff members for this project are senior leaders within American Institutes for Research (AIR), with extensive expertise in the use of a range of rigorous research methods and supporting research-driven changes for practitioners.

The specific content covered in the workshop will focus on the topic of school climate, an issue of particular interest to REL Midwest constituents and members of ICSEI. We will draw from the principles of research quality as described by What Works Clearinghouse (IES); National Registry of Evidence-Based Programs and Practices (Substance Abuse and Mental Health Services Administration); Crime Solutions (Office of Justice Programs); and Shadish, Cook, and Campbell (2002).

The workshop will be presented according to best practices in adult leaning and presenting as drawn from Knowles, Holton, and Swanson, (2012) and Zoller and Landry (2010). Workshop materials will cover the following topic areas:

  1. Introduction?Participants will be grouped into tables of elementary, middle, and high school, and urban, rural, and suburban, with accommodations made to ensure that there are between four and eight people in each group. Clustering groups will give them a common frame of reference in the group activities. The workshop will begin with introductions, the goals of the session, and the agenda for the workshop. (15 minutes).

  2. Identifying a Root Cause?Participants will use a Fishbone diagram to present the various factors and subfactors thought to relate to the problem under consideration. Participants will analyze the results and use the diagram to identify a malleable factor for intervention. After the analysis, each table will report on the factors related to the problem and how they chose which factor to target for intervention. (1 hour)

  3. Identifying Sources of Research?Participants will brainstorm at their table about what they might consider if they were buying a television. During the reporting out phase, participants are likely to have come up with answers such as asking a friend, using Consumer Reports, figuring out what their specific needs were (such as space). The leaders will discuss these answers and relate them back to the types of resources that might be used in identifying a research-based policy or program to address a particular problem. For example, instead of asking a friend about which television they purchased, an educator might ask another educational leader what strategies they used to improve attendance. Similarly, instead of using Consumer Reports, an educator might turn to research syntheses provided by a trusted source. In this way the leaders will demysticize the process of finding good, research-based interventions by relating them to a big decision many people have made in their nonprofessional life. (30 minutes)

  4. Break (15 minutes)

  5. Assessing the Strengths and Weaknesses of Research?Leaders will present the Research Quality Tool, a one-page matrix for taking notes on various aspects of quality and relevance of an individual piece of research. The components of this tool will be related back to the components recognized in the Identifying Sources of Research activity. Participants will be asked look over a set of research materials examining an intervention intended to improve school climate. These research materials will be chosen to reflect a range of material types (e.g. a research article, a research synopsis in a practitioner journal, marketing materials, a newspaper article.) Participants will work in their groups to weigh the strengths and weaknesses of the various sources and then report out. In the reporting out part of this component, workshop leaders will emphasize the advantages and disadvantages of different types of research, how strengths and weaknesses can be identified, and what tradeoffs to consider when choosing an intervention. (1 hour)

  6. Additional Tools?Leaders will provide a list of available sources of information that would be useful to educators. The advantages and disadvantages of different sources will again be emphasized. (15 minutes)

  7. Wrap-up?Leaders will summarize what was learned during the workshop. (15 minutes)


The workshop session will include a customer satisfaction assessment. Results will be incorporated into the summary of the workshop to be delivered to the Institute of Education Sciences no later than four weeks after the workshop is conducted.


References

Benjamin, J., & Morales, L. (2013). Understanding the problems you are trying to solve: Root cause analysis. New York, NY: Carnegie Endowment for the Advancement of Teaching.

Knowles, M. S., Holton, E. F., & Swanson, R. A. (2012). The adult learner. Burlington, MA: Butterworth-Heinemann.

National Institute of Justice, Office of Justice Programs. (n.d.). About Crime Solutions.gov: Program review and rating from start to finish. Washington, DC: Author. Retrieved from www.crimesolutions.gov/about_starttofinish.aspx

National Registry of Evidence-Based Programs and Practices. (2014, February). Quality of research. Washington, DC: Substance Abuse and Mental Health Services Administration, U.S. Department of Health and Human Services. Retrieved from nrepp.samhsa.gov/ReviewQOR.aspx

Shadish, W., Cook, T., & Campbell, D. (2002). Experimental and quasi-experimental designs for generalized causal inference. New York, NY: Houghton Mifflin.

Wan, Y., & Scher, L. (2014). Characteristics of research and evaluation studies sponsored by districts (Limited Distribution Memo). Naperville, IL: Regional Educational Laboratory Midwest.

What Works Clearinghouse. (2014, February). Procedures and standards handbook, version 3.0. Washington, DC: Institute of Education Sciences, U.S. Department of Education. Retrieved from ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v3_0_standards_handbook.pdf

Zoller, K., & Landry, C. (2010). The choreography of presenting: The seven essential abilities of effective presenters. Thousand Oaks, CA: Corwin Press.