In 1990, NEA embarked on a major research effort to define and quantify the critical organizational conditions of school quality. The research led to the development of the NEA KEYS initiative. Understanding the evolution of the NEA KEYS research and the theoretical underpinnings of the development efforts is important because it addresses directly questions of credibility as well as the validity and reliability of the survey instrument and the initiative upon which it is based. This section traces the development of the KEYS initiative from the original research to the most recent efforts to broaden the scope and improve the KEYS instrument.

The Original KEYS Research

The original research study involved the development of a 256-item questionnaire administered to a randomly selected group of 3,000 NEA teacher members. The “effective schools” research and Edward Deming’s quality and continuous improvement principles provided the theoretical underpinnings for the development of the overarching organizational conditions thought to affect teaching and learning and student achievement in schools. The NEA researchers then used these overarching principles to generate the set of questions that comprised the questionnaire. Teacher respondents were asked to rate their schools in three areas:

1. To what extent were the conditions enumerated in the questionnaire present in their schools?
2. How did they rate the “quality of education” in their schools?
3. How did they rate the level of student achievement in their schools?

Over 1,600 teachers completed the questionnaire. Respondents’ data were then analyzed to see how the questionnaire items clustered together and to determine the extent to which the existence of various organizational conditions could be positively related to high ratings of school quality and high student achievement.

The results were very encouraging. Based on teachers’ responses, it appeared that 35 indicators of school quality, clustered around five major dimensions (shared understanding and commitment to high goals, open communication and collaborative problem solving, continuous assessment for teaching and learning, personal and professional learning, and resources to support teaching and learning), were significantly related to student achievement, irrespective of social or economic conditions. The research findings were further validated through a series of case studies in six school districts across the country.

The Original KEYS Instrument

Based on the original research, the 256-item questionnaire was streamlined into a KEYS opscan instrument, containing 120 items. The reduction of items was accomplished without affecting the instrument’s validity. The instrument is a self-assessment tool that is completed by the school and yields a set of objective data related to the 35 organizational conditions shown empirically to support effective teaching and learning and high student achievement. These school assessments are then compared with benchmarks, thus stimulating further inquiries and meaningful staff discussions and focused efforts to improve the quality of teaching and learning processes.

The Development of the NEA KEYS 2.0 Survey

Although the original KEYS instrument has been used successfully in more than 600 schools around the country, NEA has recently undertaken the research task of revising and improving the survey. The revisions were undertaken to make the instrument more user friendly in terms of language, data reporting, and interpretation and, more significantly, to orient all survey items on a series of integrated teaching and learning concepts and conditions that directly affect the core purposes of schools — improving student achievement.

The NEA KEYS 2.0 Research

The revision of the KEYS instrument was accomplished through a two-phased process. Phase 1 included the revision of existing items and the development of new items specifically related to teaching and learning concepts. Phase 2 involved the pilot testing of the newly revised instrument in schools across the country and the development of preliminary national norms and best practices benchmarks.

Phase 1: The Revision of Survey Items

The Advisory Group
An advisory committee was formed to provide guidance and direction. Committee members included Jomills Braddock, University of Miami; Helen Mark, Ohio State University; Sylvia Rosenfield, University of Maryland; Mark Smylie, University of Illinois at Chicago; and Floraline Stevens, a private consultant from Los Angeles, California.

Since the original KEYS had been used successfully and our purposes for the revisions were to refine and improve the instrument, consensus was reached to retain from the original survey a core set of questions that was most predictive of student achievement within each of the five major dimensions. Thus, the KEYS 2.0 instrument retains the “best” of the original KEYS including the five major dimensions (shared understanding and commitment to high goals, open communication and collaborative problem solving, continuous assessment for teaching and learning, personal and professional learning, and resources to support teaching and learning). As part of the revision, all of the questions retained were reviewed and clarified, where necessary, to eliminate ambiguity that could be a source of error.

Expanding the Scope of the Instrument

In addition to the revisions of questionnaire items, KEYS 2.0 provided a major opportunity to expand the scope of the instrument. Whereas the original instrument was limited to an assessment of a school’s organizational conditions that support teaching and learning, KEYS 2.0 includes a large set of questions that focus most directly on issues of teaching and learning such as the school’s quality of its content standards, curriculum, assessments, instruction, and professional development.

Theoretical Underpinning of Curriculum and Instruction Items

The curriculum and instruction concepts and items included in the revised questionnaire focused on general, research-based principles rather than specific pedagogical styles or curricular programs. The items were drawn from several sources including Fred Newman and Gary Wehlage’s work on authentic instruction and assessment at the Center on Organization and Restructuring of Schools, Charlotte Danielson’s framework for teaching, the American Psychological Association’s Learner Centered Psychological Principles, the five core propositions of the National Board for Professional Teaching Standards, and the indicators of school quality identified by the National Study of School Evaluation.

Feedback from Focus Groups

As part of NEA’s efforts to improve the instrument, the revised and newly developed draft questions (curriculum and instruction) were reviewed by groups of NEA teachers and education support professionals as part of four focus group discussions. As a result of these group discussions, the items were further revised and assembled into a draft questionnaire ready for pilot testing.

Unlike the original KEYS research where the questionnaire was administered to a random sample of NEA teacher members, the KEYS 2.0 instrument was administered to all education employees in several schools. Thus, for the KEYS 2.0 research, the data points used to identify the indicators of school quality represent very stable and valid measures.

Thirty-eight schools, representing 68 percent of the 45 schools designated, participated in the KEYS 2.0 pilot test, including 28 schools affiliated with NEA and 10 with AFT. The sample was a national random sample of schools drawn from the Department of Education, National Center for Education Statistics database. The randomness of the sample was constrained, however, to ensure that all six NEA regions were represented and that the sample included elementary, middle, and high schools in urban, suburban, and rural settings. The sample included 12 elementary schools (five urban, six suburban, two rural); 13 middle schools (four urban, three suburban, six rural); and 12 high schools (four urban, five suburban, three rural).

The total number of individuals responding to the questionnaire was 1, 491 representing 52 percent of the total population. Seventy-one percent (1,061) were teachers; the other 29 percent included 378 education support professionals and 52 administrators.

Analysis of the KEYS 2.0 Data

In addition to the questionnaire data from the staff in each school, several measures of student achievement data were collected. The measures included all available school test data as well as respondents’ perceptions of how well students were achieving.

The school questionnaire data were aggregated and examined using a series of factor analyses to determine if and how items clustered together; that is the extent to which certain items statistically tended to measure the same underlying concepts. This procedures allows for the identification of the conditions or indicators of quality that could be measured. The second part of the analyses involved a series of linear regressions to determine the extent to which the indicators of quality identified were positively related to student achievement.

Results of KEYS II Research

Forty-two indicators of quality clustered into six main “KEYS” were identified. The first five keys are the same as the ones identified in the original KEYS. The sixth key, curriculum and instruction, is new. Based on the NEA research as well as reports in the education research literature, all 42 quality indicators are considered important elements of a high performing school. High scores on all of the 42 indicators, clustered around the six KEYS, represent an integrated view of a wellfunctioning, high-achieving school. In addition to identifying the six major keys and the 42 indicators of a high performing school, the NEA KEYS 2.0 research allows each school the opportunity to compare its scores on each indicator with the average scores of all the schools in the pilot as well as with the school scoring in the 90th percentile of the distribution of all the schools in the sample.