Making Reform Work for the Educationally Disadvantaged: Lessons Learnt from the Success of HOTS

by Dr. Stanley Pogrow. Educational Leadership. Feb 95
(With a few clarifying changes)
by Dr. Stanley Pogrow

Dr. Stanley Pogrow is an Associate Professor of Educational Administration at the University of Arizona where he specializes in instructional and administrative uses of computers. Dr. Pogrow is the developer of the HOTS and SUPERMATH programs.


The first article on the Higher Order Thinking Skills (HOTS) project appeared in this journal in September 1985 (Pogrow, 1985). Today approximately 2,000 schools in 49 states are using HOTS as their Chapter 1 and/or LD programs in grades 4-7. This makes HOTS one of the most successful reform networks developed to date. This has all been done without a sales force. Why has this non-traditional program and network been so successful?


When I first got involved in the HOTS program I did not start out to develop a network or even a national program. I had been invited by a school to develop a Chapter 1 intervention around some ideas I had proposed in a speech. At the time I had no background in working with Chapter 1 students or in developing thinking skills. I accepted the offer largely because my University salary necessitated that I work in the summer. I still remember sitting around a table with a blank sheet of paper, and everyone looking at me and asking me what they should do when the school opened in September.

That was 14 years ago. Today there are approximately 2,000 schools using HOTS making it one of the most successful reform networks to date. This has all been done without a sales force. Why has this nontraditional program and and network been so successful? In retrospect, the humbleness and uncertainty of the beginning became a strength.

Over the past fourteen years, we developed and refined, a sophisticated and creative HOTS curriculum, long with a model of Socratic teaching and teacher training. While the literature is replete with the promised benefits of 'new paradigms', the biggest payoffs in education, as well as in business and life, come from the ongoing refinement of powerful techniques. Indeed, I felt fortunate to work with wonderful teachers and administrators during this refinement period as I watched others chase hither and yon over the same period after the new paradigm of the day. One year the new paradigm was self-concept development, then it was higher order thinking skills, then it was thematic approaches, then it was authentic testing, inclusion, schoolwide, etc. etc. etc. The 14 years of refining the HOTS techniques resulted in ever-increasing success in producing learning across a wide range of students on a consistent basis. The success has crossed cultures and regions.

The Benefits of a Network

As the number of HOTS sites grew it seemed natural to form a network to exchange information and ideas. The network makes possible a different, more realistic type of research than is possible in the highly controlled (contrived) settings of limited scope and limited duration in which most educational research takes place. Spontaneous, constant feedback from hundreds of teachers around the country as to their students' response to the curriculum became a continuous flow of valuable data. Such information flows, a normal part of the HOTS network, would be tremendously expensive under normal conditions.

As the network grew over the years to its current size, I was pleasantly surprised as a professor as to how much we were learning about the nature of the learning needs and problems of disadvantaged students. Professors generally do not have the opportunity to study sophisticated, large scale interventions in realtime, i.e., as they are happening. As a result, Such research artificiality leads to simplistic and misleading findings.

Over time, patterns emerged in our data that lead to precise conclusions about the conditions and reasons for success of the HOTS intervention. I call this type of research around large communications flows 'Pattern Sense Making'. Our conclusions are then transmitted back to the teachers via a newsletter and incorporated into updates of the HOTS curriculum and teacher training techniques.

Even though the primary motive behind the research is to make HOTS more effective, I have been amazed how 'Pattern Sense making' has generated fundamental new knowledge about the nature of the learning problems and needs of educationally disadvantaged students. (See the sidebar for a summary of basic research findings.) In addition, this approach to research has generated very different conclusions from those conventional research—conclusions that I believe are more valid and valuable for making national policy than those generated from either the prevalent quantitative or qualitative research techniques.

Why the HOTS Network is Successful

The primary reason for the success of HOTS as a large-scale, enduring reform is that the program works, and works consistently for a wide range of students and circumstances. That success has crossed regions and cultures. The fundamental concepts that make HOTS an enduring reform are probably critical for making any reform work with kids on a large-scale.

For a reform to work on a large scale over an extended period of time it must have a state-of-the-art curriculum. If an intervention does not have a highly sophisticated and creative curriculum, it does not provide anything that teachers cannot develop on their own. Unfortunately, in my research I've found very little in the way the of exemplary curricula for the middle school grades, despite more than 35 years of reform rhetoric (Pogrow, 1993). Most of the available curricula are simplistic variations on a theme, designed to be consistent with the marketing techniques of publishing houses.

New forms of curricular materials are the fuel of effective reform. One advantage of a network is that it provides an alternative mechanism for disseminating curriculums. Unfortunately, most of the reform networks either do not provide curricula, or use prosaic materials. The rhetoric of excuses for not developing state-of-the-art curricula are that: a) it is undemocratic to 'impose' a curriculum, and b) a curriculum is an individual affair that must represent local needs. Ultimately, however, networks that focus primarily on disseminating rhetoric and enthusiasm for change get bogged down when the enthused have to build the substance of the change on their own, and/or when the simplistic curricula do not perform.

Most advocates of reform consistently underestimate the importance of new forms of curricula, or the difficulty in generating them, largely because they have never tried to develop them. It is easy to physically move students for one part of a building or classroom to another to implement cooperative learning, inclusion, and schoolwide approaches—but where is the new curriculum to make them work?

In addition to a state-of-the-art curriculum, a successful reform needs an effective pedagogical approach and intensive training in these pedagogical techniques. Good curriculums offer the potential for new types of interactions between students and teachers. This potential is seldom realized on a consistent basis. I often see 'whole language teachers', for example, teaching literature using the same pedagogical techniques they used with basals, or teachers presenting hands-on science activities using direct instruction techniques. The results from mixing new techniques with old pedagogy are usually awful.

The keys to developing better pedagogical techniques are to have: a) a very clear set of appropriate teaching practices, and b) state-of-the-art training in these techniques. The teacher training must emphasize the development of new talking and listening reflexes around the use of the curricular materials, as opposed to training teachers in the philosophy and theories of teaching. The teaching techniques and training must develop skills in talking and listening to students in these new ways. HOTS training techniques were designed from the ground up and were drawn from the theater.

To be successful a reform must also focus on those grade levels and students who can benefit from it. Contrary to popular myth, the most effective interventions are those that specialize in particular grade levels and types of students—as opposed to those that pretend to be in clusive. This is especially true for interventions designed for educationally disadvantaged students. For example, Reading Recovery is effective with first graders, and is so spe cialized and focused that its benefits carryover for several grades thereafter.

Contrast this with an unfocused reform such as the use of manipulatives in math. Because they seem to be effective in the early primary grades, manipulatives are now being advocated for all grade levels. As a result, traditional manipulatives have become almost as boring for older students as textbooks.

Indeed, the experience of math manipulatives is very common. Most reforms used after the third grade are based largely on success in the earlier grades, and when used inclusively they simply do not work. In addition, while whole school reform remains an ideal, and has been attempted for at least 100 years, it has never been accomplished successfully in more than a few schools in this country.

Recommending that reforms be highly focused is at odds with the current rhetoric which advocates all inclusive approaches; i.e., lump all students together, treat every one the same, get everyone learning cooperatively, do things schoolwide, etc. However, all of our research and experience suggest that students are even more heterogeneous that previously suspected and require very different interventions for at least part of the day.

For example, our research suggests that there are three very different learning problems and needs even within the Chapter 1 population. From 60 tp 80 percent of the students suffer from metacognition deficits, about 5 to 10 percent have undiagnosed severe dyslexia, and the rest are borderline mentally handicapped.

I am as idealistic about achieving inclusion as anyone. By overcoming metacognition problems, HOTS provides the specialized help that can enhance the skills the majority of Chapter 1 and LD students to the point where they can not only be physically included, but more important, to be academically included, that is to be academically successful in heterogeneous settings.

Unfortunately, I have no idea of how to help the other two categories of Chapter 1 and LD needs. So I have to wonder whether it is really possible for any teacher to deal effectively with all three Chapter 1/LD learning problems—not to mention all the other students' special needs? Inclusive rhetoric about how it is wrong to label students, or how "all kids can learn", or how kids do not have deficits, etc., do not make these special problems and needs go away.

The results from the 'Pattern Sense Making' research suggest that those who argue that all students should be treated the same, and those who argue that all have different needs, are both wrong. It appears that we can develop specialized interventions and make rational decisions about placement for a clearly identifiable, manageable number of students with special learning needs. This is very good news for designing practical and effective large scale reforms.

At the same time, the specific nature of the different learning problems suggest needs limited set of special needs are very different. This means that there is no such thing as an intervention or curriculum that is effective across the board, anymore than miracle drugs work under all conditions. Even highly effective programs such as HOTS are beneficial only under a limited set of conditions. Ignore those conditions and the programs fails to provide substantial help. Fortunately, we know what the conditions are for HOTS. For example, we know that HOTS does not work with borderline educationally men tally handicapped or severely dyslexic students, but it does work with metacognition deficient students, regardless of whether they are in Chapter 1 or LD programs. Such a focused, knowledge-based 'truth in disseminating' approach is the only way for reforms to work on a large scale.

In addition, for a reform to be successful, it must maintain high levels of quality control every step of the way. Weak teachers, weak trainers, inadequate materials and communication processes can quickly render the best program ineffective. Maintaining and improving quality in all the details has to be a mindset.

HOTS has been blessed with wonderful teachers, trainers, staff. HOTS also invests in a great deal of up front communication with principals, coordinators, and teachers, as to the conditions that must be followed if the program is to succeed. To be eligible to become a HOTS site, schools must first sign an agreement as to how they will implement the program. While such communication dissuades about two-thirds of those interested in the program from using it, such communication also insures a high success rate among those who choose to do it.

Finally, for a reform to work on a large scale it has to be structured and detailed—and also highly creative. Reforms such as Career education or pure Whole Language that are so amorphous as to be officially undefined, run out of steam and/or survive in a form that has little resemblance to the original conception. HOTS combines a high level of creativity with a high level of specificity—the same strategy successfully used throughout time by the performing arts.

Why Aren't There More Examples of Effective Reform Networks?

The experience of the HOTS network has generated the type of knowledge needed to help educationally disadvantaged students in a scientific manner. We have been able to define far more precisely what the real learning problems are, and how to focus needed services. The knowledge generated also provides the basis for understanding why traditional and progressive approaches are both not working.

Unfortunately, effective large-scale reform networks are rare. The only two networks that I know of that practice the principles of reform effectiveness outlined earlier are HOTS and Reading Recovery. The new version of the Junior Great Books comes very close and is making the types of changes needed to make it a focused intervention.

Why aren't there more such networks? First, it takes a lot of hard work and luck. Second, the reform climate isn't right. Ultimately, the existence of reform networks rest not only on the effectiveness of the interventions, but also on the attitudes of practitioners and reform advocates. Most reforms today are based on good intentions, idealistic philosophy, and tons of advo cacy. The reform focus in the 80's was on increasing students' self-concrept, and the sim plistic stratagem in the 90's is empowerment. To maximize buy-in, reformers now suggest that the best approach is to encourage everyone to do their own thing.

Such conceptions of reform, however, are mostly fantasy. The reality is that successfully helping most children to learn and to develop a sense of understanding is a very complex process; it requires sophisticated and specialized interventions with well-designed curricula in the hands of a good, well-trained teachers. Increasing learning on a consistent basis requires new tools developed with a blend of hard work and creativity.

In addition, the primary instinct of most teachers is to help their students, not to develop their own materials and tools. Most teachers prefer to use existing programs instead of developing their own if: a) they are allowed to decide whether to buy in to the program, and b) the program helps their children learn more powerfully.

Anticipated changes in Federal legislation and current conceptions of political correctness threaten the existence of even successful networks. The notion of a highly focused intervention is politically incorrect at the present time. Much like the early 70's,we are at the apex of a historical cycle where reformers are hell-bent to change whole schools, and anyone trying to do something else is considered reactionary. In addition, the drive for democratic participation at any cost makes organized programs highly suspect.

The formation and success of substantive reform networks require that the profession move away from generating reform primarily through philosophy and advocacy to an orientation of producing far more powerful and creative tools, and using them more precisely. For example, instead of saying "all kids can learn", HOTS specifies how we can get them to learn. This reorientation of reform perspective requires very different policies coming out of the profession and Washington than those now coming down the pike.

It's somewhat ironic that when we started fourteen years ago most of the criticism came from conservatives who were suspicious of anything that was not basic skills. Now the criticism is coming from progressives who reject focused approaches—regardless of whether they help kids. Our network will be ready when 4-5 years from now the field becomes disenchanted with the current wave of simplistic reforms. Indeed, that is how this network got started in the first place—as an alternative to the failed reforms of the time.

So you want to be a reformer?

Despite all the brave talk within the profession, there is a desparate need for additional, highly effective programs. My work demonstrates that an unknown individual with energy, passion, commitment, and chutzpah can design an effective intervention and build it into an effective reform network. To readers who are interested in taking their own ideas and building a reform network, I offer some parting advice.

1. Ignore conventional University wisdom, and proceed in a scientific way with your own hunches and observations. Trust what you see happening with kids and teachers. I was lucky that I did not know the prevalent theories of the time. It was only after the basic conception was developed that the work of scholars such as Vygotsky, Sternberg, Ann Brown, and others became valuable to help work out some of the details.

2. Develop a perspective that you do not have expertise until you are able to make something actually work with teachers and students on a reasonable scale. This perspective keeps the focus on producing student learning as opposed to supporting one's ego or proving a pet theory or philosophy. Also, when something does not work, do not blame the test. Instead, make the curriculum and training better.

3. Redefine your role to balance expertise and entrepreneurism with integrity. One of the things that I am proudest of is that whenever financial gain conflicted with the data about the best way to implement the program, all decisions were made on the basis of the data regardless of whether that meant that there would be fewer sites and less revenue. This is something that conventional entrepreneurship cannot achieve.

Is all the work involved with building a network worth it? I think so. Professors who satisfy themselves with just producing ideas, and not going the next step of trying to make their ideas work in the real world, are missing out on one of the great experiences in life. There is nothing more satisfying than watching a good teacher transform a group of students over a one-two year period and knowing that your ideas helped make it possible.


Oneil, J. "Aiming for New Outcomes: The Promise and the Reality." Educational Leadership, March 1994, pp. 6-10.

Pogrow, S. "Finding The Beef: Are There Exemplary Middle School Curricular Materials?" Educational Leadership, May 1993, pp. 62-66.

NOTE: Use the 'Back' button on your browser to go back to the previous page/article, or press the HOTS button at the top of this page to go back to the Home page-----------------------

back to top