Evaluations of the Higher Order Thinking Skills (HOTS) Program

 

The following pages contain a series of evaluations conducted recently, and over the past decade, which show that:

  • HOTS has consistently produced substantial effects across a wide variety of tests including the new state tests
  • HOTS has been highly effective in a wide range of settings, from high poverty urban to high poverty rural
  • HOTS has been highly effective as a key element in comprehensive school reform
  • HOTS produces a wide variety of powerful effects simultaneously
  • HOTS has been highly effective both as a part of a comprehensive approach or as a stand-alone targeted intervention

HOTS remains one of the few national interventions to consistently show gains for educationally disadvantaged students in grades 4-8.

For more information on HOTS, as well as Supermath and Socratic workshops, contact: 

•     www.HOTS.org.

•     info@HOTS.org

•     800-999-0153

Table of Contents

 

District/School

 

 

Year

 

Evaluation Significance

Catherine Strehle Elem & Estelle Elem

Both are in Jefferson Parish District in Louisiana

2002

2002

Both schools made such high gains on the state test that they both received the states highest gain ranking level of "Exemplary Academic Growth"

Estelle also showed substantial growth on national Iowa test in all content areas.

Cleveland County District, NC

2001

Gains in schools with HOTS on NC state test exceeded state growth targets, including exemplary growth, and Non-HOTS schools generally failed to meet state growth targets.

Elvira Elementary in Tucson

1995

A landmark study showing the wide variety of gains in academics and thinking in HOTS students as compared to control students.  These data and a follow-up study will be the basis of forthcoming articles. 

The possibility of multiple simultaneous gains was confirmed at Estelle Elem above.

Williams Elementary, TX

1995

First study to show that gains on nationally normed tests were also occurring on state tests such as TAAS.

Mann Middle, San Diego

1994

Showed substantial gains in a large very high poverty urban middle school.

Detroit

1993

Showed a large number of students across many schools in a distressed urban district making twice the growth as compared to comparison students.

Soldotna Alaska

1992

HOTS students make twice the gains even in a remote rural native setting with small schools.



Title of Evaluation: Results from Catherine Strehle Elementary School in Jefferson Parish School District on the Louisiana LEAP Test.

Author: Dr. Geraldine Settoon, Principal

Relation of author to model developer: None, other than being the principal of a school using the HOTS Program.

Date evaluation report completed: 2002

 Months/years covered by evaluation:  2 years

 N:  All fourth grade students in the school, 57 students.  Fourth grade is the only elementary grade that the LEAP test is administered.  

 Demographics of sample population:  Low income students, 87% free and reduced lunch.

 Purpose of study:  To study the effects of the HOTS Program on the Louisiana state test (LEAP) and on meeting state accountability growth standards

 Methodology: Analyzing the growth in the school in relation to state growth accountability standards, as well as overall growth in English/Language Arts component of the state LEAP test. 

Measures/indicators used to assess effectiveness:  English/Language Arts scores on the Louisiana Educational Assessment Program (LEAP), and the School Performance Score calculated by the state.

Findings:  The table below shows the progress that Strehle Elementary made at the fourth grade, which is the only elementary grade where the state's high stakes test, LEAP, is administered.

1999

2001

Percent passing English/Language Arts component

48%

83%

School Performance Score (SPS)

59

81


Catherine Strehle Elementary embarked on comprehensive school reform in 1999 using an FIE grant and made HOTS the center of their improvement effort.  The results dramatically exceeded the state's accountability benchmark for expected School Performance Score (SPS) growth of 7.5, and the school won an "Exemplary Academic Growth Award" from the state. In addition, 93% of the students passed the overall state test.

back to table of contents


Title of Evaluation: Results from Estelle Elementary School in Jefferson Parish School District on the Louisiana LEAP Test.

Author: Dr. Joan Gremillion, District Director of Research

Relation of author to model developer: None, other than being the principal of a school using the HOTS Program.

Date evaluation report completed: 2002

Months/years covered by evaluation:  2 years

N:  All fourth grade students in the school.  Fourth grade is the only elementary grade that the LEAP test is administered.  Corroborating data is also provided on the fifth grade Iowa exam.  

Demographics of sample population:  Estelle is a large urban elementary school of 970 students, 80% of whom receive free and reduced lunch, and 52% of whom are minorities.

Purpose of study:  To study the effects of the HOTS Program on the Louisiana state test (LEAP) and on meeting state accountability growth standards, and on the Iowa test.

Methodology: Analyzing the growth in the school in relation to state growth accountability standards, as well as overall growth in English/Language Arts component of the state LEAP test and Iowa test. 

Measures/indicators used to assess effectiveness:  English/Language Arts scores on the School Performance Score based on the Louisiana Educational Assessment Program (LEAP), and results on the Iowa test in Grade 5.

Findings:  The table below shows the progress that Strehle Elementary made at the fourth grade, which is the only elementary grade where the state's high stakes test, LEAP, is administered, and in Language and Math for fifth graders on the Iowa test.

1999

2001

School Performance Score (SPS) Grade 4

67.4

83.2

Iowa Language Arts/Grade 5

48

63

Iowa Math/Grade 5

44

53


Strehle Elementary embarked on comprehensive school reform in 1999 using a CSRD grant and made HOTS the center of their improvement effort. 

The results dramatically exceeded the state's accountability benchmark for expected School Performance Score (SPS) growth, and the school won an "Exemplary Academic Growth Award" from the state.  In addition, the gains in the Iowa are an independent confirmation that the gains were real, and that the thinking development approach of HOTS, in combination with skill development produced gains in literacy, that transferred to gains in Math. (Iowa scores in the other content areas also increased.) This approach to improving literacy produced a real schoolwide academic improvement benefit.

back to table of contents



Title of Evaluation: A Study of the Effects of the HOTS program in Title I schools in Cleveland Country Schools District.

Author: Janet Walker, Director of Testing

Relation of author to model developer: None

Date evaluation report completed: April 23, 2002

Months/years covered by evaluation:  1 year

N: All of the Title I schools   

Demographics of sample population:  Low income Title I students.

Purpose of study:  To study the effects of the HOTS Program on student growth relative to state benchmarks for expected growth in Cleveland County Schools in North Carolina in grades 3-5.

Methodology: All Title I schools in Cleveland County School District using HOTS were compared to all other Title I schools not using HOTS, by grade level for grades 3-5.    

Measures/indicators used to assess effectiveness:  Growth in reading relative to the state's 'expected' and 'exemplary' growth benchmarks (in scale scores) on the North Carolina state test.  The state's benchmarks of desired growth take key demographics into account.

Findings:  The Table on the next page summarizes the results in reading for the Title I schools in the Cleveland County School District on the North Carolina state test administered in May 2001.  The state sets expected scale score growth standards, which takes into account demographic factors.  The state also sets exemplary growth targets.

The Table shows that all HOTS schools and grades not only exceeded state expected growth targets, but also exceeded exemplary growth targets, for both targeted and schoolwide use of Title I funds, at all grade levels where HOTS was used.

At each grade level using HOTS, the students exceeded state growth standards, both in terms of expected goals and exemplary growth goals.  In addition, the grades where HOTS was used produced greater growth relative to state standards than the same grades in non-HOTS schools—regardless of whether a targeted assistance or schoolwide model of Title I services was used.  Indeed, for the grade at Washington where HOTS was not used the students under performed state standards, and for the grade in which it was used it exceeded state growth standards.

One-Year Growth in Reading Relative to State Scale-Score Growth Standards for the North Carolina Test Administered May 2001  

HOTS Schools/Grades

 

Non-HOTS Schools/Grades

 

School/

Type of Title I

School Gains vs. State Growth Targets

School/

Type of Title I

School Gains vs. State Growth Targets

Expected

Exemplary 

Expected

Exemplary 

Grade 3

Casar/

Targeted.

+2.65

+1.98

Township3/

Schoolwide

-.94

-1.61

Fallston/

Targeted

+0.97

+0.3

Union/

Schoolwide

-1.12

-1.79

Washington/

Schoolwide

-0.75

-1.42

Grade 4

Fallston/

Targeted

+1.29

+0.91

Township3/

Schoolwide

+0.86

+0.48

Union/

Schoolwide

-0.41

-0.79

Washington/

Schoolwide

-1.64

-2.02

Grade 5

Union/

Schoolwide

+1.46

+1.08

Township3/ Schoolwide

+0.15

-0.22

Washington/ Schoolwide

+1.43

+1.06

Average gain across all grades relative to state growth standards

+1.56

+1.07

-.55

-1.05


In all cases the grades with HOTS exceeded State accountability growth standards, for both expected and exemplary growth, and for both targeted assistance and schoolwide models.  The equivalent grades in Title I schools not using HOTS generally failed to meet growth standards, and under performed the HOTS schools in terms of growth in reading.

This suggests that the use of the HOTS program has schoolwide effects on those grade levels where it is used in helping a school meet state benchmark accountability growth standards.

back to table of contents



Title of Evaluation: A Study of the Effects of the Metacognitive Intervention HOTS on Educationally Disadvantaged Students

Author: Dr. Mary Ann Darmer

Relation of author to model developer: Dissertation student

Date evaluation report completed: 1995

Months/years covered by evaluation:  1 year

N:  The experimental group consisted of 28 fourth graders and 25 fifth graders.  The control group consisted of 24 fourth graders and 25 fifth graders in Elvira Elementary in Tucson Unified School District.  There was no statistical difference in the pre-test reading scores of the two groups.

Demographics of sample population:  Low income native American and Hispanic students.

Purpose of study:  To study the effects of the HOTS Program on a wide variety of educational measures

Methodology: HOTS students in grades 4,5 were compared to a group of students from the same classrooms who got an alternate Title I intervention.

Measures/indicators used to assess effectiveness:

This study is that it was the first to use a wide variety of measures of cognitive development to more accurately and comprehensively assess how widespread the benefits of the HOTS program are.  Previously, most studies had collected data primarily on standardized test scores.

Dr. Darmer studied the effects of HOTS training on the following outcomes:  a) Metacognition, b) reading comprehension, c) grade point average, d) general intelligence, e) writing, and f) novel problem solving tasks.  Metacognition was measured using the questionnaire developed by Lee Swanson (internal consistency of .92).  Reading comprehension was measured using the Nelson Reading Comprehension Test.  General intelligence was measured using five selected scales from the Woodcock Johnson Cognitive Abilities Test.  Writing was measured using student writing samples (holistic scoring with an inter-rater reliability of .93).   Novel problem tasks were developed by the researcher (Pre and Post test).

      The following table summarizes the scales and methodology used in this study.

Instrument

Measure/Scale

Group Tested

Type of Measurement

Metacognition

Knowledge of:

Problem solving

Person Variables

Reading Strategies

Task Awareness

Experimental & Control

Post test only

(Spring)

Nelson Reading Comprehension

Reading Comprehension

Experimental & Control

Pre and Post test

(Spring to spring)

Grades

GPA

Experimental & Control

Pre and Post

(Spring to Spring)

Woodcock Johnson

Listening Comprehension

Concept Formation

Verbal Analogies

Spatial Relations

Analysis-Synthesis

Experimental

Pre & Post test

(Fall to spring)

Novel problem tasks

Transfer

Schema Usage

Strategy Development

Experimental

Pre and Post test

(Fall to spring)

Writing

Writing

Experimental

Pre and Post test

(Fall to Spring)

The result was a total of 15 scales, and 30 group comparisons (4th and 5th grade was compared separately).

Findings:

Results were compiled for the period Spring 93 to Spring 94.

First year HOTS students increased substantially and significantly in all 22 pre-post comparisons.  HOTS students significantly and substantially outperformed the control Title I students in all 12 comparisons between the groups.

While space does not permit reporting all the data, the following two tables illustrate the significant differences between the groups:

A COMPARISON OF THE GROWTH IN READING COMPREHENSION

Grade

Group

n

Pre

Post

Gain (NCE)

SD

t

(post-test) *

p

Fourth

HOTS

28

33.93

46.71

12.78

17.59

Control

25

31.42

34.37

2.95

17.47

2.78

.0078

Fifth

HOTS

24

30.60

44.72

14.12

12.39

Control

25

30.08

27.84

-2.24

7.02

4.73

.0001

* There was no statistical difference on the pretest

The gains for the HOTS students are substantially larger than in the original study.  In addition, gains for the HOTS students are noteworthy given the fact that the scores of fifth graders in the same school are declining.

A COMPARISON OF THE GROWTH IN GPA *

Grade

Group

n

Pre

Post

Gain

SD

t

p

Fourth

HOTS

28

77.43

86.86

9.43

6.50

Control

25

74.70

74.17

-.53

6.53

7.0

<.0001

Fifth

HOTS

24

68.90

76.93

8.03

5.63

Control

25

71.96

71.20

-.76

7.94

3.64

<.0007

* A gain of 10 points equals a gain of one letter grade 

Grade Point Average is an excellent measure of transfer for a specialized general thinking program.  Control students spent more time in the classroom than HOTS students, yet are declining in GPA.  The decline in control students mirrors conventional wisdom about how the longer students are in Title I the worse they seem to do.  Yet, HOTS students are defying conventional wisdom and are benefiting from additional service.  Why are HOTS students improving almost a whole letter grade? This phenomenon appears to validate Dr. Pogrow's belief that after the third grade the major inhibitor of learning is not prior content knowledge, but rather, an inability to process the more cognitively difficult tasks that occur after the third grade.  That is why the development of general thinking skills through a program such as HOTS has more payoff to the learning of classroom content than increased exposure to the content.   The increased thinking skills allow students to learn content the first time it is taught. 

The gains in GPA for HOTS students are substantial.  Indeed, by the end of the year, 18 of the HOTS students (35%) made honor roll.  In addition, 23 of the students entered the school's science contest.  

The thinking skills appear to translate directly into improved academic skills.  For example, while the first year curriculum has students do some writing, writing is not emphasized to any great extent.  At the same time, the table below shows that HOTS students showed significant improvement in writing (based on a rating rubric of 0-4).

   

GROWTH OF THE HOTS STUDENTS IN WRITING

Grade

Group

n

Pre

Post

Gain

SD

t

p

Fourth

HOTS

28

1.082

2.457

1.375

.571

12.74

p<.0001

Fifth

HOTS

24

1.06

2.52

1.460

.406

17.97

p<.0001

The transfer of the thinking skills into improved academic performance seems to be a result of overall intellectual development.  There is evidence of a general cognitive development transfer effect.  The table below shows the growth of HOTS students on solving novel problem solving tasks.  (The tasks on the post-test were different than those on the pre-test.) 

GROWTH OF THE HOTS STUDENTS ON NOVEL PROBLEM SOLVING TASKS

Grade

Group

n

Pre

Post

Gain

SD

t

p

Fourth

HOTS

28

64.64

97.10

32.54

16.022

10.72

<.0001

Fifth

HOTS

24

52.56

92.32

39.76

18.075

11.00

<.0001

And it does appear that HOTS students make a variety of intellectual development gains.  For example, the table below shows the growth for the Woodcock-Johnson scale of Synthesis-Analysis:

GROWTH OF THE HOTS STUDENTS ON THE COGNITIVE ABILITY SCALE OF SYNTHESIS-ANALYSIS

Grade

Group

n

Pre

Post

Gain

t

p

Fourth

HOTS

28

90.67

107.99

17.32

8.37

<.0001

Fifth

HOTS

24

88.80

107.60

18.80

8.37

<.0001

This pattern of significant growth was consistent across all the other four Woodcock-Johnson scales of cognitive ability used in this study.

This growth in general thinking ability is also mirrored, for example, in the relative performance of the groups in the metacognition scales of the instrument developed by Lee Swanson.  For example, in the Knowledge of Task Awareness scale, The table below indicates that the HOTS students had significantly higher post-test scores than the control students.

A COMPARISON OF STUDENTS ON THE METACOGNITIVE SCALE OF KNOWLEDGE OF TASK AWARENESS

Grade

Group

n

Post

SD

t

p

Fourth

HOTS

28

81.21

9.34

Control

24

62.50

11.70

6.46

<.0001

Fifth

HOTS

24

79.52

7.48

Control

25

57.00

12.00

7.91

<.0001

This pattern of relative performance of HOTS versus control students on the three other scales of metacognition is the same as in the above table, with a substantial and significant advantage for the HOTS students.

Dr. Darmer has begun analysis of a second year of data that includes a new cohort of first year students, and the second year performance of the original cohort of students.  Preliminary results show that HOTS students continue to make progress in the second year, although at a much slower rate of growth, while control students continue to decline.  In addition, the preliminary analysis of results from the second cohort of first year students appear very similar to those of the first group reported in this study.

This study shows that HOTS provides a major advantage on a wide variety of measures, and consistently produces a wide variety of substantial gains.  The gains in GPA, performance on novel problem-solving tasks, and in writing make a convincing case that HOTS activities produce a powerful form of transfer.

back to table of contents


Title of Evaluation:  The Effectiveness of HOTS On The Texas TAAS Test

Author: HOTS teacher and school principal

Relation of author to model developer: HOTS teacher and school principal.

Date evaluation report completed: 1996

Months/years covered by evaluation: 1 year, 1994-95

N: 49 students in Williams Elementary in San Antonio

Demographics of sample population:  Mostly low income Hispanic students

Purpose of study: To determine whether HOTS was effective.

Methodology: A comparison of the percentage of Title I fifth graders in Williams Elementary passing TAAS a year after the school adopted the HOTS program.

Measures/indicators used to assess effectiveness: Percent of Title I students passing TAAS test.

Findings:

A COMPARISON OF FIFTH GRADERS PASSING THE TEXAS TAAS TEST IN WILLIAMS ELEMENTARY SCHOOL BEFORE AND AFTER HOTS

1994 (Pre HOTS)

1995 (HOTS)

Percent passing

Percent passing

Reading

8

47

Math

38

61

The results show a sharp rise in the percentage of Title I students passing TAAS after the implementation of HOTS.

In addition, classroom teachers reported that approximately 80% of HOTS students had made improvement in the classroom over the course of the year.

back to table of contents


Title of Evaluation:  The Performance of HOTS Students in Mann Middle School

Mann Middle School is one of the lowest SES schools in San Diego Unified School District and has 80% free and reduced lunch.

Author: Part A which contains the 1992-93 data were compiled by the Office of the Deputy Superintendent for the San Diego City Schools, and Part B, equivalent data for 1993-94 were compiled by one of the HOTS teachers at the school.

Relation of author to model developer: Part A compiled by the District Office, no relation, Part B no relation other than being a HOTS teacher.

Date evaluation report completed: 1995

Months/years covered by evaluation: 2 years, 1992-94

N: 141 students in Horace Mann Middle School in San Diego Unified

Demographics of sample population:  Low income mostly minority students

Purpose of study: To determine whether HOTS was effective in improving overall academic performance.

Methodology: Analysis of the academic performance of HOTS students.

Measures/indicators used to assess effectiveness: Percent of Title I students passing TAAS test.

Findings:

PART A:   Results from this school confirm the effects of HOTS on transferring to academic gains.  In 1992-93, 14% of the 65 seventh graders in the second year of HOTS had an A average, 49% had a B average or better, and 83% had a C average or better.  15% made the honor roll.   Only one of the students was failing.

PART B:   The results were even better in 1993-94.  21% of the first year students (sixth grade) had an A average, and 33% made honor roll.  23% of the second year (seventh grade) HOTS students had an A average, and 29% made honor roll.  In addition, 16 students were recommended for being tested to determine their eligibility for placement into the Gifted program, and 4 HOTS Title I students were placed in Gifted programs. 

back to table of contents


Title of Evaluation: An evaluation of the Effectiveness of the HOTS program

Author: Evaluation Office, Detroit Public Schools

Relation of author to model developer: None.

Date evaluation report completed: 1994

Months/years covered by evaluation: 1992-93

N: 368 experimental students, 175 control Title I students.

Demographics of sample population: 368 low income, primarily African American, students in grades 4-5. 

Purpose of study: To determine if the district should continue to use the program.

Methodology:  The reading scores of HOTS Title I students were compared to those of Title I students in other programs in demographically similar schools on the California Achievement Test from Spring 92 to Spring 93. 

Measures/indicators used to assess effectiveness:   The study is based on results from the California Achievement Test from Spring 92 to Spring 93.  (The control group had statistically significantly higher pre-test scores in both reading and math.)

Findings:

A COMPARISON OF THE GROWTH OF STUDENTS IN DETROIT IN READING 

SUBJECT

GRADE

n

GROWTH

(In Grade Mean Equivalents)

HOTS

Control

HOTS

Control

Reading

       4

       5

245

123

107

68

.9

1.6

.5

.9

HOTS students consistently made significantly greater progress, almost twice as much growth, except for fifth grade math where both groups made substantial gains.  (If reading comprehension had been reported separately, the difference in reading growth in favor of the HOTS students would probably have been even greater.)

It is impossible to compare the amount of gains for HOTS students in this study as compared to the Detroit study in the original validation study since the earlier study used a criterion referenced test instead of the current norm-referenced test.  However, the percentage difference in favor of HOTS students in reading in this study on a norm-referenced test is consistent with the overall findings from the original study (i.e., approximately twice the gain).   

The Detroit study also found that interviewed teachers felt that the thinking skills of the HOTS students had definitely improved.  The conclusion of the study was that the HOTS program "...should be continued and expanded."

back to table of contents


Title of Evaluation:  The Effectiveness of HOTS In Soldotna Alaska

Author: Sam Bushon, Title I coordinator

Relation of author to model developer: None

Date evaluation report completed: 1993

Months/years covered by evaluation: 2 years, 1990-92

N: 30 HOTS Title I students in Soldotna Alaska

Demographics of sample population:  Low income native students

Purpose of study: To determine whether HOTS was effective.

Methodology: A comparison of gains in reading Spring to Spring on the ITBS of HOTS Title I fifth and sixth graders in Soldotna Alaska and Title I students in an alternate Title I intervention in demographically similar schools.

Measures/indicators used to assess effectiveness: Percent of Title I students passing TAAS test.

Findings:

A COMPARISON OF THE GROWTH OF STUDENTS IN SOLDOTNA ALASKA IN READING (1990-92, Spring to Spring) ON THE ITBS TEST

SUBJECT

GRADE

n

GAINS

(NCEs)

HOTS

Control

HOTS

Control

Reading

       5

       6

17

13

17

12

8.22

6.15

4.41

2.25

  

HOTS students made approximately twice the growth of control Title I students.

back to table of contents