Halstead Category Test

The Category Test was developed by Ward Halstead in the 1930s and 1940s¹⁾. It assesses the ability of abstract concept formation. The task consists of finding an organizing principle according to which a number between one and four can be assigned to four simple geometric patterns displayed side by side. The concept must be discovered freely. If the example item in Figure 1 were the first in a subtest (all items in a subtest follow the same concept), one could respond with the number 2, because the second square, unlike all the others, is blue. The number 2 could also be correct because the concept is given by the number of large squares. Of course, it would also be conceivable that, quite the opposite, it is the number of small squares. One must therefore guess at first. But even if the response (here the response “2”) is correct for this picture, only the application of the conceived concept to subsequent pictures clarifies whether the presumed underlying concept is actually the right one.

In Halstead's original version, the pictures were presented to the examinee by an apparatus. As a response, the examinee pressed one of four buttons, which was acknowledged either by the sound of a bell (meaning correct) or by a buzzer (meaning incorrect). No interaction with the test user, who at the time still operated the apparatus by hand, was supposed to take place. Halstead's original version was revised by Reitan²⁾ in the early 1950s and, in this form, constituted one of the most important subtests of the Halstead-Reitan Neuropsychological Test Battery. Until the 1990s, most HCT assessments were probably administered with this version.

Figure 1: Example item of the HCT

As a less expensive alternative to the unwieldy original apparatus, paper-and-pencil versions of the HCT emerged in the 1970s and 1980s, of which the one by DeFillippis and McCampbell (1991)³⁾ has become the most widely adopted internationally. One problem of the paper-and-pencil versions is that the interaction with the test user is necessarily more intensive than with the apparatus or computer versions. Over the years, a number of short forms were also published, none of which has become established.

In 2007, Fast and Engel⁴⁾ published a computer version for the “Hogrefe Test System 4”. It is now out of print. In terms of the administration situation, this version was similar to the original version. The accompanying manual dealt extensively with the concept and background of the HCT. Another computer version is offered by the “Neuropsychology Center (NPC)”. The NPC is a company in Texas that has for some time reissued all procedures of the classic Halstead-Reitan Neuropsychological Battery and distributes them worldwide.

Halstead's Category Test is also discussed in all the more important English-language textbooks of neuropsychological assessment ^{5) 6) 7)}.

In the first version of the test, the Category Test consisted of nine subtests with 336 items ⁸⁾. The version customary today, revised by Ralph Reitan at the beginning of the 1950s, has seven subtests with 208 items ⁹⁾. Almost all of the more frequently used versions of the HCT consist of these 208 items, including the German version of the test by Fast and Engel ¹⁰⁾.

The copyright situation is somewhat unclear. There is still a website of the “Reitan Neuropsychology Laboratory”, even though it seemed to be abandoned for some time in 2012. Among other things, it points out that it is important to work with the original materials, with which most of the norming studies were also conducted. However, no copyright was explicitly claimed for the materials. It would presumably also have expired, because the original pictures were created in the 1930s. Other versions of the HCT, for example the booklet version published by PAR in Florida, are themselves copyrighted. For the German version in the Hogrefe Test System 4, the HCT pictures were recreated from the old original templates. For this realization of the procedure, the copyright is held by the publisher Hogrefe.

Most studies in the literature were probably conducted with one of the “HCT apparatuses” from Reitan's Neuropsychological Laboratory, at least the older ones. The German computer version by Fast and Engel ¹¹⁾ comes very close to this version. What they have in common is the mechanical form of feedback to the examinee, in which a pleasant tone signals “correct” and an unpleasant noise signals “incorrect”.

With the paper-and-pencil variants, the test administrator must take over this task. In practice, this is not so easy: Especially patients who are not so fit naturally always ask after an “incorrect” response and want to know what was wrong with the response. The test user must not answer such questions. The test user may only repeat the general instruction (“… You are to find out which number this picture represents…”). With verbal feedback from the test user, this restricted response behavior in practice generates more reactance on the part of the patients than when a machine does it.

Otherwise, the administration is the same: all items from 1 to 208 are presented. The items are divided into 7 subtests, each of which again has its own subtest instructions.

The apparatus, computer, or test administrator records the response for each item. There is only one test raw score, namely the number of errors across all 208 items.

Young healthy adults make on average between 20 and 30 errors on the HCT. Theoretically, the number of errors with 208 items could range between 0 and 208; in practice, the spread is more restricted. The lower end of the error distribution results from the fact that one can hardly make fewer than 8 errors. For the first items of a series, one must guess, because one cannot know the concept a priori. At the upper end of the error distribution, the guessing probability sets a certain limit. If an examinee guesses throughout, one could expect (as a mean across several such guessing exercises) 52 correct responses (208/4) and accordingly 156 errors (208-52). Even though such a result is extremely rare in practice, error counts of around 100 do occur, especially in old examinees.

In the higher-performing half of the examinees, the errors thus lie within a very narrow range of variation, say between 8 and 25. In the lower-performing half, by contrast, the variability is much greater, ranging between 25 and 100. Such boundary conditions must lead to a positively skewed distribution of the raw scores.

Empirical data on this are available in the collection of norms by Heaton et al. (2004)¹²⁾. It compiles the results of 634 White and 578 Black Americans aged between 20 and 85 years on a series of tests, including the HCT. The authors converted the raw scores via an percentile-to-normal-transformation (across all persons in the sample) into scaled scores (mean 10, standard deviation 3) and presented the result as a table of values (raw score to scaled score). From this, the original distributions of the data at the level of errors could be back-calculated (after interpolation and smoothing of the coarse scaled score scale). Figure 2 shows the raw score distribution back-calculated in this way across the total sample of 1212 persons.

Figure 2: Frequency distribution of the HCT raw scores in the total sample (N=1212) of Heaton et al. (2004)

As expected, the raw score distribution is quite positively skewed and therefore cannot be interpreted well without a nonlinear transformation. Two examples are intended to illustrate this:

• A direct consequence of skewed distributions is the unpleasant property that equal differences at different points of the scale mean different things. The difference between 15 and 20 errors is more serious in terms of content than that between 75 and 80 errors. The raw score scale, however, does not reflect this.

• Skewed distributions have another unpleasant property. In arbitrary subgroups that differ in performance, for example age cohorts, the standard deviations are not equal but correlate positively with the means. This, too, can be shown well with the data of Heaton et al. (2004). In Figure 3, the raw score distributions back-calculated from the percentile tables are contrasted for White women and men with 12 years of schooling in two age groups on the HCT, the age group 20 to 34 years in blue, the 60-64-year-olds in red. It can be seen at a glance that the distribution is wider in the older examinees than in the younger ones, corresponding to the higher mean. For the most part, however, this widening is not due to a greater variability of the trait itself but to the properties of the error scale, which depicts the psychological trait in a distorted way because it has a fixed limit on one side (zero errors) and is widely open on the other side (limit 208 errors), while the mean lies fairly close to one of the sides. This is more noticeable in the younger examinees, who on average perform better, than in the older ones.

Figure 3: Back-calculated frequency distributions of the HCT raw scores for two age groups (data from Heaton et al., 2004)

With the HCT, there was the rare opportunity to draw on empirical data for the linearization, albeit only in summarized form and with certain compromises. In the large study by Heaton et al. (2004)¹³⁾ already mentioned above, a percentile-to-normal transformation of the raw error scores into normally distributed standard scores had been carried out. Unfortunately, this had been done only across the total sample, which is not ideal, because in this heterogeneous sample, besides the scale influences of interest here, further effects are reflected that influence the shape of the distribution. In particular, these are effects of age, especially with a test as age-sensitive as the HCT. But at least, after interpolation and smoothing of the scaled score scale, a finely graded table of values could be created from the data, which could serve as the basis for the linearization of the performance scores.

For the calculation of the performance scores, however, the data of a young healthy sample are needed. These are not presented separately in the manual by Heaton et al. (2004). The authors had chosen to use polynomial regression to calculate the influences of age, sex, and years of education on the raw scores. The results are tabulated in the form of age-, sex-, and education-corrected T scores. The length of education of the total sample of Heaton et al. (2004) deviated markedly upward from the American average. For the calculation of the performance scores, therefore, only the subsample with 12 years of education was used (N=114). At that time, this corresponded to the mean length of education of the American population. Owing to the relatively coarse classification, a broader range formation would not have been possible. This would have required accepting a large deviation from the average, which was judged to be more serious. As presented below, the performance of the subsample chosen here corresponded relatively well to the performance of the meta-analytically pooled samples.

Halstead's Category Test has been in use for around 60 years. Like many classic neuropsychological tests, it was published at that time without a sufficient normative data basis. The original publication gave only an upper cutoff score for the normal range of the number of errors. For the form with 336 items used initially (Halstead, 1947)¹⁴⁾, a cutoff score of 80 was given, which was later transferred, purely arithmetically, to the version with 208 items in use since about 1950 ¹⁵⁾ and led to a cutoff score recommendation of 50. Given the high correlations of the CT with intelligence, education, and age, this was always far too coarse, even when one takes into account that the further interpretation of the test results took place against the background of clinical experience and mostly in conjunction with further psychological test examinations.

Over time, for the HCT, many book and journal contributions published data from clinical groups as well as from healthy examinees – an advantage that the HCT likewise shares with classic neuropsychological procedures (for example with the Trail Making Test, the Rey Figure, word list memory tests, and word fluency tests). For the HCT, one can by now draw on a considerable number of examinees of all age groups.

Clinical psychologists in their practical work often use a single one of these norm sources. The choice probably follows pragmatic considerations (familiarity, availability) rather than systematic ones. The systematic overviews in the standard neuropsychological works do not always help with the choice either. For example, in the fourth edition of the standard work “Neuropsychological Assessment” (Lezak, Howieson, & Loring, 2004)¹⁶⁾, three quite different normative datasets are cited with references, without any preference of the authors for one of them being discernible. In Spreen & Strauss' (1998) “A Compendium of Neuropsychological Tests” ¹⁷⁾, the presentation of the norm situation is even more confusing. On the one hand, the norming of healthy persons by Heaton et al. (1991) ¹⁸⁾ is mentioned, but the sample size (N=486) is dismissed as “very small”. On the other hand, the data collected from 112 patients of various clinics by Alekoumbides et al. (1987) ¹⁹⁾ are reproduced ready for use on three book pages, without even mentioning the sample size and the origin of the data.

For the construction of the TDB2 norms of the HCT, a less eclectic strategy was chosen. First, studies containing HCT data of healthy examinees were systematically sought. Since the HCT is a largely language-free test, no restriction was imposed on the national and/or linguistic origin of the normative data. The condition for inclusion of a study was that means and standard deviations of the raw error scores of a sample of healthy examinees were reported in one or more differentiated age ranges. Ten studies could be located that gave normative data of 1622 examinees aged between 15 and 84 years. By means of a meta-analytic summary, these norms were integrated and thus made more independent of the peculiarities of the individual studies.

For the norms of the HCT calculated here, there were two large reference works. One is a collection of normative data for a series of standard neuropsychological procedures, published as a monograph under the title Handbook of Normative Data for Neuropsychological Assessment (Mitrushina et al., 1999)²⁰⁾. For the HCT, this handbook discusses a total of 19 studies with normative data. Six of them, namely those cited as studies 5²¹⁾, 6²²⁾, 9²³⁾, 12²⁴⁾, 13²⁵⁾, and 18²⁶⁾, met our inclusion criteria. From study 5, only the two groups in the IQ range between 89 and 112 were included. Reasons for exclusion of the other studies were: an age range that was too wide and undifferentiated in studies 1, 2, 3, 7, 11, 14, 16, 17, and 19 (references in Mitrushina et al., 1999) and the inclusion of ill persons in the samples of studies 4 and 8.

A special case is formed by studies 10²⁷⁾ and 15²⁸⁾. Both were not included because they are precursor studies to a more recent norming manual²⁹⁾ that incorporates the older data. The revised version of this norms manual³⁰⁾ contains detailed, statistically processed data of a sample of 1212 healthy US citizens who completed almost all procedures belonging to the Halstead-Reitan Neuropsychological Test Battery and further intelligence tests. This manual plays a special role for the psychometric processing of the HCT data. The linearization of the scale is based exclusively on the data of this study. In the meta-analysis, however, only a subsample of this study, namely the White US Americans with 12 years of education, could be included, because the sample had a very high level of education. Even this subsample, with 320 persons aged between 20 and 84, still makes the second-largest contribution to the HCT norming. Also important is the aspect that the manual also contains the results of further tests, including the Trail Making Tests, for which it likewise makes an essential contribution to the norming. Since the sample of Heaton et al. (2004) is the only one whose values were available as a percentile distribution and not only as a mean and standard deviation that had to be normalized afterwards by simulation calculations, it enters the overall norming with the same contribution as the remaining studies together.

Three further samples that are not listed in Mitrushina et al. (1999) were also included in the meta-analysis:

1. In a smaller norming study³¹⁾, 101 healthy examinees aged between 18 and 85 years were assessed with the HCT and other neuropsychological procedures. Despite the relatively small sample size, this study was distinguished by the fact that age, education, and sex of the examinees largely corresponded to the population census.
2. Elias et al.³²⁾ reported in 1993 on age and sex effects on neuropsychological test performance on the basis of a sample of 427 healthy examinees aged between 15 and 74. A critical remark on this study is that the level of education was in the range between 12 and 19 years and thus certainly higher than in most other studies.
3. A smaller study³³⁾ reported HCT results of 51 healthy college students for the narrow age range between 17 and 34 years, thus also a study with a relatively high level of education.

A further norm source for the HCT would actually also be the Halstead-Russell Neuropsychological Evaluation System (HRNES-R;Russell & Starkey, 2001).³⁴⁾ Over many years, Russell and coauthors compiled a group of around 200 persons, mostly men, mean age 48 years, who were examined on suspicion of neurological disorders but in whom no deficits could be demonstrated even after careful examination. In the HRNES-R, this group is referred to as the control group. Unfortunately, the manual does not contain the original HCT raw scores for this group. Instead, “expected values” can be looked up for seven age groups and four education or IQ groups, which, however, were calculated by linear regressions. In the middle age and IQ range, the values correspond to the mean of the other studies included in the meta-analysis. In the high age range, however, the means are markedly lower, which is probably due more to the regression analysis, which (wrongly) assumes homoscedasticity. By including these norms, one would have imported more artifacts, at least in some ranges, which is why they were not included. Lezak et al. (2004)³⁵⁾ in particular also dispute the status of the examinees of Russell & Starkey (2001)³⁶⁾ as “normal examinees”, which, however, has been rejected by Russell on several occasions ³⁷⁾³⁸⁾.

Table 1: Overview of the norming studies pooled meta-analytically

A total of 1622 persons were included in the meta-analytic norming of the HCT. Table 1 gives some details of the studies. It can be seen that even the total sample still has a relatively high level of schooling. For the first 9 studies, only means and standard deviations per age group were available. For Heaton et al. (2004), the reported data were more extensive, which is why they were given higher weight in the meta-analysis.

In the second edition of the neuropsychological data handbook by Mitrushina et al. (2005), meta-analyses of existing norm sources in the literature can be found for many neuropsychological test procedures. The authors' approach differs from that in TDB2Online mainly in two points:

1. The meta-analyses by Mitrushina et al. are regression-based: Means and standard deviations of subsamples that can be characterized by age, sex, and years of education are integrated with regression analyses. If effects of the demographic variables are found, means and standard deviations are estimated for subgroups. Effects are almost always found for age; here the values are given in five-year age intervals if the data allow. For other demographic variables, effects are found less often.
2. All analyses take place at the raw score level; the distributions of the raw scores are not taken into account further.

For the HCT, 22 studies were described and 21 of them summarized in a table. 11 studies were included in the meta-analysis (it was not clear to me which ones these were). The age of the examinees had a strong effect on means and standard deviations. For five-year groups, means and standard deviations were estimated by regression analysis and the results were given as an equation and as a table. Since the table contains a printing error (one age group is missing, so the results are shifted), the expected values for the presentation here in the wiki were estimated with the help of the equations.

Figure 6 shows the results of the meta-analysis by Mitrushina et al. (2005) in a form that allows a comparison with the TDB2Online norms. For each age group, the raw scores at the points mean minus two sigma, mean minus one sigma, mean, mean plus one sigma, and mean plus two sigma were calculated from the predicted means and standard deviations and plotted in the TDB2Online performance score grid.

Figure 6: Results of the meta-analysis by Mitrushina et al. (2005) as means plus/minus one and two sigma, plotted in the TDB2Online performance score grid.

Looking only at the means, there is no serious difference between the norms of Mitrushina and those of TDB2Online. Across the entire age span, the means move from a performance score of 100 in the young to about 70 in the old, in both data treatments. This corresponds to raw scores of about 27 to 80. Even though differences of up to 10 raw score points are found in the middle age groups, the means overall are stable enough to cope with regression statistics.

It becomes problematic, however, when one considers the standard deviations, which are needed for the scaling of the age-normed scores. Here the raw-score-based analytical approach of regression analysis quickly reaches its limit. In all age groups, the values for very good performance (mean plus two sigma) are too high, in the younger age groups under 30 years much too high: The attempt to calculate them leads to negative raw scores. A mean of 27 and a standard deviation of 16 do not go well together when the possible raw score range ends at a raw score of about 8. Given the skewness of the HCT raw score distribution, the formal application of parametric statistics at the raw score level is not meaningful. The effects take place mainly in the outer ranges of the distributions, less around the mean.