On the triangle test with replications.pdf

Food Quality and Preference 10 (1999) 477±482

www.elsevier.com/locate/foodqual

On the triangle test with replications

Joachim Kunert*, Michael Meyners

Fachbereich Statistik, University of Dortmund, D-44221 Dortmund, Germany

Accepted 9 May 1999

Abstract

We consider the triangle test with replications. A commonly used test statistic for this situation is the sum of all correct assess-

ments, summed over all assessors. Several authors argue that the binomial distribution cannot be used to analyse this kind of data.

Brockho and Schlich [Brockho, P.B., & Schlich, P. (1998). Handling replications in discriminations tests. Food Quality and

Preference, 9, 303±312.] propose an alternative model for the triangular test with replicates, where the assessors have dierent

probabilities to correctly identify the odd sample even if the products are identical. Although we agree that assessors will have

dierent probabilities of correct assessment if there are true dierences, we do not think that Brockho and Schlich's model makes

sense under the null hypothesis of equality of treatments. We show that all assessments are independent and have success prob-

ability 1 = 3, if the null hypothesis is true and the experiment is properly randomized and properly carried out. This implies that the

sum of all correct assessments is binomial with parameter p=1 = 3. Therefore the usual test based on this sum and the critical values

of the binomial distribution is a level a test for the null hypothesis of equality of the products, even if there are replications. # 1999

1. Introduction

i.e. the sample that diers from the other two is identi®ed.

The number x i of correct assessments of the ith assessor is

calculated, and these numbers are added over the asses-

sors to get the number x of all correct assessments. Then

in this naõÈ ve approach x is compared to the critical value

of the binomial distribution with parameters m and 1 = 3.

It has been argued that the binomial distribution is

not adequate for the evaluation of such triangle tests

with replications (see e.g. Brockho & Schlich, 1998; or

O'Mahony, 1982). If an assessor is able to perceive the

dierence between the products and therefore gives a

right answer once, then he will most likely perceive it

again in a second replicate. If, however, another asses-

sor is not sensitive enough to perceive the dierence

between the products in one trial, then he will most

likely not perceive it in a second trial. Therefore the

assessors have dierent probabilities of successes and

Because they are carried out easily and provide a sim-

ple and straightforward analysis, triangle tests are widely

used in sensory analysis. In a triangle test, an assessor is

presented with three samples which come from two pro-

ducts. Two of the samples are from the same product, the

third sample is from the other. The assessor is asked to

identify which is the odd sample. He/she is asked to make

a choice, even if no dierence is perceived.

With triangle tests, many observations may be needed

to get suciently high power to show signi®cance if

there are only small dierences between the products.

There may be not enough assessors available to have the

desired number of assessments. Then it is convenient to

let each assessor test repeatedly. Such experiments are

commonly analysed as if there were no replications, i.e.

as if all assessments came from dierent assessors.

Using the notation of Brockho and Schlich (1998) let n

denote the number of assessors each of which per-

formed k replications. If m denotes the number of

assessments, then m=n k. We say that the assessor had

a success in a given replicate, if the right answer is given,

P i x i is not binomially distributed.

For discrimination tests with replications, Brockho

and Schlich (1998) therefore propose to adjust the

number m of observations according to some variability

criterion. This criterion depends on the overdispersion

observed in the data. The larger the overdispersion is,

the more the number of observations gets reduced.

We show, however, that the naõ È ve binomial test can

also be used in this situation. Under the null hypothesis

of equality of products and under proper randomization

* Corresponding author.

E-mail addresses: kunert@statistik.uni-dortmund.de (J. Kunert)

meyners@statistik.uni-dortmund.de (M. Meyners)

PII: S0950-3293(99)00047-6

478

J. Kunert, M. Meyners / Food Quality and Preference 10 (1999) 477±482

of the design the number of correct assessments is

binomial with success probability 1 = 3. This implies that

(under the null hypothesis) the observations can be

treated as if they were all produced by dierent asses-

sors and there were no replications. Therefore, the naõ È ve

test which compares the number of correct assessments

to the 1 ÿa critical value of the binomial distribution

with parameters m and 1 = 3 is a level a test for this null

hypothesis.

For the situation that there are dierences between

the products, we suggest an alternative model, which is

a variant of Brockho and Schlich's (1998) model. If the

products are equal, then in our model all assessors have

the same success probability 1 = 3.

Our considerations indicate that Brockho and

Schlich's (1998) method is too conservative. We recal-

culate two of the arti®cial examples in Brockho and

Schlich which, we think, do give strong hints on product

dierences.

position that the assessor chooses, remains 1 = 3 for the

second trial, independent of the outcome of the ®rst

evaluation. Note that this holds if we do a third,

fourth, ... replicate, and it remains true both if we

always have the same assessor and if the assessor is

replaced by somebody else after some trials. It is only

necessary that there are no sensory dierences between

the products. Therefore, under assumption H we have

the following result: If there are m presentations and the

ordering of the products is randomized independently

for each presentation, then the total number x of correct

guesses follows a binomial with parameters m and

p=1 = 3. This remains true, whether or not we have

replicates. It is the number of assessments that counts.

Usually, we rely more on a result that was produced

by 100 assessors, each of which made one choice, than

on a result that was produced by just one assessor who

made 100 choices. However, a signi®cant result that was

derived from just one assessor also controls the type I

error, that our test might indicate sensory dierences

which are not really there. The null hypothesis implies

assumption H. Even if we have only one assessor and

he/she gives a correct answer in signi®cantly more than

one third of the assessments, then sensory dierences

have been proven.

The number of assessors gets important for the power

of the procedure. If there are 100 assessors, and only 35

of them correctly identi®ed the odd sample, then we

have good reasons to believe that the dierence between

the products is negligible. If we have just one assessor,

who correctly identi®ed in only 35 out of 100 replicates,

then we can only be sure that the dierence between the

products is too small for this assessor.

Whenever a sensory dierence is present between the

two products, then we can assume that there are ``good''

assessors, who do experience the dierence, and ``poor''

assessors who do not. Let us consider the extreme case

that exactly one half of our assessors will always give

the right answer, while the other half will only guess.

Assume that we do a test with signi®cance level 5% and

m =100 assessments, and assume there are two possible

ways to do the experiment.

2. Model assumptions

As a ®rst step, we assume that there is no sensory

dierence between the two products A and B. Then an

assessor has a certain strategy to decide which of the

three samples he selects as the odd one. This strategy

may be random or systematic. For our considerations,

there is one basic assumption: We assume that under the

null hypothesis of product equality, the response of the

assessor is independent of the order in which the pro-

ducts are presented. Call this assumption H (because it

is valid only under the null hypothesis). This assump-

tion is plausible, if the experiment is carried out prop-

erly. This includes, for instance, that the two products

presented are of equal appearance, temperature, etc.

The experimental design is a random process which

determines which of the six possible orderings AAB,

ABA, BAA, ABB, BAB, BBA is presented to the assessor.

Assume the process which determines the ordering is such

that each of the six possible orderings is presented with

equal probability. Due to assumption H, the probability

of a correct selection of the odd product then is 1 = 3.

Now assume a second presentation is made to the

same assessor. It is clear that the second choice of the

same assessor is not independent of the ®rst choice. It is

possible, for instance, that the assessor always changes

position, that is he chooses another position at the sec-

ond presentation. It is also possible that another assessor

may always select the same position.

However, we still have assumption H. Assume the

ordering for the second presentation is randomized

independently of the ®rst presentation, such that it gives

equal probability to all six possible orderings. Whatever

strategy the assessor might apply, under assumption H

the probability that the odd sample is placed on the

Case 1: We have just one assessor who gives 100

answers. Then we have probability 1 = 2 that this one

assessor is ``good'', in which case we will get 100 correct

answers. There also is probability 1 = 2 that he/she is

``poor'', which will lead to a number of correct answers

that is a binomial with parameters m=100 and p=1 = 3.

Therefore, the probability of a signi®cant result at the

5%-level is 1 if the assessor is ``good'' and 0.05 if the

assessor is ``poor'', giving an overall probability of 0.525

of rejecting the null hypothesis.

Case 2: We have 100 assessors each giving exactly one

answer. Then for each assessor, we have probability 1 = 2

J. Kunert, M. Meyners / Food Quality and Preference 10 (1999) 477±482

479

that he/she is good. With a good assessor the answer is

correct with probability 1. We also have probability 1 = 2

that the assessor is poor, in which case the answer is

correct with probability 1 = 3. In all, each assessor's

answer has probability 2 = 3 to be correct. Therefore the

number of correct answers is a binomial with para-

meters m=100 and p=2 = 3. This implies that there is a

probability of more than 99% to observe more than 42

correct answers. Since 42 is the critical value of the tri-

angle test with 100 assessments, we therefore have a

probability of more than 99% of correctly rejecting the

null hypothesis.

i . This distribution depends on the population of the

assessors, on the dierence between the products, etc.

The distribution of the i has to be modelled, it is not

determined by the experimental design and the rando-

mization. This is dierent from the distribution of x

under the null hypothesis.

For assessor i we assume p i to be constant during all

of his/her replications. This is reasonable only under the

restriction that the number of replications is small

enough that fatigue and learning eects can be neglected.

3. Comparison to Brockho and Schlich (1998)

After this arti®cial example, we try to ®nd a general

model for the alternative that there are product dier-

ences. We assume that there are two dierent groups of

assessors, those who are able to perceive the dierence

between the two given products and those who do not

perceive the dierence between these two products

because it is too small for them. Since the following

considerations also hold for other discrimination tests,

let some more general c be the probability to succeed by

chance (e.g. c=1 = 3 for the triangle test or c=1 = 2 for the

duo±trio test). Furthermore let p i be the probability for

assessor i to have a success, i=1, ... ,n.

The proportion of perceivers is denoted by , where

0 44 1. If there is no dierence between the samples,

then there are no perceivers, i.e. 0. In fact, we might

de®ne that no sensory dierence exists if and only if

For each perceiver the probability of a success

increases to a number p i i 1 ÿ i

Brockho and Schlich (1998) propose a model with

random assessor eects, i.e. we have

p i " i ; i 1 ; ...; n ;

c > c. Here i is

the probability that assessor i actually identi®es the odd

product and not only guesses. We might consider i to

be a random variable (if we assume that the assessors

are drawn from some superpopulation), but it is clear

that i > 0 because it is a probability. In the examples in

Section 4, we assume that all i 1.

We do not, however, generally assume that i 1,

because even a perceiver might miss the dierence with

some presentations, e.g. due to random variation

between the samples or positional eects. If the dier-

ence between the products increases, then as well as

the i will tend to 1.

Now assume that the assessors are drawn at random

from some superpopulation, such that each assessor has

a probability of to be a perceiver, and a probability of

1 ÿ to be a non-perceiver. Therefore the probability p i

of assessor i to succeed can be written as

where is the average probability of an assessor i to

succeed, and the average is taken over a population of

possible assessors. If there is no dierence between the

products, then is 1 = 3 for the triangle test. The random

variable " i has zero mean and an unknown variance. It

does not vanish if there is no dierence between the

products.

Note that with these assumptions, if there is no dif-

ference between the products, then since =1 = 3 there

will be some assessors i with p i < 1 = 3. That is, the model

of Brockho and Schlich (1998) implies that for some

assessors the probability of a correct result gets less than

what we would expect from pure guessing. We do not

think that this is reasonable if the data come from a

properly designed experiment: If there is no sensory

dierence between the products, how should an assessor

manage to systematically get the wrong sample? There

are two possible ways, both of which can be excluded by

the design of the experiment.

. First option: The assessor might ®nd out which

sample comes from which product by some other

means than the sensory dierence, for instance the

products are identi®ed in such a way that the

assessor can solve the code. It is clear that a prop-

erly designed experiment will exclude such possi-

bilities. We assume here that the experiment is run

in such a way that assumption H holds.

. Second option: The assessor has a strategy which

has a tendency to select a position where the

experimenter did not put the odd sample. Experi-

ence shows that most assessors have a preference

for a certain position, when they perceive no dif-

ference between the products. If the experimenter

has a tendency to place the odd sample preferably

on one of the positions that this assessor does not

prefer, then the assessor has a probability of less

p i c

with probability 1 ÿ

i 1 ÿ i

c with probability

where i is a realization of a positive random variable

less or equal 1. We do not specify the distribution of the

480

J. Kunert, M. Meyners / Food Quality and Preference 10 (1999) 477±482

than 1 = 3 of guessing correctly. This, however, can

be avoided by randomization. If the odd sample is

randomized to go to each position with equal

probability then, under H, each assessor will have

probability 1 = 3 of guessing right.

successes are not possible in Brockho and Schlich's

(1998) model, since in (2) each " i is a constant over the

replicates. Remember, however, that these correlations

only occur if the orderings for the trials are not rando-

mized independently.

Hunter (1996) suggests the number of replicates (if

any) always to be a multiple of six. He proposes to

randomize in such a way, that each assessor gets pre-

sented each ordering equally often. This is a special

instance of the method criticised above. We do not

consider this an appropriate randomization either. Only

with independent randomization, the independence of

the successes is guaranteed, provided the null hypothesis

is true.

We have here a basic property of designed experi-

ments that we think many experimenters do not su-

ciently appreciate. At least under the null hypothesis of

product equality, we can use randomization to intro-

duce a simple distributional structure into the data. The

idea of randomization was introduced by Fisher (e.g.

1935). The tool of randomization has been well exam-

ined under mathematical as well as practical aspects.

The philosophy of randomization is explained in e.g.

Bailey (1981).

Things are dierent if the null hypothesis is not true.

The way in which the dierences between the products

in¯uence the response has to be modelled. It cannot be

explained by randomization theory. We think that in

the case of the triangle test with replicates, a model as

the one described in Eq. (1) is reasonable. Therefore, for

power calculations we must take into account that there

are replicates and we must model the distribution of the

i . It is beyond the scope of this paper to deal with the

problem of how power calculation should generally be

done.

The aim of the paper is to point out that the naõÈ ve

test, which pools the number of correct guesses, pro-

vides a valid test to show that there are signi®cant dif-

ferences between products. This is true if the data were

derived from an experiment that was properly ran-

domized, and that was also properly carried out such

that assumption H can be justi®ed. Since we do not

know the power of the naõ È ve test, it cannot, however, be

used to show that there are no dierences between

products.

Let us return to the extreme situation that we have

only one single assessor. Assume this person gives sig-

ni®cantly more correct answers than that which is pos-

sible by chance. Corresponding to what we have said so

far, it is reasonable to decide that the two products

under consideration dier from each other. However, if

we have a group of assessors, then we cannot simply

take the assessor with the highest number of correct

guesses and do a triangle test based on his/her results

only. If we have e.g. 100 assessors all of which do 10

evaluations, then we can expect that there are about two

Presumably, the model of Brockho and Schlich

(1998) was intended to model correlations between the

responses of one assessor. It follows from their model

that if one assessor gave a correct answer in the ®rst

replicate, then he has a higher probability of giving a

correct answer in the second replicate. This is because

assessors who gave a correct answer have a higher

probability to have a large p i .

However, under assumption H, correlations between

the answers can be avoided by randomization. In fact,

correlations between the answers are indications that

either there are dierences between the products or that

a poor randomization has been used. An example of a

poor randomization is if the experimenter randomizes

just once for each assessor and uses the same presenta-

tion in every run. Then an assessor who has a tendency

towards a given position, and who guessed right in the

®rst replicate, has a higher probability to guess right in

the second replicate, too. He only has to stick to his

position. With independent randomization, however, the

fact that an assessor guessed right in the ®rst replicate has

no in¯uence on his chance to guess right in the second

replicate, whatever strategy the assessor might have.

There is another randomization which is used fre-

quently in triangle tests, but which might cause correla-

tions. Quite often, the randomization is not done

independently, but in a way to make sure that if an

assessor gets an ordering in the ®rst replicate, then he

gets another ordering in the second replicate. Such a

randomization is recommended in e.g. the ISO-standard

on the triangle test (ISO 4120, 1983). The ISO-norm

proposes to restrict the randomization such that each

ordering can only appear for a second time after all

other orderings have appeared at least once. Now

assume that an assessor uses the strategy to ``change

positions'', i.e. if he opted for one position in the ®rst

trial, then he will use another position in the second

trial. Then under the randomization proposed in the

ISO-standard, the result of the second replicate is no

longer independent from the outcome of the ®rst. To see

this, assume the assessor guessed right in the ®rst repli-

cate. He will choose another position in the second trial.

The experimenter will not use the ordering he had in the

®rst replicate, he will choose one of the ®ve other pos-

sible orderings. Since two of these have the position

chosen by the assessor, the probability to guess right

after a ®rst correct guess becomes 2 = 5>1 = 3. An assessor

who sticks to his position, however, will have a smaller

probability for a second success after a success in the

®rst replicate. Such negative correlations between the

J. Kunert, M. Meyners / Food Quality and Preference 10 (1999) 477±482

481

among them who have seven or more correct guesses,

even if there is no dierence between the products. The

analysis must pool all the assessors in the trial.

lead to the identi®cation of a dierence at the 10% level.

We now turn to Example 3 of Brockho and Schlich

(1998), with n=100 assessors, k=3 replicates and

x=112 correct answers. As these authors point out,

``Anna Sens'' wants to show that there is no dierence

between the products. The naõ È ve test gives a dierence

at the 10% level. As before we carry out a 2 -goodness-

of-®t-test to examine the data. The results are given in

Table 2.

Here the numbers are suciently large to do a 2 -test

of signi®cance. Then the result is highly signi®cant, the

corresponding 2 -distribution with 3 degrees of freedom

gives a p-value of less than 10 ÿ 5 . As in the previous

example this comes from the persons that guessed cor-

rectly in all replications. Once again, assume i 1.

Then if we subtract the four persons with three right

guesses that we should expect under equality of the

products, we estimate that there are nine consumers

who have really perceived the dierence in all four

trials. So we estimate that there is a dierence of the

products which is perceived by 9% of the consumers.

Note that this is totally dierent from the conclusion of

Brockho and Schlich (1998) who claimed that there is

no dierence between the products. In fact, we say the

data give strong hints that there is a perceivable dier-

ence between the products. It may be argued that 9% of

the consumers is too small a proportion for Anna Sens

to worry about. However, if the experiment was run

with 300 assessors, each of which is testing only once,

then 9% sensitive assessors would lead to a probability

of 80% to identify the dierence between the products.

So, obviously, 9% perceivers is a margin for which

Anna Sens has to expect a signi®cant result with an

experiment of this size.

Finally, we give an additional arti®cial example to

illustrate why we do not regard the method of Brock-

ho and Schlich (1998) as appropriate for dierence

tests in properly randomized experiments. We consider

a somewhat extreme situation. Let us assume two con-

sumers (i.e. n=2) that carry out k=100 replications of a

triangle test under ideal conditions, neglecting any fati-

gue eects etc. Suppose one of the consumers succeeds

4. Examples

We revisit Examples 2 and 3 of Brockho and Schlich

(1998). Both examples concern the triangle test, so again

c=1 = 3. Example 2 gives arti®cial data of a triangle test

with n=12, k=4 and x=24. The naõ È ve test therefore

gives a signi®cant dierence between the products at the

®ve percent level. We look at the data a bit closer, to see

why this is reasonable with data like this. There are 3, 2,

2, 2 and 3 assessors with 0, 1, 2, 3 and 4 correct answers,

respectively. Now assume that there is no dierence

between the products. We might want to carry out the

2 -goodness-of-®t-test to see whether the numbers x i of

correct guesses are binomial with parameters 4 and 1 = 3.

The computations are given in Table 1.

Note that the expected numbers in the cells are too

small to assume that the 2 -statistic is distributed

according to a 2 -distribution with 4 degrees of free-

dom. However, a calculated 2 -statistic of 57.13 is very

large. When we simulated the performance of 12 asses-

sors under the binomial distribution with p=1 = 3 there

was only one in 10,000 runs which produced a larger 2 -

statistic. It is obvious that the large size of the statistic is

due to the three persons that succeeded in all four

replications. If there was no dierence between the pro-

ducts, then we would expect less than one assessor with

four correct guesses in six experiments of this size. As

argued earlier we think that with proper randomization,

non-validity of the binomial distribution can only be

explained if the products dier from each other. For

simplicity, we assume all i 1, that is every sensitive

assessor, who can experience the dierence at least once,

gets it right in every replicate. Then an unbiassed esti-

mate of the proportion of sensitive assessors is (3 ÿ 0.15)/

12=23%. If we allowed for some of the i to be less than

1 we would estimate a higher proportion of perceivers.

The method by Brockho and Schlich (1998) does not

Table 1

Calculation of the 2 -statistic for the data from Brockho and Schlich

(1998), Example 2

Table 2

2 -goodness-of-®t-test for the data from Brockho and Schlich (1998),

Example 3

Number j of correct results

4 Sum

Sum

P j =Prob(x i j) 16/81 32/81 24/81 8/81 1/81

P j =Prob(x i j)

8/27 12/27 6/27

1/27

nP j

2.37 4.74 3.56 1.18 0.15 12

nP j

29.6

44.4

22.2

3.7

99.9

observed x i j

100

nP j ÿ observed 2

nP j

0.17 1.58 0.68 0.55 54.15 57.13

nP j ÿ observed 2

nP j

0.65

2.93

0.22 23.38 27.18

Plik z chomika:

Inne pliki z tego folderu:

Inne foldery tego chomika: