[ Pobierz całość w formacie PDF ]
As a result the chosen texts could become so general as to lose all trace of the very
specificity they were supposed to contain. Perhaps this meant that the whole notion
of a specific test should be abandoned and a return made to the situation in which
TOEFL (and the previous British Council test) had originated, that is a general test
of English proficiency, the same test of academic English for everyone (Alderson and
Urquhart 1985, discussed in Chapter 2 above).
The language problem presented by the British Council to its applied linguistics
consultants in the mid-1980s was what to do about the ELTS test. Should it remain
in use as it was, should it be revised, should it be rewritten? An evaluation was
needed, drawing on as wide a range of information as possible.
The detailed investigation of ELTS (Criper and Davies 1988 and see Davies 2006
for a recent overview) then carried out is described below; and in order to clarify its
relevance to our discussion of the enterprise of applied linguistics, I record my
discussion under the various heads or factors we have already noted.
6.1.1 Educational (including the psychometric) factors
ELTS was a proficiency test used in an educational setting, that is at entrance to a
British university. The educational questions that needed investigation were: to what
extent the test was being used at undergraduate and postgraduate levels, and for
temporary attachment by senior academics; and whether it was suitable at these
different levels; to what extent it was in use (and found suitable) in different faculties,
and whether it was (as had been intended) being used differentially so that test scores
deemed proficient in one faculty (or university) were rated not sufficiently proficient
in any others. Again, many universities had provision for remedial English tuition for
their foreign students; what was the extent of this provision and what use was made
by those involved of the ELTS results? Both sensitivity to the ways in which tertiary
institutions operated and an understanding of their role and complexity was called
for in the project.
In addition, there were other skills that were required. One of these was the
analysis of the ELTS test in terms of its relevance to the content of the specialist areas
it purported to test. For example, what connection, if any, was there between the test
materials in the life sciences module and the textbooks, lectures and so on in
university departments of biology, bio-chemistry, botany and so on? And how far
were these likely to be constant at the different levels, under- and postgraduate? For
02 pages 001-202:Layout 1 31/5/07 09:31 Page 77
Applied linguistics and language learning/teaching 77
analyses of this kind it was necessary to draw on the expertise of those skilled in
teaching life sciences at the different levels. The applied linguist could not be
expected to carry out the content analysis but he/she would expect to interpret the
results.
Another essential skill, which appears to be unique to the testing area but in
practice is relevant everywhere in applied linguistics research, is that of familiarity
with measurement (the psychometric aspect of educational studies) and with
elementary statistics, to which must also be added the skill of data entry and of
statistical analysis by computer. Those applied linguists who are consulted about
testing projects (as with this ELTS project) are likely to have all three skills, the
psychometric, the statistical and computer literacy. If not, then they must call on
colleagues or external consultants.
But for the purposes of understanding the working of any test these skills are all
essential since they provide the necessary evidence about the test s workings, often
called its validity and reliability. Using these skills it was possible to examine the
extent to which ELTS gave its users the decisions they needed, in particular whether
it enabled them to make the right choices of candidates, those with adequate English
for their studies.
It was also important to investigate whether there was even-handedness across
modules, so that one module (say the life sciences module) was no harder than any
other (say the social studies module). If it was harder, then it was possible it was right
that it should be, but this was a matter for empirical investigation: if the English of
life sciences is truly more difficult than the English of social studies, then it is
appropriate that the tests should mirror this distinction. If not, and one was in fact
more difficult than the other, then we had established a case of irrelevant bias which
should be corrected.
ELTS contained four parts, tests of the four skills: reading, writing, listening
and speaking. The results offered a profile score (skill by skill) and a global score,
combining the four skills. It was important to determine the extent to which the
profile scores provided useful (and usable) information to the test users (university
departments etc.). Here again what was needed was a statistical analysis across a wide
range of candidates of the various scores and a survey (through interview, question-
naire etc.) of the actual use of these different modes of reporting to the institutions.
Since ELTS information was already being used in a large number of tertiary
[ Pobierz całość w formacie PDF ]