PISA and the Development of Reading Literacy in Teacher Training
by Liesel Hermes

(Revised version of a paper given at the IATEFL Conference in Brighton, April 2003.)

The PISA study (Programme for International Student Achievement) was first conducted in 2000 and will be repeated every three years. It assesses reading (in the mother tongue), mathematical and scientific literacy. In the 2000 survey, reading literacy was at the centre of the researchers' interest. The results in Germany proved to be quite disastrous in the eyes of the German public, authorities and ministries of education. In all three fields assessed, German pupils(1) came out well below the OECD standard or average. The aims of this article are the following: it will explain what PISA is about and what the results in Germany were in general. It will then go on to demonstrate how important reading literacy is for students of foreign languages and for teacher training programmes in particular and what measures can be taken to raise students' awareness of how reading comprehension functions.

PISA Study

Reading, mathematics and science are all taught at school, but the assessments were not concerned with school curricula. Rather they assessed the pupils' capacity to meet real life challenges, i.e. the tasks were geared to situations pupils may encounter later in life. The three literacy abilities named were tested because they are thought to be essential prerequisites for pupils to manage their lives, both in their further education as well as in their occupational and private lives. The objective was for the pupils to demonstrate understanding of key concepts, to master certain processes and to apply their knowledge in every-day situations.

The tests were developed by experts from the participating countries. 15-year-old pupils were assessed because they form the age group which is still at school around the world. By the same token that means it was the age group that was tested, not necessarily classes. 15-year-old pupils in Germany may be in classes 8, 9 or 10.

Apart from solving the problems set, the pupils had to fill in questionnaires with questions about themselves and their family background. In addition, head teachers had to fill in questionnaires and answer questions about their schools. This brief description goes to show the serious limitations of the PISA study. PISA offers no more than a snapshot of pupils' performance. It is not a long-term study. It does not assess classes to compare class achievements, but individuals. It makes no assertions about the quality of teaching or about developmental factors like learning progress over several months. There was no video documentation of any class, not even observation of lessons. Teachers were not questioned to comment on their pupils or their own teaching, such as preference of teaching methods or materials used. All of these factors go to show that the results of PISA can only be judged within the rigid framework of the tests conducted.

Reading Literacy

Within the PISA study reading literacy is defined as "understanding, using, and reflecting on written texts, in order to achieve one's goals, to develop one's knowledge and potential, and to participate in society." In other words, the focus is on knowledge and skills required to apply "reading for learning" rather than on "learning to read" (www.pisa.oecd.org). Since the knowledge gap between readers and non-readers is widening, it becomes all the more important for human beings to be able to read in order to find their place in the world. Pupils are required to deal with texts of different kinds and perform a range of tasks.

The texts can be broadly differentiated into continuous and non-continuous text types. The continuous ones are narration, exposition, description, argumentation, and instruction. The non-continuous ones are diagrams, charts, tables, forms and advertisements. About 62% of the tasks to be performed relate to the continuous texts, the remaining 38% to the non-continuous ones.

Reading literacy is to be demonstrated in three dimensions:

• retrieving specific information from texts,
• interpreting texts and
• reflecting on and evaluating texts.

These three levels reflect the use of texts in everyday life, where one has to read texts for private or public, for occupational or educational purposes. In the study, all this has to be performed through closed multiple-choice tasks as well as open questions. Pupils have to demonstrate their proficiency in retrieving information, forming an understanding of the text, reflecting on its content and integrating it into their own world or prior knowledge.

Five competence levels have to be differentiated: level 1 at the bottom means solving simple multiple-choice tasks such as locating obvious information in the text. Level 2 demonstrates proficiency in solving basic reading tasks, locating information and making low-level inferences. Level 3 involves tasks of moderate complexity, and level 4 means locating embedded information and construing meaning from nuances of language. Level 5 is the most sophisticated one, requiring pupils to find information that is hard to find, show detailed understanding and evaluate texts critically.

The results in Germany proved to be disastrous. Germany came out 22nd of 32 participant countries altogether. The median was 484 points, which means 16 points lower than the OECD average of 500 points. The average reading proficiency of German pupils is between competence levels II and III. The third dimension, reflecting on and evaluating texts, was not reached adequately by German pupils. Another striking result concerns the internal variation of results, which is among the highest world-wide. That means that in Germany the weakest 5% and the strongest 5% are more than 110 points apart. By way of contrast there are countries like Finland, Japan or Korea with fairly low internal disparities plus a high mean performance. By the same token, very many students in Germany do not even perform reading at level I, namely around 10%. Here as in all the results concerning reading literacy, the proficiency of girls and boys differs widely.

10% of all German pupils performed lower than level I, another 13% performed no higher than level I. That means that nationwide a little less than a quarter of all pupils are extremely weak readers. For this astonishingly large group, extracting information, interpreting and evaluating a text will possibly pose grave problems in later life, be it for private or public or occupational purposes. By the same token around 9% of all pupils performed at level V, which is the OECD average.
All around the world, girls are better readers than boys, up to 35 points (which is one competence level). In Germany more than 26% of the boys perform below level I or just at level one, compared with 6.8% of the girls (below level 1) and 11% (level I). And only 6.7% of the boys perform at the highest level V compared with more than 11 % of the girls.

In a questionnaire the pupils were asked, among other things, if they loved reading. German pupils to a high degree do not, again with significant differences between boys and girls. More than 50% of the boys asserted that they are not avid readers, compared with only 26% of the girls. Around the world girls read more and love reading more than boys do. One consequence will have to be to guide boys towards reading. That means making interesting and motivating reading materials available as well as creating learning environments conducive to successful reading. The family background also seems to be of high importance. Pupils whose parents read themselves and who used to read to their children in early childhood produced significantly better results than pupils from families where reading is not highly valued.

Another alarming result in Germany is the high internal variance according to the type of school visited. Results vary according to the type of school the pupils attend, i. e., pupils at Hauptschul-level perform less well than pupils at Realschul- or Gymnasium-level. In other words: the tripartite school system in Germany, which selects children at the age of nine or ten and assigns them to a type of school, may be detrimental to the pupils' performance. Children have less chance to succeed once they have been assigned to Hauptschule. If one looks only at the proficiency level of the German Gymnasien, they are significantly above the OECD average. That means, by the same token, that in Germany the type of school strongly determines the pupil's success. (In Baden-Württemberg about 20% of the primary school children who have been selected by their teachers for the Gymnasium, go to Realschule, especially in rural areas, which means in the present context that they will in all probability not get the best training appropriate for their talents [oral information at a PISA conference, organized by the Ministerium für Kultus und Sport, Stuttgart])

(1)"Pupil" here refers to learners at school, whereas "student" refers to learners at university.

Reading Literacy in Teacher Training

All this has consequences for teacher training. Reading competence means to know and use reading strategies, among others skimming and scanning, global or detailed reading according to the specific text and the reading purpose (Nuttal 1996: 48 f.). It means adapting reading speed to the difficulty and structure of the text. The most comprehensive criterion for adult literacy (Oakhill/Garnham 1988: 7), which one presupposes in undergraduate students, is to be aware of one's reading strategies and to adapt them to the type of text and the reading purpose. The lowest skill is the recognition of words, which in English, more than in some other languages, is only possible through the context. It is followed in a rising cline by the recognition of phrases, syntax, whole sentences and the meaning of a text. The movement is therefore "from lower-level rapid, automatic identification skills to higher-level comprehension/interpretation skills" (Grabe 1991: 383; cf. Grabe and Stoller 2002: 20).

Understanding a text can be defined as the construction of meaning. Meaning is not always on the surface, but very often has to be guessed through "making inferences" (Oakhill/Garnham 1988: 23). In other words: it is often implicit rather than explicit. This guessing of the implicit meaning never starts from scratch, so to speak, since every act of reading presupposes the reader's general world knowledge and special previous knowledge, which are encapsulated in the two psychological terms "schema" and "script".

"Schema theory is a theoretical metaphor for the reader's prior knowledge." (Grabe 1991: 384) Formal and content schemata can be differentiated. They are called "scripts" if they refer to series of actions. Both terms have been criticised, but have proved useful for interactive theories of reading. An interactive model of reading defines the process and the interaction of bottom-up and top-down processes. The bottom-up process starts from the data given in the text, the top-down process refers to the reader's activation of his/her world and/or prior knowledge. Skilled adult readers use various cognitive reading strategies and simultaneously process data (bottom-up) while using their prior knowledge (top-down) and thus understand the text and integrate the information into their body of knowledge. Lastly they use meta-reflection to reflect about the process of their comprehension of the text.

Academic reading literacy

Reading is part and parcel of any university programme, at a general level ("educational use" in PISA parlance) just as well as with a view to the future profession ("occupational use", www.pisa.oecd.org). But universities unquestioningly presuppose that undergraduates can read. This is correct if one has a general reading literacy in mind, but it is generally not known if students come to university as the adult skilled readers they are assumed to be. In training foreign language teachers, next to nothing is known whether the greatest stumbling blocks in reaching the desired level of target language reading literacy come from generally poor reading skills or from weaknesses in target language proficiency. In his essay "Reading in a foreign language: a reading problem or a language problem?" Alderson comes to the conclusion that lack of reading competence in a foreign language may rather be a problem of the foreign language than a problem of reading. Students have to reach a "threshold level of competence" (Alderson 1984: 19) before they become competent readers in the foreign language.

My own reading of the relevant literature and my long-standing observation of students training to become teachers of English have convinced me that students come to university with very little knowledge of the reading process, without any knowledge of meta-cognition and with little independence at that. They have a basic knowledge of types of texts, i.e. literature in the narrow sense like a novel or a poem or didactic or pedagogic literature in the wider sense. They can tell a linguistic from an area studies text. In other words, they do apply their world knowledge to the general contents of a text. But the problem starts with defining the purpose of the text, if it is e. g. a description, an argumentation, an instruction, if the author presents facts or his/her own opinion, if he/she wants to inform or persuade the reader. In the same way students may find it hard to recognize the structure of a text, to determine the introduction, the main part and the ending. Moreover they often seem to be unable to transfer prior knowledge to the reading of a particular text, e.g. historical knowledge to a literary text or vice versa, or linguistic knowledge to a text that deals with methodological problems of teaching a foreign language.

Little is known about what exactly makes an academic text difficult for students. Lexical problems can be solved - or so it seems - quite easily if the students look up the words they don't know. But it is a well-known fact that reading comprehension can tend to zero if too many words have to be looked up and the overall meaning can no longer be comprehended. This may happen with academic specialist or culture-bound texts. If comprehension fails at the level of word recognition, higher-level skills cannot be achieved.

Students may have problems in understanding texts with a high degree of abstraction. Abstraction becomes all the more demanding the more distant it is from the students' world or prior knowledge. E. g. in an introductory seminar on the teaching of English as a foreign language, the Reader (anthology of texts) used may contain examples and quote extensively from works of other scholars whose names the students have never come across before and which they consequently cannot easily integrate into their knowledge basis; they also often lack the historical or theoretical context which would make it easy to classify a particular author. These problems - of course - also occur with texts in the mother tongue, but the foreign language introduces an additional barrier in understanding.

Moreover the students may have serious problems grasping the cultural background, subliminal cultural assumptions, beliefs and values in a foreign text. Thus they may be unable to understand allusions, irony or puns.

Whereas it can certainly be taken for granted that lower-level processing functions, the scope of a receptive vocabulary cannot be ascertained. Therefore the development of a broad recognition vocabulary as a "critical component of reading comprehension" (Grabe 1991: 392) is undoubtedly an objective of primary importance. However, this development is left to the students and is usually - at least to my knowledge - not made a teaching aim, although Grabe has shown that "higher-level processing" must be learnt as well. As he points out, "[b]eginning readers focus on process strategies (e.g. word identification), whereas more proficient readers shift attention to more abstract conceptual abilities and make better use of background knowledge, using only as much textual information as needed for confirming and predicting the information in the text" (Grabe 1991: 377; cf. Grabe and Stoller 2002: 15)). With first term students, it is unclear which level of abstraction they have reached and if lack of comprehension is due to lack of vocabulary rather than lack of intellectual thinking. Students have to employ their sets of schemata and scripts in order to comprehend texts in a foreign language and to overcome remaining linguistic difficulties. Grabe ascertains on the basis of research: "… a high degree of background knowledge can overcome linguistic difficulties." (Grabe 1991: 390; cf. Grabe and Stoller 2002: 12) However, these faculties have to be actively employed as cognitive reading strategies.

Three factors therefore constitute students' academic reading comprehension:

• a large reading comprehension vocabulary, both on a general and an academic level, which has to be continually broadened by the students through voluntary reading,
• the use of cognitive reading strategies according to the contents and structure of the particular text as well as the reading purposes,
• the use of world knowledge, prior knowledge, schemata and scripts, again according to the contents and structure of the text involved as well as the reading purposes.

The development of meta-reflection in teacher training

With regard to reading comprehension, autonomy can be described as the faculty to set reading objectives for oneself, to find appropriate methods of reading academic texts of various degrees of difficulty and abstraction, to use cognitive reading strategies, to monitor these strategies and to critically evaluate the outcome of the reading process. The autonomous reader is highly motivated, reliable, oriented towards success, persevering in his/her efforts to succeed, and self-critical when it comes to evaluating the outcome (cf. Williams/Burden 1997: 148).

The ability to reflect about one's own reading processes is not an automatism, but has to be initiated and actually taught in order to develop. Future teachers of a foreign language have to learn as part of their study programmes on the one hand to read academic texts and to reflect about their own comprehension process, on the other hand to analyse learners' texts, looking for hidden linguistic problems and anticipating cultural or other questions their pupils may ask, and then devising methods to guide them through the reading processes. These abilities can be developed through experiential learning, in which the students go through the steps of understanding texts themselves, of solving reading problems and analysing them with a view to their own learning processes.

These processes of reflection and meta-reflection were a constant feature of a didactic seminar on "Reading in a Foreign Language" I taught in the summer semester of 2002. At the beginning of the semester the students filled in a non-anonymous questionnaire in order to prepare them for the processes of reflection. They were asked about their awareness of reading and comprehension problems when reading English texts and if these were related to lack of vocabulary or rather to the difficulties of the texts and which sorts of texts posed special problems for them. They outlined special difficulties with texts for which they lacked background knowledge and non-continuous texts. They were also aware of the importance of cultural knowledge in text understanding.

Specific exercises were done to demonstrate to the students that progress in learning has to be monitored and that reading comprehension can only be ascertained in a communicative process and exchange or interpretation with others. The issues were discussed by the students in a framework of experiential learning.

To begin with, the students were asked to familiarize themselves with the self-evaluation grid ("Gemeinsame Referenzniveaus: Raster zur Selbstbeurteilung") in the publication Gemeinsamer europäischer Referenzrahmen (2001: 36) This grid was used to demonstrate the interdependence of the four skills of listening, reading, speaking and writing. The students analysed and discussed the various reading levels and found out that level B2 (vantage) was more or less their common ground. With a view to levels C1 (effective operational proficiency) and C2 (mastery) however, they were more sceptical. C1 reads in the German translation. "Ich kann lange und komplexe Sachtexte und literarische Texte verstehen und deren stilistische Merkmale wahrnehmen. Ich kann Fachartikel und lange technische Anleitungen verstehen, auch wenn sie nicht in Beziehung zu meinem Spezialgebiet stehen." (2001: 36) The students were very critical about this level of attainment which - they felt - they would not even achieve in their mother tongue. Since most of the students had spent some time abroad (usually at least one semester at a partner university), they were in a position to evaluate their own reading comprehension. Understanding technical instructions, the vocabulary of which may be unfamiliar to the reader, is a daunting objective even in one's own language. All of which goes to show that one has to be critical of texts and the demands made.

Next, general reading methods were discussed, the source being a grid in the publication Umgang mit wissenschaftlicher Literatur (1994: 61). Here three reading methods were explained in detail, among them the PQ4R method (preview, question, read, reflect, recite, review) and the SQ3R method (survey, question, read, recite, review). The former was criticized because it presupposes that the reader asks questions about the text three times. This seemed to be too much of a good thing and not wholly convincing as a realistic technique. The SQ3R method, on the other hand, found favour with the students because it includes a step in which the reader takes note of the structure of the text as well as of its communicative function. Both steps made the students aware of their own roles as readers as well as of their objectives as foreign language teachers.

Among the numerous texts the students read and analysed, the following three proved to be especially enlightening with a view to their future profession. One was taken from Oakhill/Garnham (1988: 116):

Ants
Everywhere they go they put out a special chemical from their bodies. They cannot see this chemical, but it has a special odor. An ant must have a nose in order to smell this chemical odor. Another thing about ants is they do not have a nose. Ants cannot smell this odor. Ants can always find their way home by smelling this odor to follow the trail.

It took them a long time to pluck up courage and criticise the text as being grossly inconsistent. Here they learnt that a text requires cohesion and coherence in order to be understandable. Unwillingly they had succumbed to the same error that had been committed by 8-and 10-year-olds who had professed the text was "fully comprehensible", i. e. the readers' "unwillingness to criticize written material" (Oakhill/Garnham 1988: 116). Being critical of reading material is of course of paramount importance for their own development as students capable of the kind of analysis and reflection their future profession will demand of them. Starting at this relatively simple level is a first step toward developing these crucial skills of criticism.

A PISA task was also discussed and analysed:

Bad Taste
Did you know that in 1996 the Australian population spent roughly the same amount of money on chocolate the Australian Government spent on international aid? (letter to the editor of an Australian daily by Arnold Jago, 1997)

Task 5:
How does Jago want his readers to react? With feelings of
a) guilt b) amusement c) fear d) complacency
Task 6:
What kind of action do you think Jago wants the readers of his letter to take?
(German version in
www.heute.t-online.de/ZDFheute/artikel/0.1251,MAG-0-11531,00.html)

Here the main objective for the students was meta-reflection, i.e. what steps they took to arrive at which solution. The structural schema was easy to determine: students know what a letter to the editor is and also what the usual purpose of such a letter is, namely to draw attention to something. The ultimate aim, however, was to find out if this task can be solved without making cultural assumptions or if background knowledge of Australian politics and culture are necessary. One student who had just come back from England asked for the price of chocolate in Australia. Her English background knowledge told her that chocolate in England is very expensive and a luxury commodity. She activated her cultural knowledge and would have liked to draw conclusions as to Australia in order to determine the communicative purpose of the letter. In other words she reasoned: if I know the price of chocolate in Australia, I will be able to determine what the Australian government does in point of fact spend on international aid. But this did not help her.

Another student had just come back from a semester at an Australian university and activated her background knowledge of Australian politics and told the others that Australians spend very little on international aid. The general discussion yielded the following result: Readers never start from scratch, but activate their world or prior knowledge (more often than not unconsciously), and even an apparently simple text like the one quoted will be read and understood on the basis of various cultural assumptions, which may aid the readers in understanding or may pose obstacles.

Another example was given to raise the students' awareness of the process of inference when reading a text:

Tanya and Darren went to Luigi's.
They ate pizzas.
On the way home they praised the chef.
(Oakhill/Garnham 1988: 28)

If one uncovers the sentences one after the other, the first inference that has to be made is with "Luigi". This requires world knowledge: people acquainted with Italian cuisine, particularly outside Italy, will know immediately that it can only refer to a pizzeria, or perhaps an ice-cream parlour. This inference is borne out by the second sentence. The word "chef" in the third sentence may pose a small problem for German readers. From the context, German readers will be able to guess that this "chef" means the chef de cuisine, which means that the English word "chef" has a different or narrower meaning than the German one, depending on the speaker's cultural and social background.
It is interesting for German readers to read in Oakhill/Garnham that the authors include in their "restaurant script": "Tanya and Darren were shown to a table" (1988: 28), which is based on the English/American cultural rule of waiting to be seated. This is usually not included in the German restaurant script. Thus the authors take their own cultural script for granted.

Conclusion

Reading tasks of the kinds described aroused in the students an awareness of the processes they go through in understanding a written text. They learn that reading comprehension is achieved by a combination of top-down and bottom-up processes. The activation of world and prior knowledge, of formal text and cultural schemata or scripts are absolutely necessary in order to understand a text. Thus the linguistic difficulty of a text in a foreign language is only one aspect in a whole array of problems. It is the teacher's task in teaching a foreign language to anticipate problems to help the pupils find their own ways of understanding texts and of developing reading comprehension in a foreign language. This demanding task can only be fulfilled by foreign language teachers if they have gone through these processes themselves, if they have had the opportunity of experiential learning, including stages of reflection and - above all - meta-reflection.

Bibliography

Alderson, J. C. and A. H. Urquhart (edd.): Reading in a Foreign Language. London: Longman, 1984.

Alderson, J. C.: "Reading in a foreign language: a reading problem or a language problem?" In: Alderson/Urquhart: 1-24.

Benson, P. & P. Voller (edd.): Autonomy and Independence in Language Learning. London: Longman, 1997.

Carrell, P. K., J. Devine and D. E. Eskey (edd.): Interactive Approaches to Second Language Reading. Cambridge University Press, 1988.

Deutsches PISA-Konsortium (Hrsg.). PISA 2000. Basiskompetenzen von Schülerinnen und Schülern im internationalen Vergleich. Opladen: Leske + Budrich, 2001.

Deutsches PISA-Konsortium (Hrsg.). PISA 2000. Ein differenzierter Blick auf die Länder der Bundesrepublik Deutschland. Opladen: Leske + Budrich, 2003.

Goethe-Institut Inter Nationes (Hrsg). Europarat, Rat für europäische Zusammenarbeit. Gemeinsamer europäischer Referenzrahmen für Sprachen: lernen, lehren, beurteilen. Berlin, München: Langenscheidt, 2001.

Grabe, W.: "Current Developments in Second Language Reading Research". TESOL Quarterly, 25 No. 3 (1991): 375-406.

Grabe, W. and F. L. Stoller: Teaching and Researching Reading. Harlow: Pearson Education, 2002.

Hermes, L.: "Leseverstehen". In: J. P. Timm (Hrsg.): Englisch lernen und lehren. Didaktik des Englischunterrichts. Berlin. Cornelsen, 1998: 229-236.

Hermes, L.: "Überlegungen zur Theorie und Praxis von Lernerautonomie". K. Aguado und A. Hu (Hrsg.): Mehrsprachigkeit und Mehrkulturalität. Dokumentation des 18. Kongresses für Fremdsprachendidaktik. Berlin: Pädagogischer Zeitschriftenverlag, 2000: 335-341.

Little, D.: Learner Autonomy. vol. 1: Definitions, Issues and Problems. Dublin: Authentik Books for Language Teachers, 1991.

Little, D.: "Learner autonomy: A theoretical construct and its practical applications". Die Neueren Sprachen, 93 Nr. 3 (1994): 430-442.

Little, D.: "Learner autonomy in the foreign language classroom: theoretical foundations and some essentials of pedagogical practice". Zeitschrift für Fremdsprachenforschung 8 Nr. 2 (1997): 227-244.

Nuttall, C.: Teaching Reading Skills in a Foreign Language. Oxford: Heinemann, 1996.

Oakhill, J. and A. Garnham: Becoming a Skilled Reader. Oxford: Blackwell, 1988.

OECD. Reading for Change. Performance and Engagement across Countries. Results from PISA 2000. Paris: OECD Publications, 2002.

OECD. The PISA 2003 Assessment Framework. Mathematics, Reading, Science and Problem Solving Knowledge and Skills. Paris: OECD Publications, 2003.

OECD. Lesen kann die Welt verändern. Leistung und Engagement im Ländervergleich. Ergebnisse von PISA 2000. Paris: OECD Publications, 2003.

Stary, J, und H. Kretschmer: Umgang mit wissenschaftlicher Literatur. Eine Arbeitshilfe für das sozial- und geisteswissenschaftliche Studium. Berlin: Cornelsen Scriptor, 1994.

Williams, M. and R. L. Burden: Psychology for Language Teachers. A social constructivist approach. Cambridge University Press, 1997.

URL:

www.heute.t-online.de/ZDFheute/artikel/0,1251,MAG-0-11542,00.html (April 20, 2002)

www.pisa.oecd.org

www.pisa.oecd.org

Biodata

Prof. Dr. Liesel Hermes is president of the University of Education, Karlsruhe, Germany. She is the editor of the journal Neusprachliche Mitteilungen aus Wissenschaft und Praxis, i. e. the journal of the Fachverband Moderne Fremdsprachen (FMF) in Germany. She is a member of IATEFL and TESOL, among others. Before coming into office she was a professor of English literature and didactics.
Her research interests are 20th century English literature, Australian literature, teaching literature, Intercultural Education, Action Research and Learner Autonomy in University Education.
Email: hermes@ph-karlsruhe.de

To the original article

To the article index