CEFR-Referenced Item Specification Analysis of TOEIC Incomplete Sentences Part on Intermediate Thai Learners
Keywords:
CEFR, TOEIC®, Incomplete Sentences, Item specificationAbstract
While there have been many studies the exploring reading section in TOEIC, there are a few studies examining item specification. The objective of this study is to analyze item specification of the part-5 test scores of 52 mostly intermediate learners taking intermediate TOEIC-formatted test. The analysis was conducted by identifying the item specification in each test item as well as its CEFR level in test items whose correct responses are low. The result shows that the CEFR levels of the test items can predict, to a large extent, each test taker’s proficiency and the distribution of the test items with low-correct responses is in line with the CEFR levels of the test items and learners’ TOEIC scores. Test takers at the B1 level by and large cannot answer test items at the B2 or above level. The test takers at the B1 level may not necessarily answer all the B1 level questions should they be at B1 threshold. The item specification analysis also reveals that the most frequent type of errors found in the study is word choice and tense-aspect, respectively. Along with CEFR, other possible explanations for errors in each test item are proposed. Pedagogical implications are that instructors should focus on vocabulary learning in context, collocation and colligation, verb tense, respectively.
References
Amornwongpeeti, A., & Pongpairoj, N. (2014). The Acquisition of English Restrictive and Non-restrictive Relative Clauses by L1 Thai Learners. NIDA Journal of Language and Communication, 19(21), 1-54.
Biber, D., & Conrad, S. (2019). Register, Genre, and Style. Cambridge: Cambridge University Press.
Cantos, P., & Almela, M. (2017). Colligational effects of collocation: Lexically-conditioned dependencies between modification patterns of the noun cause. In M. Pace-Sigge, & K. J. Patterson, Lexical Priming: Applications and advances (pp. 232–250). Amsterdam: John Benjamins.
Cheng, W. (2012). Exploring corpus linguistics : language in action. London: Routledge.
Chiang, H.-H. (2018). English Vocabulary Size as a Predictor of TOEIC Listening and Reading Achievement among EFL Students in Taiwan. Theory and Practice in Language Studies, 8(2), 203-212.
Chomsky, N. (2006). Language and Mind. Cambridge: Cambridge University Press.
Council of Europe. (2001). Common European framework of reference for languages: learning, teaching, assessment. Cambridge: Cambridge University Press.
Crosthwaite, P. (2017). Does EAP writing instruction reduce L2 errors? Evidence from a longitudinal corpus of L2 EAP essays and reports. International Review of Applied Linguistics in Language Teaching, 56(3), 315-343.
DeKeyser, R. (2005). Grammatical development in language learning. Malden, Mass.: Blackwell.
Deuber, D. (2010). Modal Verb Usage at the Interface of English and a Related Creole: A Corpus-based Study of Can/Could and Will/Would in Trinidadian English. Journal of English Linguistics, 38(2), 105-142.
Deylamsalehi, A., Barjasteh, H., & Vaseghi, R. (2015). Anchoring Effects of English Vocabulary Instruction: The Case of Pre-University Students. Advances in Language and Literary Studies, 6(3), 138-143.
DÃez-Bedmar, M. B. (2018). Fine-tuning descriptors for CEFR B1 level: insights from learner corpora. ELT Journal, 72(2), 199–209.
Dissington, P. A. (2018). Addressing the Problem of Negative Lexical Transfer Errors in Chilean University Students. Profile Issues in Teachers' Professional Development, 20(1), 25-40.
Duan, M. (2011). A Corpus-based Study of the Misuse of Tenses in the English Composition of Chinese College Students. English Language Teaching, 4(4), 173-180.
Ellis, N. C. (2006). Cognitive Perspectives on SLA: The Associative-Cognitive CREED. AILA Review, 19, 100 - 121.
Feist, J. (2014). Premodifiers in English: Their Structure and Significance. New York: Cambridge University Press.
Flowerdew, J., & Forest, R. W. (2015). Signaling Nouns in English: A Corpus-Based Discourse Approach. Cambridge: Cambridge University Press.
Foley, J. A. (2019). Issues on Assessment using CEFR in the Region. LEARN Journal: Language Education and Acquisition Research Network Journal, 12(2), 28-48.
Fries, C. C. (1925). The Periphrastic Future with Shall and Will in Modern English. PMLA, 963-1024.
Furnham, A., & Boo, H. C. (2011). A literature review of the anchoring effect. The Journal of Socio-Economics, 40(1), 35-42.
Gardner, D. (2013). Exploring Vocabulary: Language in Action. London: Routledge.
Gvozdanović, J. (2012). Perfective and Imperfective Aspect. In R. I. Binnick, The Oxford Handbook of Tense and Aspect (pp. 781–802). Oxford: Oxford University Press.
Harada, S. (2016). Effects of the TOEIC course on students’ TOEIC scores. The Bulletin of St. Margaret’s, 48, 1-17.
Hinnon, A. (2014). Common Errors in English Writing and Suggested Solutions of Thai University Students. Journal of Humanities and Social Sciences, Khon Kaen University, 31(2), 165-180.
Hoad, T. F. (2003). The Concise Oxford Dictionary of English Etymology. Oxford: Oxford University Press.
Hoey, M. (2005). Lexical Priming: A New Theory of Words and Language. London: Routledge.
Hokuriku University. (2021). Hokuriku International Exchange Center. Retrieved from Hokuriku University: https://iec.hokuriku-u.ac.jp/en/41.html
Hunston, S. (2011). Corpus Approaches to Evaluation: Phraseology and Evaluative Language. London: Routledge.
Kamijo, T. (2010). TOEIC Reading Comprehension: Analysis of the Texts and Questions, Practical Application and Its Implications. Kyoto:「立命館高ç‰æ•™è‚²ç ”究ã€ç«‹å‘½é¤¨å¤§å¦æ•™è‚²é–‹ç™ºæŽ¨é€²æ©Ÿæ§‹ç´€è¦, (10), 173-185.
Katip, P., & Gampper, C. (2016). English as a Foreign Language Student Productive Conditional Verb Form Errors in Thailand. Language Education and Acquisition Research Network, 9(2), 1-13.
Kim, M., & Crossley, S. A. (2020). Exploring the Construct Validity of the ECCE: Latent Structure of a CEFR-Based High-Intermediate Level English Language Proficiency Test. Language Assessment Quarterly, 18(3), 434-457.
Kusdianty, S. R. (2016). The Analysis of Students’ Error in Using Relative Clause. Indonesian EFL Journal, 2(1), 60-70.
Lawal, A. M., Idemudia, E. S., & Adewale, O. P. (2017). Academic self-confidence effects on test anxiety among Nigerian university students. Journal of Psychology in Africa, 27(6), 507-510.
Lee, J.-Y. (2018). The Use of Test-Taking Strategies and Students' Performances in Answering TOEIC Reading Comprehension Questions. Taiwan Journal of TESOL, 15(2), 33-64.
Leech, G. N. (2004). Meaning and the English Verb. London: Routledge.
Leow, R. P. (2015). Explicit Learning in the L2 Classroom: A Student-Centered Approach. New York: Routledge.
Listia, R., & Febriyanti, E. R. (2020). EFL Learners’ Problems in Using Tenses: An Insight for Grammar Teaching. IJET (Indonesian Journal of English Teaching), 9(1), 86–95.
Longman. (2014). Longman Dictionary of Contemporary English. Harlow: Pearson Education Limited.
Maliwan, S. (2018). TOEIC Preparation Course for Aviation Personnel Development Institute Students. KASEM BUNDIT JOURNAL, 19(June), 234–243.
Nation, P. (2014). How much input do you need to learn the most frequent 9,000 words? Reading in a Foreign Language, 26(2), 1-16.
Nuyts, J., & Auwera, J. V. (2016). The Oxford Handbook of Modality and Mood. Oxford: Oxford University Press.
Papol, R., Boonsue, W., Nakcharoen, J., Sriratanaban, S., Silachai, N., & Khamsri, T. (2020). The Analysis of Listening and Reading Problems in Taking Toeic of English Major Students, Faculty of Humanities and Social Sciences, Chiang Mai Rajabhat University. Journal of Buddhist Education and Research, 6(2), 239-251.
Phoocharoensil, S., & Simargool, N. (2010). English Relativization and Learners’ Problems. Pan-Pacific Association of Applied Linguistics, 14(1), 109-129.
Pienemann, M., & Keßler, J.-U. (2011). Studying Processability Theory: An Introductory Textbook. Amsterdam: John Benjamins.
Powers, D. E., & Powers, A. (2015). The Incremental Contribution of TOEIC Listening, Reading, Speaking, and Writing Tests to Predicting Performance on Real-Life English Language Tasks. Language Testing, 32(2), 151–167.
Radford, A. (2016). Analysing English Sentences. Cambridge: Cambridge University Press.
Richard, J.-P. J. (2020). Investigating CEFR-J Self-Assessment and TOEIC Listening and Reading Scores. Global Managment, 3, 21-32.
Robb, T. N., & Ercanbrack, J. (1999). A Study of the Effect of Direct Test Preparation on the TOEIC Scores of Japanese University Students. TESL-EJ. 3(4), 25-54.
Rogers, B. (2006). Complete Guide to the TOEIC Test. Florence, KY: Thomson.
Runnels, J. (2016). Self-Assessment Accuracy: Correlations between Japanese English Learners' Self-Assessment on the CEFR--Japan's Can Do Statements and Scores on the TOEIC®. Taiwan Journal of TESOL, 13(1), 105-137.
Saville-Troike, M., & Barto, K. (2016). Introducing Second Language Acquisition. Cambridge: Cambridge University Press.
Sinclair, J. (1996). Textus. The Search for Units of Meaning, 9(1), 75-106.
Taguchi, K. (2015). An Exploratory Study on Correlations among Vocabulary Size, Vocabulary Learning Strategy Use, TOEIC Scores and Self-Efficacy. The Economic Review of Toyo University, 41(1), 55-67.
Taladngoen, U., & Esteban, R. H. (2022). Assumptions on Plausible Lexical Distractors in the Redesigned TOEIC Question-Response Listening Test. LEARN Journal: Language Education and Acquisition Research Network, 15(2), 802-829.
Tannenbaum, R. J., & Wylie, E. C. (2013). Mapping TOEIC® and TOEIC Bridge™ Test Scores to the Common European Framework of Reference. Princeton, NJ: Educational Testing Service.
Trew, G. (2007). A Teacher's Guide to TOEIC® Listening and Reading Test: Preparing Your Students for Success. Oxford University Press.
VanPatten, B. (2004). Processing Instruction: Theory, Research, and Commentary. New York: Routledge.
Walenta, M. (2019). Form-Function Mapping in Content-Based Language Teaching. Cham: Springer International Publishing.
Wang, S. (2015). An Empirical Study on the Role of Vocabulary Knowledge in EFL Listening Comprehension. Theory and Practice in Language Studies, 5(5), 989-995.
Wangmanee, P., & Vongtangswad, M. (2022). Effectiveness of the Potential Promoting Strategies on the TOEIC Test of English Major Students, Chiang Rai Rajabhat University. Journal of Human and Society, Sisaket Rajabhat University, 6(1), 40-60.
Wanner, A. (2009). Deconstructing the English Passive. Berlin: De Gruyter Mouton.
Wei, Y., & Low, A. C. (2017). Monitoring Score Change Patterns to Support TOEIC® Listening and Reading Test Quality. ETS Research Report, 2017(1), 1-22.
Wilson, K. M., Komarakul Na Nagara, S., & Woodhead, R. (2004). TOEIC®/LPI RELATIONSHIPS IN ACADEMIC AND EMPLOYMENT CONTEXTS IN THAILAND. ETS Reserach Report Series, 2004(1), 1-33.
Wu, H.-l. (2013). Implementation of an elite sophomore English class at a university in Taiwan. Hsiuping Journal of Humanities and Social Sciences, 21(2013), 21-30.
Yang, S., & Huang, Y. Y. (2004). The Impact of the Absence of Grammatical Tense in L1 on the Acquisition of the Tense-Aspect System in L2. International Review of Applied Linguistics in Language Teaching, 42(1), 49-70.
Zahar, R., Cobb, T., & Spada, a. N. (2001). Acquiring Vocabulary through Reading: Effects of Frequency and Contextual Richness. The Canadian Modern Language Review, 57(4), 541-572.
