ISCL Hauptseminar (Summer semester 2018, Meurers)

Linguistic Corpus Annotation


Annotated corpus resources are the primary way in which linguistic insight still feeds current computational linguistic research and applications. This seminar will provide an overview of the creation and use of linguistically annotated corpora in theoretical and computational linguistics. It will start with basic (but surprisingly non-trivial and consequential) questions such as how to tokenize or sentence segment a corpus as well as conceptual considerations relevant to the creation of annotation schemes, and will then explore different types of corpora and different types of annotations (morphological, constituency, dependency, semantic and formal pragmatic), both covering commonly used standard resources (such as the Penn Treebank) as well as linguistic corpus resources for languages other than English.

Instructor: Prof. Dr. Detmar Meurers

Course meets: 4 SWS in Seminarraum 0.01 Blochbau (Wilhelmstr. 19)

Credit Points:

Online syllabus:

Moodle page:

Please enroll in this course by logging into this moodle course with your ordinary ZDV university login.

Nature of course and our expectations: This is a research-oriented Hauptseminar, in which we jointly explore perspectives and approaches. You are expected to

  1. regularly and actively participate in class, read the papers assigned by any of the presenters and post a meaningful question on Moodle to the “Reading Discussion Forum” on each reading at the latest on the day before it is discussed in class.
  2. explore and present a topic:
  3. if you pursue the 9 CP option, work out a project term paper

Academic conduct and misconduct: Research is driven by discussion and free exchange of ideas, motivations, and perspectives. So you are encouraged to work in groups, discuss, and exchange ideas. At the same time, the foundation of the free exchange of ideas is that everyone is open about where they obtained which information. Concretely, this means you are expected to always make explicit when you’ve worked on something as a team – and keep in mind that being part of a team always means sharing the work.

For text you write, you always have to provide explicit references for any ideas or passages you reuse from somewhere else. Note that this includes text “found” on the web, where you should cite the url of the web site in case no more official publication is available.

Class etiquette: Please do not read or work on materials for other classes in our seminar. All portable electronic devices such as cell phones and laptops should be switched off for the entire length of the flight, oops, class.

Session plan:

  1. 24./26.4. Overview of course and the topic [Detmar Meurers]
  2. 1.5. Tag der Arbeit
  3. 3.5. Tokenization [Detmar Meurers]
  4. 8.5. Text Processing with the Command Line Interface [Individual Project]
  5. 10.5. Christi Himmelfahrt
  6. 15.5. POS Overview, Which and how many tags?
    [Jerry Chin-Chun Chang, Lisa Stiller, Ekaterina Lazaruk]
  7. 17.5. cont.
  8. 22.5/24.5. Pentacoste
  9. 29.5. Lexical analysis using unix tools
    [Lukas Stein, Matthias Drews]
  10. 31.5. Fronleichnam
  11. 5.6. Tagset Comparison [Katherine Munro]
    Different Sources of Evidence [Sarah Taylor]
  12. 7.6. English POS Tagsets: Susanne, WSJ, BNC-CLAWS5/7
    [Zivile Rasimaite, Nora Kumpikova]
  13. 12.6. Syntax overview and hands-on-exploration
    [Anna Bródy, Theresa Bertele, Nazanin Khazaee Farid]
  14. 14.6. Querying syntactically parsed corpora (hands-on) [Nadine Balbach]
  15. 19.6. English Penn Treebank [Xuefeng Luo]
  16. 21.6. syntactic annotation of exciting languages:
  17. 26.6. syntactic annotation of exciting languages:
  18. 28.6. universal dependencies
  19. 3.7. Semantics
  20. 5.7. cont.
  21. 10.7. Discourse: RST, QUD
  22. 12.7. cont.
  23. 17.7. Evaluation
  24. 19.7. cont.
  25. 24./26.7. Tübingen SLA Summer School

Topics we can chose from

We focus on the conceptual issues, in particular questions relating to linguistic modeling. Which properties and insights can be and have been identified and annotated in corpora?


   Abeillé, A. (ed.) (2003). Treebanks: Building and using syntactically annotated corpora. Dordrecht: Kluwer.

   Abeillé, A., T. Brants & H. Uszkoreit (eds.) (2000). Proceedings of the Second Workshop on Linguistically Interpreted Corpora (LINC-00). Luxembourg. Workshop information at

   Abeillé, A., L. Clément & F. Toussenel (2003). Building a Treebank for French. In Abeillé (2003).

   Artstein, R. & M. Poesio (2009). Survey Article: Inter-Coder Agreement for Computational Linguistics. Computational Linguistics 34(4), 1–42. URL

   Atalay, N., K. Oflazer & B. Say (2003). The annotation process in the Turkish treebank. In Proceedings of the 4th International Workshop on Linguistically Interpreteted Corpora (LINC).

   Atwell, E., G. Demetriou, J. Hughes, A. Schiffrin, C. Souter & S. Wilcock (2000a). A comparative evaluation of modern English corpus grammatical annotation schemes. International Computer Archive of Modern and Medieval English (ICAME). Issue on Computers in English Linguistics 24, 7–23. URL

   Atwell, E., G. Demetriou, J. Hughes, A. Schiffrin, C. Souter & S. Wilcock (2000b). Comparing linguistic interpretation schemes for English corpora. In Abeillé et al. (2000). URL Workshop information at

   Atwell, E., J. Hughes & C. Souter (1994). AMALGAM: Automatic Mapping Among Lexico-Grammatical Annotation Models. In J. Klavans & P. Resnik (eds.), Proceedings of The Balancing Act - Combining Symbolic and Statistical Approaches to Language, Workshop in conjunction with the 32nd Annual Meeting of the Association for Computational Linguistics. New Mexico State University, Las Cruces, New Mexico, USA. URL

   Bies, A., M. Ferguson, K. Katz & R. MacIntyre (1995). Bracketing Guidelines for Treebank II Style Penn Treebank Project. University of Pennsylvania. URL

   Boyd, A., M. Dickinson & D. Meurers (2008). On Detecting Errors in Dependency Treebanks. Research on Language and Computation 6(2), 113–137. URL

   Brants, T. (1995). Tagset reduction without information loss. In Proceedings of the 33nd Annual Meeting of the Association for Computational Linguistics (ACL 95). Cambridge, MA: MIT. URL

   Brants, T. & W. Skut (1998). Automation of Treebank Annotation. In Proceedings of New Methods in Language Processing (NeMLaP-98). Syndey. URL

   Brill, E. (2000). Part-of-Speech Tagging. In R. Dale, H. Moisl & H. Somers (eds.), Handbook of Natural Language Processing, New York: Marcel Dekker. URL

   Cheung, J. C. K. & G. Penn (2009). Topological Field Parsing of German. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP. Suntec, Singapore: Association for Computational Linguistics, pp. 64–72. URL

   Cloeren, J. (1999). Tagsets. In van Halteren (1999), chap. 4, pp. 37–54.

   Déjean, H. (2000). How to Evaluate and Compare Tagsets? A Proposal. In Gavrilidou et al. (2000). URL

   Díaz Negrillo, A., D. Meurers, S. Valera & H. Wunsch (2010). Towards interlanguage POS annotation for effective learner corpora in SLA and FLT. Language Forum 36(1–2), 139–154. URL

   Dickinson, M. & D. Meurers (2003a). Detecting Errors in Part-of-Speech Annotation. In Proceedings of the 10th Conference of the European Chapter of the Association for Computational Linguistics (EACL-03). Budapest, Hungary, pp. 107–114. URL

   Dickinson, M. & W. D. Meurers (2003b). Detecting Inconsistencies in Treebanks. In Proceedings of the Second Workshop on Treebanks and Linguistic Theories (TLT-03). Växjö, Sweden, pp. 45–56. URL

   Dickinson, M. & W. D. Meurers (2005a). Detecting Annotation Errors in Spoken Language Corpora. In The Special Session on treebanks for spoken language and discourse at NODALIDA-05. Joensuu, Finland. URL

   Dickinson, M. & W. D. Meurers (2005b). Detecting Errors in Discontinuous Structural Annotation. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL-05). pp. 322–329. URL

   Dienes, P. & C. Oravecz (2000). Bottom-up tagset design from maximally reduced tagset. In Abeillé et al. (2000), pp. 42–47. URL Workshop information at

   Džeroski, S., T. Erjavec & J. Zavrel (2000). Morphosyntactic Tagging of Slovene: Evaluating Taggers and Tagsets. In Gavrilidou et al. (2000), pp. 1099–1104. URL

   Elworthy, D. (1995). Tagset Design and Inflected Languages. In Proceedings of the ACL-SIGDAT Workshop. Dublin. URL

   Forst, M., N. Bertomeu, B. Crysmann, F. Fouvry, S. Hansen-Schirra & V. Kordoni (2004). Towards a Dependency-Based Gold Standard for German Parsers. The TIGER Dependency Bank. In S. Hansen-Schirra, S. Oepen & H. Uszkoreit (eds.), 5th International Workshop on Linguistically Interpreted Corpora (LINC-04) at COLING. Geneva, Switzerland: COLING, pp. 31–38. URL

   Gaizauskas, R. (1995). Investigations into the grammar underlying the Penn Treebank II. Tech. Rep. Research Memorandum CS-95-25, University of Sheffield. URL

   Garside, R., G. Leech & T. McEnery (eds.) (1997). Corpus annotation: linguistic information from computer text corpora. Harlow, England: Addison Wesley Longman Limited.

   Gavrilidou, M., G. Carayannis, S. Markantonatou, S. Piperidis & G. Steinhauer (eds.) (2000). Proceedings of the Second International Conference on Language Resources and Evaluation (LREC-00). Athens.

   Grefenstette, G. (1999). Tokenization. In van Halteren (1999), chap. 9, pp. 117–133.

   Grefenstette, G. & P. Tapanainen (1994). What is a word, what is a sentence? In Proceedings of the 3rd International Conference on Computational Lexicography (COMPLEX-94). pp. 79–87. URL

   Hajič, J., A. Böhmová, E. Hajičová & B. Vidová-Hladká (2003). The Prague Dependency Treebank: A Three-Level Annotation Scenario. In Abeillé (2003), chap. 7, pp. 103–127. URL

   Hajič, J., B. Vidová-Hladká & P. Pajas (2001). The Prague Dependency Treebank: Annotation Structure and Support. In Proceedings of the IRCS Workshop on Linguistic Databases. University of Pennsylvania, Philadelphia, pp. 105–114. URL

   Hajič, J. & B. Hladká (1998). Tagging Inflective Languages: Prediction of Morphological Categories for a Rich, Structured Tagset. In Proceedings of COLING-ACL Conference. Montreal, Canada, pp. 483–490.

   Hajič, J., J. Panevová, E. Buráňová, Z. Urešová & A. Bémová (1999). Annotations at Analytical Layer. Instructions for Annotators. Tech. rep., ÚFAL MFF UK, Prague, Czech Republic. URL English translation by Zdeněk Kirschner.

   Hajičová, E., J. Panevová & P. Sgall (2000). A Manual for Tectogrammatic Tagging of the Prague Dependency Treebank. Tech. Rep. TR-2000-09, ÚFAL MFF UK, Prague, Czech Republic. In Czech.

   Hana, J. & D. Zeman (2005). A Manual for Morphological Annotation, 2nd edition. Tech. Rep. 27, ÚFAL MFF UK, Prague, Czech Republic. URL

   King, T. H., R. Crouch, S. Riezler, M. Dalrymple & R. M. Kaplan (2003). The PARC 700 Dependency Bank. In Proceedings of the 4th International Workshop on Linguistically Interpreted Corpora, held at the 10th Conference of the European Chapter of the Association for Computational Linguistics (EACL-03). Budapest. URL

   Kingsbury, P., M. Palmer & M. Marcus (2002). Adding Semantic Annotation to the Penn TreeBank. In Proceedings of the Human Language Technology Conference. San Diego, California. URL

   Kübler, S. & A. Wagner (2000). Evaluating POS Tagging under Sub-optimal Conditions. Or: Des Meticulousness Pay? In Proceedings of International Conference on Artificial and Computational Intelligence for Decision, Control and Automation in Engineering and Industrial Applications (ACIDCA’2000). Monastir, Tunisia. URL

   Leech, G. (1997). Grammatical Tagging. In Garside et al. (1997), chap. 2, pp. 19–33.

   Lu, X. (2014). Computational Methods for Corpus Annotation and Analysis. Springer.

   Marcus, M., G. Kim, M. Marcinkiewicz, R. MacIntyre, A. Bies, M. Ferguson, K. Katz & B. Schasberger (1994). The Penn treebank: Annotating predicate argument structure. URL

   Marcus, M., B. Santorini & M. A. Marcinkiewicz (1993). Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics 19(2), 313–330. URL

   McEnery, T. & A. Wilson (1996). Corpus Linguistics. Edinburgh Textbooks in Empirical Linguistics. Edinburgh, UK: Edinburgh University Press.

   Meurers, D. (2005). On the use of electronic corpora for theoretical linguistics. Case studies from the syntax of German. Lingua 115(11), 1619–1639. URL

   Meurers, D. & S. Müller (2009). Corpora and Syntax. In A. Lüdeling & M. Kytö (eds.), Corpus linguistics, Berlin: Mouton de Gruyter, vol. 2 of Handbooks of Linguistics and Communication Science, pp. 920–933. URL

   Nivre, J., M.-C. de Marneffe et al. (2016). Universal Dependencies v1: A Multilingual Treebank Collection. In Proceedings of LREC. URL

   Nivre, J., J. Nilsson & J. Hall (2006). Talbanken05: A Swedish Treebank with Phrase Structure and Dependency Annotation. In Proceedings of the fifth international conference on Language Resources and Evaluation (LREC-06). Genoa, Italy. URL

   Oflazer, K., D. Z. Hakkani-Tür & G. Tür (1999). Design for a Turkish Treebank. In Uszkoreit et al. (1999), pp. 28–34.

   Oflazer, K., B. Say, D. Z. Hakkani-Tür & G. Tür (2003). Building a Turkish Treebank. In Abeillé (2003).

   Palmer, D. D. (2000). Tokenisation and Sentence Segmentation. In R. Dale, H. Moisl & H. Somers (eds.), Handbook of Natural Language Processing, New York: Marcel Dekker, pp. 11–35. URL

   Palmer, M., D. Gildea & P. Kingsbury (2005). The Proposition Bank: An Annotated Corpus of Semantic Roles. Computational Linguistics 31(1), 71–105. URL

   Riester, A., L. Brunetti & K. De Kuthy (in press). Annotation Guidelines for Questions under Discussion and Information Structure. In E. Adamou, K. Haude, & M. Vanhove (eds.), Information structure in lesser-described languages: Studies in prosody and syntax, John Benjamins, Studies in Language Companion Series.

   Sampson, G. & A. Babarczy (2003). Limits to annotation precision. In Proceedings of the 4th International Workshop on Linguistically Interpreted Corpora (LINC-03). pp. 61–68. URL

   Santorini, B. (1990). Part-Of-Speech Tagging Guidelines for the Penn Treebank Project (3rd Revision, 2nd printing). Ms., UPenn.

   Schiller, A., S. Teufel & C. Thielen (1995). Guidlines für das Taggen deutscher Textcorpora mit STTS. Tech. rep., IMS-CL, Univ. Stuttgart and SfS, Univ. Tübingen. URL

   Stegmann, R., H. Telljohann & E. W. Hinrichs (2000). Stylebook for the German Treebank in VERBMOBIL. Verbmobil-Report 239, Universität Tübingen, Tübingen, Germany. URL

   Taylor, A., M. Marcus & B. Santorini (2003). The Penn Treebank: An Overview. In Abeillé (2003), chap. 1, pp. 5–22.

   Telljohann, H., E. W. Hinrichs, S. Kübler & H. Zinsmeister (2005). Stylebook for the Tübingen Treebank of Written German (TüBa-D/Z). Tech. rep., Seminar für Sprachwissenschaft, Universität Tübingen, Germany.

   Teufel, S. (1995). A Support Tool for Tagset Mapping. In Proceedings of the SIGDAT Workshop at EACL 95. Dublin. URL

   Teufel, S., H. Schmid, H. Heid & A. Schiller (1996). EAGLES Study of the relation between Tagsets and Taggers. Document eag clwg tags/v, EAGLES. URL

   Thielen, C. & A. Schiller (1996). Ein kleines und erweitertes Tagset fürs Deutsche. In H. Feldweg & E. W. Hinrichs (eds.), Lexikon und Text: wiederverwendbare Methoden und Ressourcen zur linguistischen Erschließung des Deutschen, Tübingen: Max Niemeyer Verlag, vol. 73 of Lexicographica: Series maior, pp. 215–226.

   Tufiş, D., P. Dienes, C. Oravecz & T. Váradi (2000). Principled Hidden Tagset Design for Tiered Tagging of Hungarian. In Gavrilidou et al. (2000). URL

   Uszkoreit, H., T. Brants & B. Krenn (eds.) (1999). Proceedings of the Workshop on Linguistically Interpreted Corpora (LINC-99). Bergen, Norway: Association for Computational Linguistics.

   van Halteren, H. (ed.) (1999). Syntactic Wordclass Tagging. Dordrecht: Kluwer Academic Publishers.

   Váradi, T. & C. Oravecz (1999). Morpho-syntactic ambiguity and tagset design for Hungarian. In Uszkoreit et al. (1999), pp. 8–12. URL

   Voutilainen, A. & T. Järvinen (1995). Specifying a shallow grammatical representation for parsing purposes. In Proceedings of the 7th Conference of the EACL. Dublin, Ireland. URL