In this post I collect all the suggestions I got for the following request: “Looking for Corpora in….” http://www.forum.santini.se/2014/03/looking-for-corpora-to-explore-cross-linguality/
Big thanks to (hope I have not forgotten anybody): Johannes Heinecke, Dominika Rogozinska, Mohamed-Zakaria KURDI, Bartosz Ziólko, Olga Whelan, Margarita Borreguero, Zuloaga, Ayesha Zafar, Will Snellen, Katherine (Katie) Skees Hund, Anna Matyszczyk, Massinissa Ahmim, Marcin Feder, Maria Pia Montoro, Lawrence Niculescu, Jesus Vilares, Ewa Gwiazdecka, Jack Bowers, Taner Sezer, Yvonne Adesam, Kadri Muischnek, Anne Tamm, Ralf Steinberger, Ricardo Campos, Edyta Jurkiewicz-Rohrbacher, Pat, Sara Castagnoli, Adam Przepiorkowski, Hung Le Khanh, Kristian Kankainen, Norton Roman, Mansur Sayhunov.
Suggestions were sent through:
Mailing Lists: Corpora List (email@example.com), BCS IRSG (IR@jiscmail.ac.uk)
LinkedIn Groups: Corpus Linguistics, Computational Linguistics, Natural Language Processing, Applied linguistics, Terminology Services.
Hope this list of corpora is useful for everybody working with multi- and cross-linguality. Please do not hesitate to contact me if you wish to contribute and add more corpora/resources to this list (firstname.lastname@example.org)
Please, let me know if I have inadvertently disregarded you pointers.
— start-of-the-list — Last Updated 13 May 2014
- Multi-lingual corpora & resources
If you can live with some noise, why not using Wikipedia? You can download a dump of the entire Wikipedia of the languages you need (without old version and images), e.g. http://dumps.wikimedia.org/etwiki/ for Estonian (replace ee in eewiki by the ISO language code of the other languages you need. Once you have the dump (XML file) you can extract the page tag which contains the WikiPage including the mediawiki syntax. The latter can be parsed with toolkits like http://sweble.org/ . I made the experience that very short pages are often stubs so they do not contribute “good” text to your corpus. Very long pages are often listings ( http://et.wikipedia.org/wiki/Maailma_riikide_loend ) too and therefore may not be adequate for your task. In order to parse the wiki syntax and get the text, several tools exist, for instance http://sweble.org .
texts published by the European Parliament
The Gutenberg project contains many books in several languages. http://www.gutenberg.org/wiki/Main_Page
Some other ideas for finding texts you might get checking OPUS: http://opus.lingfil.uu.se/ (parallel corpora)
Massive multi-lingual corpora:
CLEF: (http://www.clef-initiative.eu) It may be a good starting point.
The NLTK (Python) toolkit includes corpora in several of the languages you are looking for (and several others), in a variety of genres. Languages that are included for sure are: English, Italian, Polish, I think there may also be Swedish and Hungarian as well, but I’m not 100% sure…. http://www.nltk.org/
You can also get the corpora without the rest of the NLTK tools: http://www.nltk.org/nltk_data/
At the JRC’s Language Technology page http://ipsc.jrc.ec.europa.eu/index.php?id=61, you find parallel corpora for all the languages you are searching for, and more.
Semantics-based Multilingual dictionary a thttp://laconec.com:8088/codemoi
Google’s Cross-Lingual Dictionary (GCLD) — Released: 2012.175M short, unique text strings that were used to refer to one of 7.6M Wikipedia articles (source:
http://ebiquity.umbc.edu/blogger/2012/05/19/google-releases-database-linking-strings-and-concepts/). The details of the data and how it was constructed are in an LREC 2012 paper by Valentin Spitkovsky and Angel Chang, A Cross-Lingual Dictionary for English Wikipedia Concepts (http://nlp.stanford.edu/pubs/crosswikis.pdf). Get the data here:http://www-nlp.stanford.edu/pubs/crosswikis-data.tar.bz2/.
11) Multi-Wordnet: http://multiwordnet.fbk.eu/english/home.php
12) Multilingual Corpora listed by Isabella Chiari: http://www.alphabit.net/home/index.php?option=com_content&view=section&id=6&Itemid=11
1) you can access the Estonian corpora more comfortably at http://www.cl.ut.ee/korpused/segakorpus/ as these pages have also English versions. Most of these corpora have also a morphologically analysed version (lemma, POS, grammatical categories) available at request, free for non-commercial purposes.
2) Estonian: http://www.keeletehnoloogia.ee/projektid/koondkorpus
There is also a quite comprehensive list of all sorts of resources for
Estonian (wordlists, biographical data collections, dialect data,
phonetical resources, spoken language, internet language, learner
language corpora, etc) here: http://viki.keeleleek.ee/wiki/
All descriptions are in estonian only now
you can get an access to quite a decent corpora of Finnish from The Language Bank of Finalnd. For that, however, you would need to register (which is pretty simple), link here: http://www.csc.fi/english/
Other options are:
– Corpus of Institute for the languages of Finland, which contains also some older texts http://kaino.kotus.fi/korpus/
– project Gutenberg.
- BADIP: BAnca Dati dell’Italiano Parlato (http://badip.uni-graz.at/en/).
- CLIPS: Corpora e Lessici dell’Italiano Parlato e Scritto (http://www.clips.unina.it/en/index.jsp)
- CoLFIS: Corpus e Lessico di Frequenza dell’Italiano Scritto (http://linguistica.sns.it/CoLFIS/Home.htm)
- CORIS/CODIS: A corpus of written Italian – CORIS/CODIS is available on-line for research purposes (http://corpora.dslo.unibo.it/coris_ita.html)
- PAISÀ: Piattaforma per l’Apprendimento dell’Italiano Su corpora Annotati (http://www.corpusitaliano.it/en/index.html) (see also http://recremisi.blogspot.com/2013/08/corpus-italiano.html)
- Italian Corpora listed by Isabella Chiari: http://www.alphabit.net/home/index.php?option=com_content&view=category&id=37:corporaitaliani&Itemid=121&layout=default
- Corpora and other linguistic resources in Italian: http://www.societadilinguisticaitaliana.net/index.php?option=com_weblinks&view=category&id=21%3Acorpora-italiani&Itemid=21
- You can download this MS Word file for a list of Italian Corpora and Lexica
- Interesting corpus tool showing syntactic frames and slots: http://sesia.humnet.unipi.it/lexit/index.php?lang=en
1) You can find a comprehensive list of Polish corpora here:
- Portuguese Brazilian
Two corpora of human-produced dialogue summaries. It will soon also present a set of tools for corpus segmentation and annotation (specifically tailored to dialogue summary annotation), along with publications and technical documents relating to the corpora: http://www.each.usp.br/norton/resdial/index_ing.html
Here a few links for Romanian corpora:
• Various corpora compiled by Real Academia Española, all containing both European and American texts: http://www.rae.es/recursos/banco-de-datos
Some of the corpora can even be downloaded. Some of them are restricted but a lot of corpora are free. And most of the others are available as ‘sentence sets’ if the sentence is enough for you context wise (the texts are scrambled so that the sentence is intact, but not the entire text. Contact Språkbanken (http://spraakbanken.gu.se/eng/information/contact), if you would like some help picking out corpora!
CORPUS OF WRITTEN TATAR: http://corpus.tatfolk.ru/index_en.php
Taner Sezer says: I’m working on building Turkish corpora, and yet, I build four.
Respectively, they are composed of online newspapers and some other various websites (+491M tokens), Turkish Wikipedia (+47M tokens), a tweets corpus (2009-2011) composed of 1 million tweets (+13M tokens) and a small corpus of online newspapers (~500 thousand tokens).
Each corpus is PosTagged and they also include morphological tagging.
The small newspaper corpus has metada for year, genre and source. And tweets corpus has a new tag set including internet abbreviations, smileys, internet emphasis (the words where a character is written multiple times in order to emphasis the word in context) and misspelled words.
Turkish is one of the widely spoken languages. On the other hand, because of missing tools, software and lack of useable corpora for linguistic studies, Turkish take a backseat in computational linguistics.
To sum up, I’ll be very pleased if I may provide data for your study and help about Turkish as a native speaker.
Best Regards, Taner Sezer
• For Urdu language u may get corpora from http://www.cle.org.pk/clestore/
• http://www.etcml.com this is a tool for text analysis- might help you with the social media aspects of your research-and it is free
• Ricardo Campos says: you can find some interisting datasets in my webpage (tab datasets): http://www.ccc.ipt.pt/~ricardo/software.html