Araneum Slovacum Slovak Web Corpus

• Crawled in March to June & November to December 2013
  by SpiderLing 0.72
• TLD restriction initially set to include the most relevant national (.sk) and generic (.eu, .com, .edu, .info, .org, .net) TLDs only (Spring crawl), later this restriction has been released (Winter crawl)
• Language similarity threshold set to 0.65, remaining non-Slovak (mostly Czech) documents removed by subsequent filtration based on character frequencies
• Tagged by Tree Tagger using the Slovak parameter file based on the strongly simplified SNK Tagset
• Native tagset mapped to Araneum Universal Tagset
• Paragraph-level deduplicated by Onion, tokens in duplicate paragraphs marked

Versions available

• Araneum Slovacum Maius: 1,200,005,746 tokens, 816,125,010 umarked words
• Araneum Slovacum Minus: 119,385,837 tokens, 90,164,126 unmarked words

Revision history

• 14.08 Filtration & dedupliction improved
• 14.04 Initial publicly released version