Models
multi_cased_bert_base_uk.zip — a multilingual cased base BERT model fine-tuned on a combined Ukrainian dataset (SQuAD 2.0 and SDSJ 2017) with the following parameters: epochs: 5, learning rate: 5e-5, max sequence length: 180, batch size: 16. Training used the script developed by the authors of the model (https://github.com/google-research/bert) and ran on an RTX 2080 Ti (11GB). On the squad-2.0-uk test set the model achieves 61.2% exact answers.
CLIP model (ViT-H-14 + XLM-RoBERTa-large) supporting Ukrainian for zero-shot image classification.
A family of ELECTRA-base models pre-trained from scratch on Ukrainian texts (discriminators and a generator).
Flair models for Ukrainian: named entity recognition (NER), POS tagging and contextual string embeddings.
Thanks to our annotated corpus of Ukrainian we were able to train a model for automatic recognition of named entities (people, geographic names, company names, etc.) in unseen texts. For NER we chose the MITIE library. The library is open source, and its license permits free use even in commercial projects. MITIE also delivers high accuracy by combining familiar text features with CCA embeddings. Although MITIE is written in C++, it has bindings for other languages: C, Python, Java, Matlab. Please read the MITIE documentation and usage examples before you start.
To compute the CCA embeddings we used the corpus we collected of Ukrainian news, Wikipedia articles and fiction.
Download the NER model for Ukrainian
We also built a named entity recognition model for Russian, using the annotated corpus prepared by the organizers of the Dialogue 2016 conference. For its CCA embeddings we used a corpus of Russian Wikipedia articles.
Download the NER model for Russian
The OmniGEC family of grammatical error correction (GEC) models for Ukrainian.
Transformer-based spaCy NER models for Ukrainian (RoBERTa-large), trained on the NER-UK corpus.
From the corpora we collected — news, articles, fiction, laws and legal texts — we computed the most widely used word embeddings: Word2Vec (and its improved version LexVec) and GloVe. We publish these models because computing them takes considerable time and server resources.
We built separate 300d models for every text category, and also computed them for lemmatized versions of the same corpora. If you need different model settings, take the corpora we prepared and compute the models to your needs.
| Corpus |
300d |
300d lowercase |
300d lemmatized |
300d lemmatized lowercase |
|---|---|---|---|---|
| Fiction | ||||
| News | ||||
| Ubercorpus |
With the help of Tetiana Kodliuk we created test sets, for evaluating the quality of the vectors, analogous to those that existed for English.
Quality evaluations were also performed; the results are available here.
word2vec parameters for small corpora (fiction):
./word2vec_standalone.py -size 300 -negative 7 -window 4 -threads 6 -min_count 10 -iter 5 -alpha 0.030
lexvec parameters for small corpora (fiction):
./lexvec -dim 300 -verbose 2 -negative 7 -subsample 1e-3 -window 4 -threads 6 -minfreq 10 -iterations 5 -alpha 0.030
GloVe parameters for small corpora (fiction):
MEMORY=4.0
VOCAB_MIN_COUNT=10
VECTOR_SIZE=300
MAX_ITER=15
WINDOW_SIZE=9
BINARY=2
NUM_THREADS=12
X_MAX=10:
For large corpora, minfreq is 25
fastText word vectors for Ukrainian, trained on the UberText 2.0 corpus.
State-of-the-art English-to-Ukrainian machine translation model (PEFT/LoRA on top of Mistral), with an online demo.
To expand the tone dictionary, Serhiy Shekhovtsov and Oles Petriv built a neural network model that finds words similar to existing ones using word2vec and lexvec vectors.
Examples of working with the model and its data:
Sentence-transformers model for semantic embeddings of Ukrainian sentences, based on paraphrase-multilingual-mpnet.