|
|
Low-resource Text Classification
Szabó, Adam ; Straka, Milan (advisor) ; Popel, Martin (referee)
The aim of the thesis is to evaluate Czech text classification tasks in the low-resource settings. We introduce three datasets, two of which were publicly available and one was created partly by us. This dataset is based on contracts provided by the web platform Hlídač Státu. It has most of the data annotated automatically and only a small part manually. Its distinctive feature is that it contains long contracts in the Czech language. We achieve outstanding results with the proposed model on publicly available datasets, which confirms the sufficient performance of our model. In addition, we performed ex- perimental measurements of noisy data and of various amounts of data needed to train the model on these publicly available datasets. On the contracts dataset, we focused on selecting the right part of each contract and we studied with which part we can get the best result. We have found that for a dataset that contains some systematic errors due to automatic annotation, it is more advantageous to use a shorter but more relevant part of the contract for classification than to take a longer text from the contract and rely on BERT to learn correctly. 1
|
|
|
Adaptive Handwritten Text Recognition
Procházka, Štěpán ; Straka, Milan (advisor) ; Straňák, Pavel (referee)
The need to preserve and exchange written information is central to the human society, with handwriting satisfying such need for several past millenia. Unlike optical character recognition of typeset fonts, which has been throughly studied in the last few decades, the task of handwritten text recognition, being considerably harder, lacks such attention. In this work, we study the capabilities of deep convolutional and recurrent neural networks to solve handwritten text extraction. To mitigate the need for large quantity of real ground truth data, we propose a suitable synthetic data generator for model pre-training, and carry out extensive set of experiments to devise a self-training strategy to adapt the model to unnanotated real handwritten letterings. The proposed approach is compared to supervised approaches and state-of-the-art results on both established and novel datasets, achieving satisfactory performance. 1
|
|
|
Permutation-Invariant Semantic Parsing
Samuel, David ; Straka, Milan (advisor) ; Mareček, David (referee)
Deep learning has been successfully applied to semantic graph parsing in recent years. However, to our best knowledge, all graph-based parsers depend on a strong assumption about the ordering of graph nodes. This work explores a permutation-invariant approach to sentence-to-graph semantic parsing. We present a versatile, cross-framework, and language-independent architecture for universal modeling of semantic structures. To empirically validate our method, we participated in the CoNLL 2020 shared task, Cross- Framework Meaning Representation Parsing (MRP 2020), which evaluated the competing systems on five different frameworks (AMR, DRG, EDS, PTG, and UCCA) across four languages. Our parsing system, called PERIN, was one of the winners of this shared task. Thus, we believe that permutation invariance is a promising new direction in the field of semantic parsing. 1
|
|
|
Cooperative Multi-Agent Reinforcement Learning
Uhlík, Jan ; Pilát, Martin (advisor) ; Straka, Milan (referee)
Deep Reinforcement Learning has achieved a plenty of breakthroughs in the past decade. Motivated by these successes, many publications extend the most prosperous algorithms to multi-agent systems. In this work, we firstly build solid theoretical foundations of Multi-Agent Reinforcement Learning (MARL), along with unified notations. Thereafter, we give a brief review of the most influential algorithms for Single-Agent and Multi-Agent RL. Our attention is focused mainly on Actor-Critic architectures with centralized training and decentralized execution. We propose a new model architec- ture called MATD3-FORK, which is a combination of MATD3 and TD3-FORK. Finally, we provide thorough comparative experiments of these algorithms on various tasks with unified implementation.
|
|
|
Qudratic field based cryptography
Straka, Milan ; Stanovský, David (advisor)
Imaginary quadratic fields were first suggested as a setting for public-key cryptography by Buchmann and Williams already in 1988 and more cryptographic schemes followed. Although the resulting protocols are currently not as efficient as those based on elliptic curves, they are comparable to schemes based on RSA and, moreover, their security is believed to be independent of other widely-used protocols including RSA, DSA and elliptic curve cryptography. This work gathers present results in the field of quadratic cryptography. It recapitulates the algebraic theory needed to work with the class group of imaginary quadratic fields. Then it investigates algorithms of class group operations, both asymptotically and practically effective. It also analyses feasible cryptographic schemes and attacks upon them. A library implementing described cryptographic schemes is a part of this work.
|
|
|
Entity Relationship Extraction
Šimečková, Zuzana ; Straka, Milan (advisor) ; Straňák, Pavel (referee)
Relationship extraction is the task of extracting semantic relationships between en- tities from a text. We create a Czech Relationship Extraction Dataset (CERED) using distant supervision on Wikidata and Czech Wikipedia. We detail the methodology we used and the pitfalls we encountered. Then we use CERED to fine-tune a neural network model for relationship extraction. We base our model on BERT - a linguistic model pre-trained on extensive unlabeled data. We demonstrate that our model performs well on existing English relationship datasets (Semeval 2010 Task 8, TACRED) and report the results we achieved on CERED. 1
|
|
|
Smoothness of Functions Learned by Neural Networks
Volhejn, Václav ; Musil, Tomáš (advisor) ; Straka, Milan (referee)
Modern neural networks can easily fit their training set perfectly. Surprisingly, they generalize well despite being "overfit" in this way, defying the bias-variance trade-off. A prevalent explanation is that stochastic gradient descent has an implicit bias which leads it to learn functions that are simple, and these simple functions generalize well. However, the specifics of this implicit bias are not well understood. In this work, we explore the hypothesis that SGD is implicitly biased towards learning functions that are smooth. We propose several measures to formalize the intuitive notion of smoothness, and conduct experiments to determine whether these measures are implicitly being optimized for. We exclude the possibility that smoothness measures based on first derivatives (the gradient) are being implicitly optimized for. Measures based on second derivatives (the Hessian), on the other hand, show promising results. 1
|
|
|
Analysing and Optimizing GPU Kernels with Machine Learning
Šťavík, Petr ; Kruliš, Martin (advisor) ; Straka, Milan (referee)
Graphics processing units (GPUs) were originally used solely for the purpose of graph- ics rendering. This changed with the introduction of technologies like CUDA that enabled to use graphics processors as any other computing device. However, writing an efficient program for GPUs, also called GPU kernel, is one of the most difficult programming disciplines. The latest research in the field suggests that these difficulties could be po- tentially mitigated with machine learning methods. One especially successful approach is based on the utilization of recurrent neural networks (RNNs) over different representa- tions of source code. In this work, we present two RNN-based solutions that are able to derive performance characteristics of a CUDA GPU kernel directly from its intermediate representation called PTX. We assess the applicability of our two methods in two GPU op- timization tasks. In heterogeneous device mapping task, our methods are able to achieve accuracies of around 82%, results that are slightly worse than the current state of the art. In a more challenging achieved occupancy task, where the goal is to correctly predict one out of ten classes, our two methods achieve accuracies above 50%. These promising results indicate great potential in additional research focused in a similar direction. 1
|
|
|
Crosslingual Transfer in Question Answering
Macková, Kateřina ; Straka, Milan (advisor) ; Rosa, Rudolf (referee)
Question answering is a computer science discipline in the field of natural language processing and information retrieval. The goal is to build a system that can automatically find an answer to a certain question in the text. Nowadays, there exist a lot of models trained on huge training data sets in English. This work focuses on building similar models in Czech without having any Czech training datasets. In this work, we have used SQuAD 1.1 and translated it to Czech to create training and development datasets. Then, we have trained and tested BiDirectional Attention Flow and BERT models. The best obtained result on the Czech dataset is from BERT model trained on Czech with exact match 60.48% and F1 score 73.46%. In addition, we have also trained BERT model on English dataset and we have evaluated it on Czech testing dataset without translation. We have reached exact match 63.71% and F1 score 74.78%, which is extremely good in spite of the fact that the model has not seen any Czech question answering data before. Such a model is very flexible and provide a question answering system in any language for which we have enough monolingual raw texts.
|
|
|
Multilingual Learning using Syntactic Multi-Task Training
Kondratyuk, Daniel ; Straka, Milan (advisor) ; Mareček, David (referee)
Recent research has shown promise in multilingual modeling, demonstrating how a single model is capable of learning tasks across several languages. However, typical recurrent neural models fail to scale beyond a small number of related lan- guages and can be quite detrimental if multiple distant languages are grouped together for training. This thesis introduces a simple method that does not have this scaling problem, producing a single multi-task model that predicts universal part-of-speech, morphological features, lemmas, and dependency trees simultane- ously for 124 Universal Dependencies treebanks across 75 languages. By leverag- ing the multilingual BERT model pretrained on 104 languages, we apply several modifications and fine-tune it on all available Universal Dependencies training data. The resulting model, we call UDify, can closely match or exceed state-of- the-art UPOS, UFeats, Lemmas, (and especially) UAS, and LAS scores, without requiring any recurrent or language-specific components. We evaluate UDify for multilingual learning, showing that low-resource languages benefit the most from cross-linguistic annotations. We also evaluate UDify for zero-shot learning, with results suggesting that multilingual training provides strong UD predictions even for languages that neither UDify nor BERT...
|
guest ::
RSS feed.