|
|
Sentence representations with similarity interpretation
Svobodová, Zuzana ; Hudeček, Vojtěch (advisor) ; Libovický, Jindřich (referee)
Sentence representations - embeddings - obtained from neural network models are the core part of many applications in both academia and industry. Although embeddings reach great results in correlation with human sense of sentence similarity, there is often a lack of explanation for why models choose sentences to be similar. In this thesis, we strive to increase the interpretability of model embeddings by incorporating different semantic sentence level annotations in the learning process. We introduce a model called SBERTslice that produces embeddings that can distinguish nuanced semantic variations in text, including elements like negation, sentiment, named entities, emotional tone, and verb-oriented relation between words in a text. We evaluated SBERTslice embeddings in various text classification and semantic sim- ilarity tasks and for a majority of them, SBERTslice outperformed the original SBERT. 1
|
|
|
Understanding cross-lingual abilities in large multilingual language models
Del Valle Girón, José Jacobo ; Libovický, Jindřich (advisor) ; Limisiewicz, Tomasz (referee)
Cross-lingual abilities have been evident in large multilingual language models over the past few years. However, understanding why and under what circumstances they work is not entirely clear. In this work, we work towards a better understanding of these aspects in a specific subset of multilingual models, namely modular multilingual models with cross-lingual transfer learning abilities. We try to quantify claims in Pfeiffer et al. [2022] regarding their proposed model, X-MOD, as it was tested in a very specific setting which may not align with common low-resource settings. Specifically, we evaluate how the following factors may affect downstream performance: the amount of available pre- training data; hyperparameters such as number of training steps, checkpoint selection criteria, available overlapping lexicon. With the help of our findings, we also aim to provide guidelines on how to best use X-MOD, especially from a low-resource perspective. 1
|
|
|
Gender stereotypes in neural sentence representations
Al Ali, Adnan ; Libovický, Jindřich (advisor) ; Dušek, Ondřej (referee)
Neural networks have seen a spike in popularity in natural language processing in re- cent years. They consistently outperform the traditional methods and require less human labor to perfect as they are trained unsupervised on large text corpora. However, these corpora may contain unwanted elements such as biases. We inspect multiple language models, primarily focusing on a Czech monolingual model - RobeCzech. In the first part of this work, we present a dynamic benchmarking tool for identifying gender stereotypes in a language model. We present the tool to a group of annotators to create a dataset of biased sentences. In the second part, we introduce a method of measuring the model's perceived political values of men and women and compare them to real-world data. We argue that our proposed method provides significant advantages over other methods in our knowledge. We find no strong systematic beliefs or gender biases in the measured political values. We include all the code and created datasets in the attachment. 1
|
|
|
Neural Concept-to-text Generation with Knowledge Graphs
Szabová, Kristína ; Dušek, Ondřej (advisor) ; Libovický, Jindřich (referee)
Modern language models are strong at generating grammatically correct, natural lan- guage. However, they still struggle with commonsense reasoning - a task involving making inferences about common everyday situations without explicitly stated informa- tion. Prior research into the topic has shown that providing additional information from external sources helps language models generate better outputs. In this thesis, we explore methods of extracting information from knowledge graphs and using it as additional input for a pre-trained generative language model. We do this by either extracting a subgraph relevant to the context or by using graph neural networks to predict which information is relevant. Moreover, we experiment with a post-editing approach and with a model trained in a multi-task setup (generation and consistency classification). Our methods are evaluated on the CommonGen benchmark for generative commonsense reasoning using both automatic metrics and a detailed error analysis on a small sample of outputs. We show that the methods improve over a simple language model fine-tuning baseline, although they do not set a new state of the art. 1
|
|
|
Implicit information extraction from news stories
Kydlíček, Hynek ; Libovický, Jindřich (advisor) ; Helcl, Jindřich (referee)
This work deals with information extraction from Czech News Stories. We focus on four tasks: Publishing server, Article category, Author's textual gender and Publication day of week. Due to the absence of a suitable dataset for the tasks, we present CZEch NEws Classification dataset (CZE-NEC), one of the most extensive Czech classification datasets, composed of news articles from various sources, spanning over twenty years. Tasks are solved using Logistic Regression and pre-trained Transformer encoders. Emphasis is put on fine-tuning methods of the Transformer models, which are evaluated in detail. The models are compared to human evaluators, revealing significant superiority over humans on all tasks. Furthermore, the models are pitted against the commercial large language model GPT-3, outperforming it on half of the tasks, despite GPT-3 being significantly larger. Our work sets strong baseline results on CZE-NEC allowing for further research in the field.
|
|
|
Automatic generation of medical reports from chest X-rays in Czech
Chaloupský, Lukáš ; Rosa, Rudolf (advisor) ; Libovický, Jindřich (referee)
This thesis deals with the problem of automatic generation of medical reports in the Czech language based on the input chest X-ray images using deep neural networks. The first part deals with the analysis of the problem itself including a comparison of existing solutions from several common points of view. In order to interpret medical images in the Czech language, we present a fine-tuned Czech GPT-2 model specialized on medical texts based on the original pre-trained English GPT-2 model along with its evaluation. In the second part, the created Czech GPT-2 is used for training a neural network model for generating medical reports. The training was conducted on freely available data along with data preprocessing and their adjustment for the Czech language. Furthermore, the model results are discussed and evaluated using standard metrics for natural language processing to determine the performance. 1
|
|
|
Deep Learning a Soucially Constructed Technology
Libovický, Jindřich ; Orhan, Mehmet A. (advisor) ; Lütke Notarp, Ulrike (referee)
The presented thesis focuses on deep learning and artificial intelligence as a socially constructed technology. Unlike the traditional view which explains the emergence of the technology via the inner state of technological reality, I try to follow Bijker's theoretical framework of social construction of technology and explain the development via interests of relevant social groups (general public, technology fans, IT specialists and AI researchers) and values they attribute to the technology. For each of the groups I selected several English-language online media and analyzed their content between years 2012 and 2016. The analysis showed a shift from scientific to more technological topics in articles targeted on AI researchers and broad public. In these articles, deep learning is presented as a breakthrough technology. Articles targeted on technology fans cover the news about artificial intelligence in details, but they do not attribute any special status to the technology. Similarly to IT professionals, they consider deep learning to be a technology as any other.
|
|
| |
|
|
Multimodality in Machine Translation
Libovický, Jindřich ; Pecina, Pavel (advisor) ; Specia, Lucia (referee) ; Čech, Jan (referee)
Multimodality in Machine Translation Jindřich Libovický Traditionally, most natural language processing tasks are solved within the lan- guage, relying on distributional properties of words. Representation learning abilities of deep learning recently allowed using additional information source by grounding the representations in the visual modality. One of the tasks that attempt to exploit the visual information is multimodal machine translation: translation of image captions when having access to the original image. The thesis summarizes joint processing of language and real-world images using deep learning. It gives an overview of the state of the art in multimodal machine translation and describes our original contribution to solving this task. We introduce methods of combining multiple inputs of possibly different modalities in recurrent and self-attentive sequence-to-sequence models and show results on multimodal machine translation and other tasks related to machine translation. Finally, we analyze how the multimodality influences the semantic properties of the sentence representation learned by the networks and how that relates to translation quality.
|
|
|
Automatický expresivní čtený projev
Výkruta, Jan ; Hajič, Jan (advisor) ; Libovický, Jindřich (referee)
Expressive reading is one of possible oral presentations. The text being read is usually prose or poetry. Little has been done in research of what affects expressiveness and whether it can be generated by computers. LibriSpeech, a large scale corpus of read prose and poetry allows us to test generation of expressive reading using machine learning methods. We have focused on poetry as it is generally more expressive. We have prepared methods, that can be used to train more models as well as to prepare different data that could be fed in our learning methods. Moreover, we have developed an extendable application that takes a poem, predicts the reading, visualizes it and plays an audio record generated from the reading using a TTS system. 1
|
guest ::
