Original title: Textové šifry jako nástroj pro lepší pochopení modelů Transformer
Translated title: Textual Ciphers as a Tool for Better Understanding the Transformers
Authors: Provazník, Jan ; Libovický, Jindřich (advisor) ; Kasner, Zdeněk (referee)
Document type: Bachelor's theses
Year: 2024
Language: eng
Abstract: [eng] [cze]
The Transformer architecture is very popular, so it is potentially im- pactful to interpret what influences its performance. We test the hypothesis that the model relies on the linguistic properties of a text when working with it. We remove interference with cultural aspects of meaning by using a character-level task with the ByT5 Transformer model. We train ByT5 to decipher sentences encrypted with text ciphers (Vigenère, Enigma). We annotate a sentence dataset with linguistic properties with published NLP tools. On this dataset, we study the relationships between the linguistic properties and the fine-tuned ByT5 decipherment error rate. We analyze correlations, train ML models to predict error rates from the properties and interpret them with SHAP. We find small significant correlations but can- not predict error rates from the properties. We conclude the properties we identified do not give much insight into the performance of the Transformer. Architektura Transformer je velmi popula ́rnı ́, takz ̌e mu ̊z ̌e by ́t potencia ́lne ̌ vy ́znamne ́ interpretovat, co ovlivn ̌uje jejı ́ vy ́kon. Testujeme hypote ́zu, z ̌e mo- del se pr ̌i pra ́ci s textem spole ́ha ́ na jeho lingvisticke ́ vlastnosti. Abychom eli- minovali vliv kultury na vy ́znam, pouz ̌ı ́va ́me u ́lohu pracujı ́cı ́ na u ́rovni znaku ̊ s Transformer modelem ByT5. Dotre ́nujeme ByT5-small na des ̌ifrova ́nı ́ ve ̌t zas ̌ifrovany ́ch pomocı ́ textovy ́ch s ̌ifer (Vigene ̀re, Enigma). Anotujeme eva- luac ̌nı ́ dataset ve ̌t pomocı ́ publikovany ́ch na 'stroju ̊ pro NLP. Na evaluac ̌nı ́m datasetu zkouma ́me vztahy mezi lingvisticky ́mi vlastnostmi a c ̌etnostı ́ chyb dotre ́novane ́ho ByT5 pr ̌i des ̌ifrova ́nı ́ ve ̌t. Analyzujeme korelace, tre ́nujeme ML modely na predikci c ̌etnosti chyb ve ̌ty z jijı ́ch lingvisticky ́ch vlastnostı ́ a interpretujeme du ̊lez ̌itost vlastnostı ́ pomocı ́ SHAP. Nacha ́zı ́me male ́ signifi- kantnı ́ korelace, ale predikce c ̌etnosti chyb z vlastnostı ́ selha ́va ́. Dospı ́va ́me k za ́ve ̌ru, z ̌e identifikovane ́ vlastnosti neposkytujı ́ vhled do vy ́konu Transfor- meru ̊.
Keywords: Transformer|interpretability|NLP|deep learning|ciphers; Transformer|interpretovatelnost|NLP|deep learning|šifry

Institution: Charles University Faculties (theses) (web)
Document availability information: Available in the Charles University Digital Repository.
Original record: http://hdl.handle.net/20.500.11956/192072

Permalink: http://www.nusl.cz/ntk/nusl-622812


The record appears in these collections:
Universities and colleges > Public universities > Charles University > Charles University Faculties (theses)
Academic theses (ETDs) > Bachelor's theses