|
|
Automated Identification of Graphical UI Elements for Robotic Quality Assurance
Válek, Lukáš ; Herout, Adam (oponent) ; Španěl, Michal (vedoucí práce)
This thesis explores the issue of graphical user interface (GUI) screen analysis using convolutional neural networks (CNN) and computer vision methods. The thesis aims to create a system which automatically identifies GUI elements based on pictogram and text information for detected components in an input image. A combination of EfficientNetB1 CNN, OCR, and traditional computer vision methods was used to develop the system. A custom dataset which contains 120k pictograms was used to train the CNN. A UI element semantic dictionary was created, which further utilises the text detected by OCR. Finally, a GUI hierarchy analysis subsystem was created to detect and semantically categorise sections in GUI. The resulting system automatically classifies detected pictograms, suggests additional text classes, and separates the GUI screen into hierarchical sections. The system achieves 81.1% UI element identification accuracy and, on average, analyses a single screen in 0.6 seconds. This system automates repetitive processes, thus decreasing needed person-hours. In the future, the system can be further developed to function as a foundation for automated exploratory testing.
|
|
|
Automated Identification of Graphical UI Elements for Robotic Quality Assurance
Válek, Lukáš ; Herout, Adam (oponent) ; Španěl, Michal (vedoucí práce)
This thesis explores the issue of graphical user interface (GUI) screen analysis using convolutional neural networks (CNN) and computer vision methods. The thesis aims to create a system which automatically identifies GUI elements based on pictogram and text information for detected components in an input image. A combination of EfficientNetB1 CNN, OCR, and traditional computer vision methods was used to develop the system. A custom dataset which contains 120k pictograms was used to train the CNN. A UI element semantic dictionary was created, which further utilises the text detected by OCR. Finally, a GUI hierarchy analysis subsystem was created to detect and semantically categorise sections in GUI. The resulting system automatically classifies detected pictograms, suggests additional text classes, and separates the GUI screen into hierarchical sections. The system achieves 81.1% UI element identification accuracy and, on average, analyses a single screen in 0.6 seconds. This system automates repetitive processes, thus decreasing needed person-hours. In the future, the system can be further developed to function as a foundation for automated exploratory testing.
|
|
|
Aktivní učení pro klasifikaci obrázků
Lorenzová, Kateřina ; Pilát, Martin (vedoucí práce) ; Křen, Tomáš (oponent)
Práce se zabývá aplikací vybraných metod ke klasifikaci obrazu, inspi- rovanou chováním lidského oka. Využívá k analýze metody, které nevyžadují kom- pletní informaci ze zkoumaného obrazu. Místo toho používá samostatné agenty, kteří se po obraze pohybují a sami si vybírají, kterou další část obrazu potřebují vidět k upřesnění výsledku. Chování agentů je řízeno neuronovou sítí, pro tento úkol naučenou. K učení sítě byl použit evoluční algoritmus. Použitá data pocházejí z databáze obrázků ručně psaných čísel MNIST (2011). Tato sbírka obsahuje také oddělenou testovací mno- žinu dat, na které je úspěšnost agentů následně testována.
|
host ::
RSS.