|
|
Outliers
Kudrnáč, Vojtěch ; Zvára, Karel (advisor) ; Anděl, Jiří (referee)
This paper concerns itself with the methods of identifying outliers in an otherwise normally distributed data set. Several significant tests and criteria designed for this purpose are described here, Peirce's criterion, Chauvenet's criterion, Grubbs' test, Dixon's test and Cochran's test. Deriving of the tests and criteria is indicated and finally the results of the use of the test and criteria on simulated data with normal distribution and inserted outlier are looked into. Codes in programming language R with the implementation of these test and criteria using existing functions are included. Powered by TCPDF (www.tcpdf.org)
|
|
|
Analysis of Outlier Detection Methods
Labaš, Dominik ; Bartík, Vladimír (referee) ; Burgetová, Ivana (advisor)
The topic of this thesis is analysis of methods for detection of outliers. Firstly, a description of outliers and various methods for their detection is provided. Then a description of selected data sets for testing of methods for detection of outliers is given. Next, an application design for the analysis of the described methods is presented. Then, technologies are presented, which provide models for described methods of detection of outliers. The implementation is then described in more detail. Subsequently, the results of experiments are presented, which represent the main part of this thesis. The results are evaluated and the individual models are compared with each other. Lastly, a method for accelerating outlier detection is demonstrated.
|
|
|
Statistical analysis of big industrial data
Zamazal, Petr ; Popela, Pavel (referee) ; Šomplák, Radovan (advisor)
This thesis deals with processing of real data regarding waste collection. It describes select parts of the fields of statistical tests, identification of outliers, correlation analysis and linear regression. This theoretical basis is applied through the programming language Python to process the data into a form suitable for creating linear models. Final models explain between 70 \% and 85 \% variability. Finally, the information obtained through this analysis is used to specify recommendations for the waste management company.
|
|
|
Image Deblurring in Demanding Conditions
Kotera, Jan ; Šroubek, Filip (advisor) ; Portilla, Javier (referee) ; Jiřík, Radovan (referee)
Title: Image Deblurring in Demanding Conditions Author: Jan Kotera Department: Institute of Information Theory and Automation, Czech Academy of Sciences Supervisor: Doc. Ing. Filip Šroubek, Ph.D., DSc., Institute of Information Theory and Automation, Czech Academy of Sciences Abstract: Image deblurring is a computer vision task consisting of removing blur from image, the objective is to recover the sharp image corresponding to the blurred input. If the nature and shape of the blur is unknown and must be estimated from the input image, image deblurring is called blind and naturally presents a more difficult problem. This thesis focuses on two primary topics related to blind image deblurring. In the first part we work with the standard image deblurring based on the common convolution blur model and present a method of increasing robustness of the deblur- ring to phenomena violating the linear acquisition model, such as for example inten- sity clipping caused by sensor saturation in overexposed pixels. If not properly taken care of, these effects significantly decrease accuracy of the blur estimation and visual quality of the restored image. Rather than tailoring the deblurring method explicitly for each particular type of acquisition model violation we present a general approach based on flexible automatic...
|
|
|
Robust estimation of autocorrelation function
Lain, Michal ; Hudecová, Šárka (advisor) ; Hlávka, Zdeněk (referee)
The autocorrelation function is a basic tool for time series analysis. The clas- sical estimation is very sensitive to outliers and can lead to misleading results. This thesis deals with robust estimations of the autocorrelation function, which is more resistant to the outliers than the classical estimation. There are presen- ted following approaches: leaving out the outliers from the data, replacement the average with the median, data transformation, the estimation of another coeffici- ent, robust estimation of the partial autocorrelation function or linear regression. The thesis describes the applicability of the presented methods, their advantages and disadvantages and necessary assumptions. All the approaches are compared in simulation study and applied to real financial data. 1
|
|
|
Implicitly weighted robust estimation of quantiles in linear regression
Kalina, Jan ; Vidnerová, Petra
Estimation of quantiles represents a very important task in econometric regression modeling, while the standard regression quantiles machinery is well developed as well as popular with a large number of econometric applications. Although regression quantiles are commonly known as robust tools, they are vulnerable to the presence of leverage points in the data. We propose here a novel approach for the linear regression based on a specific version of the least weighted squares estimator, together with an additional estimator based only on observations between two different novel quantiles. The new methods are conceptually simple and comprehensible. Without the ambition to derive theoretical properties of the novel methods, numerical computations reveal them to perform comparably to standard regression quantiles, if the data are not contaminated by outliers. Moreover, the new methods seem much more robust on a simulated dataset with severe leverage points.
|
|
|
A Nonparametric Bootstrap Comparison of Variances of Robust Regression Estimators.
Kalina, Jan ; Tobišková, Nicole ; Tichavský, Jan
While various robust regression estimators are available for the standard linear regression model, performance comparisons of individual robust estimators over real or simulated datasets seem to be still lacking. In general, a reliable robust estimator of regression parameters should be consistent and at the same time should have a relatively small variability, i.e. the variances of individual regression parameters should be small. The aim of this paper is to compare the variability of S-estimators, MM-estimators, least trimmed squares, and least weighted squares estimators. While they all are consistent under general assumptions, the asymptotic covariance matrix of the least weighted squares remains infeasible, because the only available formula for its computation depends on the unknown random errors. Thus, we take resort to a nonparametric bootstrap comparison of variability of different robust regression estimators. It turns out that the best results are obtained either with MM-estimators, or with the least weighted squares with suitable weights. The latter estimator is especially recommendable for small sample sizes.
|
|
|
How to down-weight observations in robust regression: A metalearning study
Kalina, Jan ; Pitra, Z.
Metalearning is becoming an increasingly important methodology for extracting knowledge from a data base of available training data sets to a new (independent) data set. The concept of metalearning is becoming popular in statistical learning and there is an increasing number of metalearning applications also in the analysis of economic data sets. Still, not much attention has been paid to its limitations and disadvantages. For this purpose, we use various linear regression estimators (including highly robust ones) over a set of 30 data sets with economic background and perform a metalearning study over them as well as over the same data sets after an artificial contamination.
|
|
|
Robust Metalearning: Comparing Robust Regression Using A Robust Prediction Error
Peštová, Barbora ; Kalina, Jan
The aim of this paper is to construct a classification rule for predicting the best regression estimator for a new data set based on a database of 20 training data sets. Various estimators considered here include some popular methods of robust statistics. The methodology used for constructing the classification rule can be described as metalearning. Nevertheless, standard approaches of metalearning should be robustified if working with data sets contaminated by outlying measurements (outliers). Therefore, our contribution can be also described as robustification of the metalearning process by using a robust prediction error. In addition to performing the metalearning study by means of both standard and robust approaches, we search for a detailed interpretation in two particular situations. The results of detailed investigation show that the knowledge obtained by a metalearning approach standing on standard principles is prone to great variability and instability, which makes it hard to believe that the results are not just a consequence of a mere chance. Such aspect of metalearning seems not to have been previously analyzed in literature.
|
|
|
How to down-weight observations in robust regression: A metalearning study
Kalina, Jan ; Pitra, Zbyněk
Metalearning is becoming an increasingly important methodology for extracting knowledge from a data base of available training data sets to a new (independent) data set. The concept of metalearning is becoming popular in statistical learning and there is an increasing number of metalearning applications also in the analysis of economic data sets. Still, not much attention has been paid to its limitations and disadvantages. For this purpose, we use various linear regression estimators (including highly robust ones) over a set of 30 data sets with economic background and perform a metalearning study over them as well as over the same data sets after an artificial contamination. We focus on comparing the prediction performance of the least weighted squares estimator with various weighting schemes. A broader spectrum of classification methods is applied and a support vector machine turns out to yield the best results. While results of a leave-1-out cross validation are very different from results of autovalidation, we realize that metalearning is highly unstable and its results should be interpreted with care. We also focus on discussing all possible limitations of the metalearning methodology in general.
|
guest ::
