Report Example: Machine Learning Techniques

BAGGING

It is a learning technique in the ensemble. It is among the widely used ensemble method for classification. Their common goal is to improve the accuracy of a classifier combining single classifiers which are slightly better than random guessing. The name is derived from Bootstrap aggregating constructed from a different dataset. It is a bootstrap sample from the original one. (Webb, 2017) .

This method enjoins both aggregating and bootstrapping. A classifier with better properties is achieved when the estimate of the bootstrap data distribution parameters is robust and accurate than the traditional one. Bagging is vital especially in building a better classifier in the case of noisy observations in the training set. Better results are achieved in the ensemble as compared to single classifiers to build the final classifier. This can be understood by combining the basic classifiers also combines the advantages of each one in the final ensemble. (Alfaro, Gamez, & Garcia, 2013)

MultiBoostAB

This method utilizes the use of classifiers by improving its accuracy. The classifier is used as a subroutine to build accurate classifier in the training set. In this method, boosting, the classification system is repeated on the training data with each step attention is on different examples of this set using adaptive weights. When the process finishes, the single classifier obtained are combined into a final accurate classifier in the training set. Hence, the final classifier obtained has a high degree of accuracy in the test set. There different versions of boosting algorithms but AdaBoost is the best preferred. The Limitation of this, however, is that it can only apply to classification problems. In the boosting algorithms, the selected two which are a natural extension and most simple of AdaBoost is AdaBoost M1 and SAMME.

Random subspace

Random subspace evidence classifier (RSEC) is used to address the limitation of making use of information in whole feature space and also the subspaces. It first calculates the distance of the hyperplane for each class in both the whole feature space and the randomly generated feature subspace. Computation of the basic belief assignment is then computed according to the distances. Dempster's rule is then used to pool together the evidence represented by basic belief assignments. For each test sample, RSEC assigns the class label based on the combined belief assignment. In high dimensional data, RSEC has a good performance also the minority class of imbalanced data. (HaishengLi, GuihuaWen, ZhiwenYu, & TiangangZhou, 2013)

Rotation Forest

This method makes use of feature extraction to generate classifier ensembles. The feature set is split randomly to create the training data for a base classifier, and Principal Component Analysis applied to each of the subsets. To preserve the variability information in the data, principal components are retained. To form the new feature for a base classifier, a K axis rotation occurs first. This increases the individual accuracy and diversity within the ensemble. To enhance diversity for each base classifier, future extraction is done. The name "forest" is used because decision trees are used because they are sensitive to rotation of the feature axes. Rotation Forest ensemble constructs individual classifiers using the diversity-error diagrams which happen to be more accurate than in AdaBoost while more diversity than in Bagging. (Rodriguez, Kuncheva, & Alonso, 2016)

AdaBoost

This is an approach to machine learning that is based on creating accurate prediction rule usually by joining weak and inaccurate rules. The AdaBoost algorithm is the first practical boosting algorithm and is also the most famous. AdaBoost is applied in different applications. It is one of the best Boosting algorithms. This algorithm can be used to enhance a weak algorithm with an accuracy slightly better than random guessing. For Boosting theory, it includes deducing a tighter generalization error bound and finding a more precise weak learning condition in the multiclass problem. For AdaBoost, the stopping conditions, the way to enhance anti-noise capability and how to improve the accuracy by optimizing the diversity of the base classifiers, are good questions to be in-depth researched. ( Ying, Qi-Guang, Jia-Chen, & Lin, 2013)

LogitBoost

This type of Boosting variant can either be applied to multi-class or binary classification. When you minimize the Logistic loss, LogiBoost can be viewed as additive tree regression at the statistical point of view.

Following this setting, it is still non-trivial to devise a sound multi-class LogitBoost compared with to devise its binary counterpart. The difficulties are due to two important factors arising in the multiclass Logistic loss. The first is the invariant property implied by the Logistic loss, causing the optimal classifier output is not unique, i.e., adding a constant to each component of the output vector won't change the loss value. The second is the density of the Hessian matrices that arise when computing tree node split gain and node value fittings. Oversimplification of this learning problem can lead to degraded performance. For example, the original LogitBoost algorithm is outperformed by ABC-LogitBoost thanks to the latter's more careful treatment of the above two factors. (Sun & Reid, 2014)

SMOTE

To address the problem of class-imbalanced data which is usually biased, which is even larger for high-dimensional data. In high dimensional data, the number of variables supersedes the number of samples. This type of problem can be solved by oversampling or undersampling, as a result, we will have balanced data. Undersampling is helpful compared to oversampling. SMOTE, Synthetic Minority Oversampling Technique, is the most popular oversampling technique whose main function was to improve random oversampling. Although this method's high dimensional data has not yet been thoroughly investigated. SMOTE is mostly beneficial fork-NN classifiers in the high-dimensional data if the number of variables is reduced performing some variable selection. SMOTE for k-NN without variable selection should not be used, because it strongly biases the classification towards the minority class.

F MEASURE

Record linkage involves linking and identifying records with same entities from different databases. This is considered as a classification since it's purpose is to decide whether records are a match or not. This method is not efficient because of imbalance in record linkage problem. Rather precision and recall are used and combined into F-measure. It is the harmonic mean of precision and recall. It can also be shown as a weighted sum of precision and recall, although the weights depend on linkage method used. The major weakness of F-measure is the relative importance assigned to precision and recall should not be of the linkage method used rather that of an aspect of the problem and the user.

ROC

Receiver Operating Characteristics (ROC) are usually used for visualizing and organizing classifiers performance. They are commonly applicable to medical decision making and also common in machine learning and data mining research. The area under the ROC curve is mainly used to measure the performance of supervised classification rules. It is however applicable to the case of two classes. Which is extended by averaging pairwise comparison. (Hand & Till, 2001)

Kappa

Kappa coefficient of the agreement is used for evaluating intercoder agreement for tagging tasks.

References

Ying, C., Qi-Guang, M., Jia-Chen , L., & Lin, G. (2013). Advance and prospects of AdaBoost algorithm. Acta Automatica Sinica, 745-758.

Alfaro, E., Gamez, M., & Garcia, N. (2013). adabag: An R Package for Classi. Journal of Statistical Software, 4-5.

HaishengLi, GuihuaWen, ZhiwenYu, & TiangangZhou. (2013, June 13). Neurocomputing. Random subspace evidence classifier, pp. 62-69.

Hand, D. J., & Till, R. J. (2001). Machine Learning. A simple generalization of the Area Under the ROC curve for Multiple Class Classification problem, pp. 171-186.

Rodriguez, J. J., Kuncheva, L. I., & Alonso, C. J. (2016). Rotation Forest: A New Classifier Ensemble Method. IEEE Journals & Magazines, 1619-1630.

Sun, P., & Reid, M. D. (2014). An improved multiclass LogitBoost. Machine Language, 295.

Webb, S. C. (2017). Encyclopedia of Machine Learning and Data Mining. Boston,MA: Springer.