Automated Threshold Method for Dimensionality Detection in Multivariate Data

Abstract

ABSTRACT Multivariate methods such as principal component analysis and factor analysis have been used to interpret multivariate data. However, these statistical applications are not able to determine prior to their application whether a dimension exists within the multivariate data set since it is possible to have a dimensionless multivariate dataset. In addition, these statistical applications are method dependent, it is therefore imperative to propose an independent technique for detecting dimensionality using automated threshold settings which are thresholds generated based on the structure of the data and not by the judgement of the researcher so that these statistical applications will be for purposes of interpretation or giving meaning to the data structure. Also, the formation of dimensionality in the well-known multivariate techniques is not analytically or computationally presented. They therefore offer a leave-or-take result with no understanding of the formation of the dimensions. This study therefore filled this gap by successfully proposing an independent dimensionality detection method using three automated threshold settings that generate data specific thresholds by allowing the data structure to generate the optimal threshold for detecting dimensionality of the multivariate data set for more accurate results. The study also established the robustness of the method using Pearson’s correlation which hinges on the mean and another correlation profile that does not hinge on a statistic which is affected by extreme values, in this case order statistic which hinges on the median. The algorithm converged in all cases. Confirmatory factor analysis are carried out for confirmation of results. The proposed method completely removes the challenge of subjectivity associated with dimensionality detection, and hence is highly recommended