Optimization techniques applied to machine learning model development

Fri, 11 Oct 2024 00:00:00 +0000

This PhD thesis was developed by Jose Carlos Garcia Garcia, under the supervision of Dr. Ricardo Garcia Ródenas and Dr. Jose Angel Martín Baos at the University of Castilla-La Mancha. The thesis was defended on October 11, 2024.

Photo Gallery

Abstract

Machine learning (ML) techniques have emerged as essential tools for solving a wide range of challenges, while also providing critical insights that significantly improve decision-making processes within organizations. These insights are derived from the extensive volumes of data produced by users on a daily basis. Among the various ML methods, clustering is used in data analysis to group data points into clusters, where each cluster contains elements with similar characteristics. These algorithms help identify patterns and structures within the data. The Density Peak Clustering (DPC) method stands out as a clustering algorithm known for its efficiency in identifying clusters with varying densities and non-convex shapes. However, the dependence of DPC on the manual selection of cluster centers and the tuning of key parameters presents significant challenges, particularly when applied to unfamiliar or complex datasets.

To address these limitations, this thesis proposes an optimization-based methodology that automates the selection of cluster centers and parameter configuration in DPC. The approach utilizes internal and external cluster validity indices, such as Gaussian entropy and the V-Measure, respectively, to guide the process of cluster center selection and parameter tuning. Numerical experiments conducted on real-world datasets validated the proposed methodology, showing that the automatic tuning of DPC and Fuzzy Weighted K-Nearest Neighbor DPC (FKNN-DPC) outperforms traditional methods in terms of efficiency and accuracy.

Moreover, the data collected from various sources often includes additional or supplementary labeled information, which can be used to enhance the value of machine learning methods. These labels may include socioeconomic factors or geographic data, providing richer context for analysis. In response to this, the present doctoral thesis introduces a novel piece-wise machine learning approach that incorporates these additional labeled variables into the analysis of time series data. This methodology was applied to electricity consumption data, where a bi-level optimization problem was formulated. The proposed approach enabled the segmentation of data according to geographical regions, allowing for the adjustment of regression models within each region. The piece-wise autoregressive model demonstrated lower relative error compared to traditional models, highlighting the effectiveness of this method in improving prediction accuracy and identifying patterns among electrical consumers.

To address the high computational cost associated with the previous optimization problem, this doctoral thesis investigates the application of Response Surface Methods (RSMs), particularly within parallel computing environments, using a cooperative framework for parallel RSM algorithms, defining a class referred to as Sequential Multi-point Infill Sampling Algorithms (SMISAs). This framework improves the robustness of the optimization process by enabling the sharing of solutions across multiple algorithms. The proposed algorithm, named CPEI, which integrates a radial-basis-based algorithm (CORS-RBF) with a kriging-based method (EGO-PEI), was evaluated using benchmark functions. The results demonstrated that CPEI not only reduces computational time but also improves convergence properties compared to traditional methods.

Overall, this doctoral thesis makes significant contributions to the development and application of optimization techniques for building machine learning models, offering methodologies that are both innovative and practical for addressing complex, high-dimensional problems in the field.

A comparative study of machine learning, deep neural networks and random utility maximization models for travel mode choice modelling

Fri, 11 Mar 2022 00:00:00 +0000

A comparative study of machine learning, deep neural networks and random utility maximization models for travel mode choice modelling

Wed, 08 Sep 2021 00:00:00 +0000

Abstract

Traditionally, Random Utility Maximization (RUM) models have been widely applied to travel mode choice modelling. Currently, Machine Learning (ML) models are being applied as an alternative to RUM models, since they provide better results in terms of prediction capability and they can manage large volumes of data. In this paper, a comprehensive comparison between classic RUM models and ML models, including single and ensemble classifiers as well as Deep Neural Networks (DNNs), is provided in order to assess systematically the performance of different models over two different datasets which have different sizes and nature of data. Numerical experiments show Random Forest (RF) is the best classifier in terms of accuracy index and the computational cost to train the model.

A comparison of general-purpose optimization algorithms for finding optimal approximate experimental designs

Wed, 01 Apr 2020 00:00:00 +0000

José Carlos García-García | José Ángel Martín Baos

Optimization techniques applied to machine learning model development

Photo Gallery

Abstract

A comparative study of machine learning, deep neural networks and random utility maximization models for travel mode choice modelling

A comparative study of machine learning, deep neural networks and random utility maximization models for travel mode choice modelling

Abstract

A comparison of general-purpose optimization algorithms for finding optimal approximate experimental designs