Luis Rodriguez-Benitez | José Ángel Martín Baos

Scalable kernel logistic regression with Nyström approximation: Theoretical analysis and application to discrete choice modelling

Fri, 07 Feb 2025 00:00:00 +0000

Enhancing the convergence speed of line search methods: Applications in Neural Network training

Sun, 30 Jun 2024 00:00:00 +0000

Abstract

The training of machine learning models, such as neural networks, relies on optimisation techniques that necessitate large volumes of data. The algorithms that have demonstrated satisfactory performance on this task frequently use linear searches on subsets of the data. This ensures that, despite the potential low quality of the search direction, the overall computational cost remains low, making this strategy globally efficient. In these methods, strategies employing a constant learning rate have proven to be particularly effective. This paper introduces a novel scheme designed to significantly expedite the convergence process of line search-based methods. Our approach incorporates additional high-quality linear searches derived from the convergence process of the methods, and by making use of an Armijo rule, it dynamically adjusts the step size through successive reductions or expansions based on the evaluated quality of the descent direction. This strategic adjustment enables more substantial progress in the search direction, potentially reducing the number of iterations needed to reach an optimal solution. We have applied our proposed solution to accelerate the performance of widely-used algorithms such as Gradient Descent (GD), Momentum GD, and Adaptive Moment Estimation (Adam). To illustrate the practical implications and effectiveness of our approach, we present a comprehensive case study focusing on the training of Deep Neural Networks and Kernel Logistic Regression.

PyKernelLogit: Penalised maximum likelihood estimation of Kernel Logistic Regression in Python

Fri, 01 Mar 2024 00:00:00 +0000

A prediction and behavioural analysis of machine learning methods for modelling travel mode choice

Wed, 01 Nov 2023 00:00:00 +0000

Nyström-based approximations for kernel logistic regression: Application to transport choice modelling

Mon, 17 Jul 2023 00:00:00 +0000

Abstract

The application of machine learning techniques, more specifically kernel-based techniques, to discrete choice modelling using large datasets is limited by the great number of parameters to be considered when building the kernel matrix and the size of the kernel matrix itself. The spatial and temporal complexity is such that these methods are not applicable to large sample sizes. However, there are techniques that allow generating a low-rank matrix approximation to the kernel matrix, one of them is the Nyström method. One limitation of the Nyström method is that the quality of the kernel matrix approximation depends on the proper choice of landmark points. In this work, four variants of this technique are implemented, a basic uniform method, one based on the K-means algorithm and two different implementations of a non-uniform method based on leverage scores. Later, in the experimentation, we conduct a comparison of these methods applied to two big transport mode choice datasets, which contain a large number of samples and variables. Finally, these results are compared with Multinomial Logit and other techniques currently relevant in the Machine Learning field.

Optimization techniques for Kernel Logistic Regression on large-scale datasets: A comparative study

Wed, 28 Jun 2023 00:00:00 +0000

Abstract

In recent years, machine learning techniques have been increasingly applied to modelling the decision-making processes of individuals. One technique that has shown good results in the literature for modelling complex behaviours is the Kernel Logistic Regression (KLR). However, standard KLR implementations have a time complexity of 𝒪(𝑛^{3), which is not feasible for large datasets. To overcome this limitation, one of the purposed alternatives is to approximate the kernel matrix using the Nyström method. The aim of this work is to evaluate the Nyström KLR model on large-scale datasets and to study, at the experimental level, which of the optimisation techniques that allow training this model is the most efficient. As results, the authors show that the Nyström method efficiently computes the objective function and its gradient, enabling the training of KLR models with up to 10}5 parameters. Then, it is evaluated the performance of several optimisation methods, including gradient descend, Momentum, Adam, and L-BFGS-B. It can be concluded that L-BFGS-B is the most efficient method for training the Nyström KLR model. However, given enough computational time and proper hyperparameter tuning, the Adam method can also yield good results.

image title

IoT based monitoring of air quality and traffic using regression analysis

Sat, 01 Jan 2022 00:00:00 +0000

A Python package for performing penalized maximum likelihood estimation of conditional logit models using Kernel Logistic Regression

Wed, 08 Dec 2021 00:00:00 +0000

A Python package for performing penalized maximum likelihood estimation of conditional logit models using Kernel Logistic Regression

Tue, 06 Jul 2021 00:00:00 +0000

DOI:

Abstract

In the last few years, Machine Learning (ML) methods have acquired great popularity due to their success in numerous applications such as autonomous cars, image and voice recognition systems, automatic translation systems, etc. This success has led to an increase in the use of ML methods and the extension of their applications to areas such as transport planning.

One of the main tasks within transport planning is the analysis of transport demand. To do so, it is necessary to analyse the way in which users make their decisions about the trips they make and, therefore, be able to predict the number of passengers on the transport network in relation to respect to interventions made on the transport system. Consequently, transport policies and plans can be evaluated according to the behaviour of the passengers. Discrete choice models based on random utility maximization have been developed over the last four decades and currently they have acquired a high degree of sophistication, becoming the canonical tool for transport demand analysis. Nowadays, the use of ML methods could provide an alternative to discrete choice models, as they offer a high level of accuracy in their predictions. In addition, the analyst is relieved from the need of specifying the functional expressions for the utility functions beforehand.

A Python software package called PyKernelLogit was developed to apply a ML method called Kernel Logistic Regression (KLR) to the problem of predicting the transport demand. This package allows the user to specify a set of models using KLR and the estimation of those using a Penalized Maximum Likelihood Estimation procedure. Moreover, this tool also provides a set of indicators for goodness of fit and the application of model validation techniques. Finally, it allows to obtain the willingness to pay or value of time indicators commonly used in transport planning.

Revisiting kernel logistic regression under the random utility models perspective. An interpretable machine-learning approach

Tue, 16 Mar 2021 00:00:00 +0000

A python package for performing penalized maximum likelihood estimation of conditional logit models using kernel logistic regression

Wed, 24 Jun 2020 00:00:00 +0000

Abstract

Discrete choice modeling using Kernel Logistic Regression

Wed, 01 Jan 2020 00:00:00 +0000

Discrete choice modeling using Kernel Logistic Regression

Wed, 18 Sep 2019 00:00:00 +0000

Abstract

The Kernel Logistic Regression is a popular technique in machine learning. In this work this tech- nique is applied to the field of discrete choice modeling. This approach is equivalent to specifying non-parametric utilities in random utility models. A Monte Carlo simulation experiment has been carried out to compare this approach with Multinomial Logit models, comparing the goodness of fit and the capability of obtaining the specified utilities.

A methodology for monitoring traffic flow and air pollution in urban areas

Mon, 29 Jul 2019 00:00:00 +0000

Abstract

Road transportation has become the main source of air pollution in urban areas, which has a major impact on local air quality and human health. For this reason, it is increasingly necessary to accurately estimate the contribution of road transport to air pollution in the cities, so that pollution-reduction measures can be properly designed and implemented appropriately. These pollution reduction measures are becoming increasingly important due to the continued growth in vehicle use and the deterioration of driving conditions (traffic congestion). Authorities find it difficult to meet their environmental objectives and, therefore, reliable mathematical emission models are needed to accurately predict the impact of road transport on air pollution.

Nowadays, intelligent cities are essential to prevent high-level pollution situations and to act when such situations occur. Cities must anticipate pollution peaks and take mitigating measures, such as restricting traffic to a certain number of vehicles or according to their license plates, closing traffic on certain streets, reducing speed limits, etc. In addition, traffic flows must be monitored as they affect pollution levels in that city.

Typical pollution monitoring and control systems often consist of large and expensive devices that are only limited to a few points in the city, hence, they provide information for vast areas and sometimes these systems are not scalable. However, cities are distributed environments where events occur in real time and on a massive scale. Therefore, cities should rely on a low-cost Internet of Things (IoT) infrastructure connected to a cloud platform that supports this type of systems, as well as sensor-based big data applications. These pollution control systems can be combined with a traffic monitoring infrastructure to provide a complete system that can be used as a Decision Support System (DSS) to help authorities make decisions about the environmental impacts caused by pollution before they occur.

In this work, a methodology has been developed for determining the traffic flow and the air pollution in several streets of a city using low-cost devices. An artificial intelligence heuristic algorithm has been employed to process video images from a camera and estimate the traffic flow in the street. This algorithm uses statistical techniques to process the motion vectors generated by the GPU when the video is encoded. Moreover, the computational time required by the algorithm is in the order of 2.5 milliseconds, which allows to count the number of vehicles in real time. This algorithm has been tested proving 90% of accuracy. Finally, this information is uploaded to a cloud service where machine learning techniques can be applied to predict the pollution levels in the city or recommend palliative actions.