Reviewing the 'Forward-Backward Selection' Machine Learning Paper

This report reviews the machine learning paper "Forward-Backward Selection with Early Dropping." The paper explores feature selection methods, particularly forward-backward selection algorithms (FBEDk) with early dropping, and compares them with LASSO and MMPC. It addresses the problem of selecting the most relevant input variables for predictive modeling to avoid overfitting and improve model performance. The report summarizes the problem statement, solution, experimental results, conclusions, and criticisms. The solution involves using FBEDk to speed up the forward-backward selection process while maintaining theoretical guarantees. The experimental results show that FBEDk performs comparably to other feature selection algorithms in terms of predictive performance while choosing a competitive number of variables, with a significant speed advantage. The report also highlights the advantages and disadvantages of the methods, including computational costs and interpretability. The study concludes that FBEDk is a valuable approach for feature selection, especially in high-dimensional datasets, and can be adapted to various types of data and assessment activities.

Running head: REVIEWING MACHINE LEARNING PAPER REPORT
Reviewing Machine Learning Paper Report
Name of the student:
Name of the university:
Author Note

1REVIEWING OF MACHINE LEARNING PAPER REPORT
Title: “Forward-Backward Selection with Early Dropping”
Author’s Names: Giorgos Borboudakis and Ioannis Tsamardinos
Affiliation: Computer Science Department, University of Crete, Gnosis Data Analysis
Jorurnal: Journal of Machine Learning Research 20 (2019)
Date: 1-39 Submitted 6/17; Revised 10/18; Published 1/19
Content:
 Introduction and Motivation
 Problem Statement
 Solution
 Experimental results
 Conclusions and Further Work
 Criticisms from a personal point of view
Introduction:
In the area of machine learning, the early stopping is a type of regulation. It has been utilized
for avoiding overfitting while training learners with iterative models. For instance, the gradient
descent can be considered here. The rules of early stopping give the path to the number of iteration
that could be run before the learner starts to over-fit that. An algorithm used n machine learning. The
method of forward selection has been involved in choosing a subset of various predictor variables.
These are been related to the ultimate model. The forward process can be able to be accomplished
through various steps under the setting of linear regression. This happens whenever the value of n is
been seen to be less than the value of p. Otherwise, its value is seen to be more than that of p. Again,

2REVIEWING OF MACHINE LEARNING PAPER REPORT
unlike this, the backward stepwise selection starts with the complete least squares model that
consists of p predictors and iteratively eradicates the minimum useful predict one at a time. For
performing the backward selection, one requires to be under the situation where one has more
number of observations than the variables. As p is more than the value of n, one is unable to fit the
minimum squares model. This is also not defined. As far as the machine learning is concerned, the
LASSO or “least absolute shrinkage and selection operator” has been the kind of regression analysis
process, performing the variable selection along with regularization. This is to develop the accuracy
of the prediction and interpretability of the statistical model produced.
Furthermore, the future of machine learning is associated with technologies resent around
various industries under a notable amount of software packages and part of regular lives by 2020.
This is quite impossible the predict machine learning’s future. Nonetheless, there are specific trends
in how machine learning can be currently used and the ways those cases can evolve in the upcoming
future. It is one of the necessary tools to develop and maintain the digital implementations for
upcoming years. The article reviews the case of a section of forwards and backward through early
dropping. For this, the problem statement, solution and experimental results discussed in the chosen
article are demonstrated.
Problem Statement:
The feature selection methods are able to decline number of various input variables. These
are believed to be most helpful as far as the model for forecasting the targeted variable is concerned.
Some of the issues of predictive modeling comprise a considerable amount of variables slowing
development and training models and need a huge quantity of system memory. Apart from that, the
performance of a few models degrades while including input variables never relevant to the target
variable. Two primary types of algorithms of feature selection are there. They are wrapper methods

3REVIEWING OF MACHINE LEARNING PAPER REPORT
and filter methods. The wrapper feature selection generates various models having distinct subsets of
input features. The selection of the features results in the best performing model as per the
performance metric. Besides, the methods are unconcerned with types of variables through which
they are computationally costly. For instance, RFE is an active case of the approach of the wrapper
feature selection. The article highlights that the difficulty of feature selection or the variable
selection is present under supervised learning tasks is defined as the issue to select the least-size
subset of variables. It has indications to a multivariate, optimum analytical model for the objected
consequence of the variable of interest. In this way, the task of feature selection is to alter out
various irrelevant variables and those that have been suspicious provided the chosen ones.
The main disadvantage of forwards selection is the cost of computation. For choosing k
variable, that has been performing the O(pk) testing. This for the inclusion of variables where the
number p is an overall number of variables within the input data. It has been accepted for the low-
dimension datasets and turning unmanageable with a rise in dimensionality. Here, another problem
is the fact the forward selection users from various testing issues and selection of the enormous
amount of irrelevant variables. The LASSO has the stopping criterion in the bass of L2-norm of the
coefficient of present variables. Provided with the synthetic perspective and connections present
between algorithms, it can be noted that extension to the stepwise measures like shown in the article
could be translated and applied directly with the algorithm selection described. The present study
extends the algorithm of forwards election for dealing with issues related to early dropping, which is
an easy heuristic for speeding the forward section despite sacrificing quality and then maintaining
theoretical warranty. The concept is that every iteration of forwards speech for altering variables
deemed in a conditioned manner that is autonomous of the aim delivered to the present set of
selected variables.

4REVIEWING OF MACHINE LEARNING PAPER REPORT
Solution:
Here, the effective solution provided by the paper is highlighted. Besides, its advantages, the
drawbacks are also discussed. The forward-backward selection is used here, which is a necessary
and frequently utilized available selection algorithm. This is conceptually and commonly
implemented in various kinds of data. However, the issue in supervised learning activities could be
defined as an issue to select the minimal-size subset of variables leading to a multivariate, optimal
and predictive model for the targeted outcomes of variables of interests. In this way, the task of
feature selection has been filtering out several irrelevant types of variables. They are superfluously
provided the chosen ones. Solving this process has various benefits. As per various arguments, most
essential is the discovery of knowledge through eradicating the extra variables that develop intuition
and knowing about mechanisms of data generations. Apart from that, it is also deployed to decrease
expenses to measure features to generate operational predictive models. Here, for instance, this can
decrease monetary costs and inconvenience to any patent to implement the diagnostic model by
decreasing various medical examinations. It also involves measurements needed for performing
subjects to provide the diagnosis. Again, this enhances predictive performances of the resulting
model to practice, particularly under settings of high-dimension. In the present experiment, both
non-linear and linear predictive models are used. Elastic net regularized type of logistic regression
are used as linear models. It has led to an overall five hundred combinations of hyper-parameter. It is
reminded that regularization is vital, particularly after the feature selection is made for developing
predictive performance because of different coefficients. In the form of different types of non-linear
models, the scientific and effective Gaussian support vector machines and few random forecasts are
been used.
Moreover, the article highlighted that FBEDK could resolve the problems of feature
selection. It indicates the Markov blanker of the T is to be identified. It is for the distributions

5REVIEWING OF MACHINE LEARNING PAPER REPORT
trustable for casual graphs. As far as practical implications are concerned, FBEDK might be failing
to determine Markov blanker regarding various reasons. Naturally, the distribution could never be
modeled faithfully having causal graphs. No chance is there of the way the solution could be closed
that could be an optimal solution. Nonetheless, prior comparisons have indicated that the forward
selection has been performing the finest selection of subset and has been staying in competitive
position with the LASSO. This confirms that the solutions are worthy estimates done reasonably for
the best subset explanation that can also authorize the experimental segment. Here, a much more
elusive difficult case is that the tests of coordinate independence applied are never appropriate to
detention dependencies prevailing underneath the dispersal. Here, to take as example, as every single
relation are linear and non-linear in nature, the tests are been made use of and no assurance is that
whichever of the noteworthy variables would be preferred. This is never precise to FBEDK.
Experimental results:
The summary of the outcomes has been averaged repeatedly. This has measured the AUC
and various selected variables. Here, for every algorithm, the researcher has calculated the score that
has been middling rank of that algorithm on numerous datasets. Apart from that, rank algorithm has
been then measured on the basis of that specific score. They have utilized the bootstrap-process or
measuring the probability of algorithm notably worse and better than ever competitors. This has
utilized a 95% threshold. For every selection process, LASSO-FS the best predictive performances,
outperforming statistically significant of the residual four datasets. This involves the MMPC
outperforms to the residual two and three datasets. No clear winner is there overall and the selection
depends just on the aim. As the goal line is regarding the predictive performance, effective LASSO-
FS or the measures of MMPC are required to be favored. However, as one side is interested in the
interpretability, the method from the FBEDk family comprising of small values of the k is preferred.

6REVIEWING OF MACHINE LEARNING PAPER REPORT
As far as FBS is concerned, no scenario is there where that can be preferred on the other type of
algorithm. It should be noted the outcomes are somewhat artificial. This is because the performances
of FBS and FBEDK largely relies on hyper-parameter values that are chosen for that research as
LASSO-FS has never been sensitive to such choices. Apart from that, the indication that the hyper-
parameters have been seen to be optimized as per the performance have a tendency to be favoring
means to select more variables logically. For this, it has put LASS-FS at the drawback as per
interpretability.
Conclusions and Further Work:
The study has demonstrated the prior dropping heuristic for fastening up the feature of the
forward-backward selection algorithm. This has been called as the FBEDk. The early dropping can
be seen as the simple heuristic leading to various orders of the magnitude speed-up, particularly
present under high-dimensional type of datasets. This has been still seen to be maintaining
theoretical warranties of choice of forward-backward. They have proven then FBED1 has identified
Markov blanket or optimal solution. This is as the distribution of data is faithfully determined by the
Bayesian network or the maximal ancestral plotting of the graph. This is the same as the FBS or
standard forward-backward selection. One of the essential characteristics of FBEDk has been that it
is the conventional algorithm that could be adapted to control the various types of variables like
ordinal categorical and continuous. Further, it is time-course cross-sectional data with non-linear and
linear dependencies and also various assessment activities like survival analysis, classification and
regression and so on. This is through utilizing the suitable test of conditional independence. On the
other hand, algorithms such as LASSO though being fast and performing well computationally as
per predictive performance for the common such as classification and regression, are not as standard.
On the other hand, the algorithms such as LASSO has been computationally demanding for few

7REVIEWING OF MACHINE LEARNING PAPER REPORT
issues. Through the experiments, the article investigates that the FBEDk has been behaving same to
FBS as per predictive performance along with amount of chosen variables. This has been one to two
orders of magnitude quicker. It can be compared to surplus feature selection algorithms such as the
finest examples of LASSO and MMPC. Again, the FBEDk comprise of the competitive predictive
level of performance while choosing some of the variables particularly vital as the feature selection
is made for discovering the knowledge. Again, FBEDk comprises of the competitive predictive level
performance as choosing some variables that are particularly vital as the feature selection is made for
discovering knowledge.
Criticisms from a personal point of view:
Here, an interesting outcome is that LASSO and FBEDk performing equally well as limited
to choose a similar amount of variables. That assimilated with the reality that the FBEDk has been
more common, making that attractive alternative for LASSO particularly for issues having no
effective solution for the existing LASSO problem. The great variance present in running time of the
LASSO-FS and numerous algorithms are been largely attributed for those deployments. Again,
regarding LASSO-FS, tapplication of glmset has been used that has been optimized highly and has
been written down in FORTRAN. On the other hand, for FBEDk , the MMPC and FBS are utilized
as the regression of custom logistic deployment that has been written down in Matlab. Nonetheless,
a difference of one or two orders of magnitude could be intended to be present between the similar
deployments under level language like FORTRAN, languages like C++ or C along with few higher-
level languages like the instance of Matlab. Thus one intended that the deployment at lower-level
language must be performing same as LASSO-FS. Apart from that, this has the advantage that this is
able to return complete track of resolution and could be quicker in terms of practice as the
optimization at hyper-parameters is done. The article shows through the experiment that the

8REVIEWING OF MACHINE LEARNING PAPER REPORT
suggested heuristic enhances the efficiency of computation through approximately one to two orders
of the value while choosing some or similar number of variables along with retaining the predictive
type of performances.