Regression techniques and their Applications

Motivation

Regression is a statistical measurement used to determine the strength of the relationship between one dependent variable (usually denoted by Y) and a series of other changing variables (known as independent variables). The most well known statistic of regression is R2 (R Square). During modeling a Regression problem, the following points need to be carefully considered :

• Choice of dependent and regressor variable.
• The omitted variable
• Stability of regression
• Choice of relation to be fitted
• Goodness of fit

In this blog we explore in detail two primary types of Regression techniques -Linear and Logistic Regression with reference to their limitations and practical use-cases.

This blog is structured as :

• Defining Linear Regression by explaining what type of variables it works the best.

• Stating Linear Regression use-cases along with describing different types of Linear Regression.

• Explaining the limitations of Linear Regression and how the problem can be resolved with Logistic Regression.

• Understanding different types of Logistic Regression and the advantages and disadvantages of using Logistic Regression.

• Explaining different applications of Logistic Regression in various predictive modeling.

Linear Regression 

It is used to find the relation between target and one or more predictors which uses the relationship between the predictors to determine the outcome. The dependent variable should be numeric and the response variable is a continues value. In other words, linear regression models the relationship between explanatory variables and the response variable. For example, in order to estimate the house price based upon the area of the land, linear regression can be used. It is an equation of a straight line, where the output response can take any value between -Infinity to + Infinity. In Linear Regression the dependent variable is

• Continuous
• Unbounded
• Measured on an interval or ratio scale.

Modeling with linear regression

The linear model with three predictor variables has the following general format:

where Percentage is the observed percentage, β0 is the intercept, X, Y, and Z are the predictor variables, and Cis are the coefficients.

Linear Regression Use cases:

Linear regression captures how the variation in the “dependent variable” can be captured by change in the “independent variables”.

• In Business, the dependent variable/predictor or the factor of interest can be modeled using Linear Regression. Such examples include sales of a product, pricing, performance, risk etc. Independent variables /explanatory variables as they can explain the factors that influence the dependent variable along with the degree of the impact which can be calculated using “parameter estimates” or “coefficients”.

• Linear Regression helps to build the coefficients for statistical significance by building confidence intervals around them so that the models are statistically robust and based on objective data. Negative coefficients can be interpreted to have a negative or an inverse relation with the dependent variable and positive coefficient can be said to have a positive influence.

• Linear Regression is a very powerful statistical technique and can be used to generate insights on consumer behaviour, understanding business and factors influencing profitability. Linear regressions can be used in business to evaluate trends and make estimates or forecasts. For example, if a company’s sales have increased steadily every month for the past few years, by conducting a linear analysis on the sales data with monthly sales, the company could forecast sales in future months.

• Linear regression can also be used to analyze the marketing effectiveness, pricing and promotions on sales of a product. For instance, if a company wants to know if the funds that they have invested in marketing a particular brand has given them substantial return on investment, they can use linear regression.

• The highest impact of linear regression lies in enabling us to capture the isolated impacts of each of the marketing campaigns along with controlling the factors that could influence the sales. For example, for two campaigns are run on TV and Radio in parallel, a linear regression can capture the isolated as well as the combined impact of running this ads together.

• Linear Regression can be also used to assess risk in financial services or insurance domain and create a lasting business impact. For example, a car insurance company might conduct a linear regression to come up with a suggested premium table using predicted claims to Insured Declared Value ratio. The risk can be assessed based on the attributes of the car, driver information or demographics.

Types of Linear Regression

There are different types of Linear Regression techniques described below:

1. Simple Linear Regression

This method uses a single independent variable to predict a dependent variable by fitting a best linear relationship.

2. Multiple Linear Regression

This method uses more than one independent variable to predict a dependent variable by fitting a best linear relationship, under the constraint that the predictor variables are not correlated or multicollinearity is absent.The parameters are evaluated by minimizing the cost function using Gradient Descent.

3. Linear Splines

Linear splines are used to fit the data with a piece-wise linear function to reduce the problem to Linear Regression. A linear spline, or piecewise linear function exhibits linearity at left and right and gets continuos at their intersection, as shown in the figure below. A linear spline with knots at ξk with k=1,…,K is a piecewise linear polynomial continuous at each knot, used in building a piece-wise linear function step by step.

This model can be represented as:

yi=β0+β1.b1(xi)+β2.b2(xi)+⋯+βK+3bK+3(xi)+ϵi , where

bk are basis functions comprised of the variable itself and additional variables that are a collection of truncated basis transformation functions at each of the knots.

4. Stepwise Regression

This regression model is used when we have more than one independent variable. It uses automatic procedure to select important independent variables and there is no human intervention.Stepwise Linear Regression is a method of applying Multiple Linear Regression multiple times and selecting the important variables or removing the least significant predictors each time.

5.Forward Stepwise Regression

This method is often used to provide an initial screening of the candidate variables when a large group of variables (say 100) exists and are often multi-collinear.

The approach of forward selection selects best ten to fifteen variables and then apply the all-possible algorithm to the variables in this subset. The variable selection is based on an increased value of metric R-Squared. The best variables that increases the R-Squared metric are added in each step until any of the remaining variables are significant.

6.Backward Stepwise Regression

This method starts with a model in which all candidate variables have been included. The user sets the significance level at which variables can enter the model. At each step, the variable that is the least significant is removed, such that a large value of R-Square is always retained in the model. This process continues until no nonsignificant variables remain.

7.Stepwise Selection

Stepwise regression is a combination of the forward and backward selection techniques, in which after applying forward selection, all candidate variables in the model are checked to see if their significance has been reduced below the specified tolerance level. If a nonsignificant variable is found, it is removed from the model. Stepwise regression requires two significance levels: one for adding variables and one for removing variables. It prevents infinite loop iteration by introducing a cut-off, where the probability for adding variables should be less than the cutoff probability for removing variables.

Note 1: For Backward Stepwise Linear Regression or Multiple Linear Regression to work fine, the number of observations (n) should be more than the number of variables(p). It is because we can do least squares regression only when n is greater than p. For p greater than n, least squares model is not even defined.

Limitations of Linear Regression

• Linear Regression is Limited to Linear Relationships : By its nature, linear regression only looks at linear relationships between dependent and independent variables. That is, it assumes there is a straight-line relationship between them.

• Linear Regression considers only mean of the Dependent Variable :
  Linear regression looks at a relationship between the mean of the dependent variable and the independent variables.

• Linear Regression Is Sensitive to Outliers : Outliers are data that are surprising. Outliers can be univariate (based on one variable) or multivariate.

• Data Must Be Independent : Linear regression assumes that the data are independent. That means that the scores of one subject have no relation with those of another. This creates a hard limitation in clustering applications where variables need to be clustered based on space and time.

Logistic Regression

Logistic Regression is used when that the data is linearly separable or classifiable and the outcome is Binary or Dichotomous. This can be further extended when the dependent has more than 2 categories.

The model tries to fit a two class problem like ham or spam, or malign or benign tumor (ie response variable can only take 0 or 1). A linear model ranging from ( — inf to + inf) would find it difficult to interpret the response value between 0 and 1. To represent values more than 1, double than 1, and ten times of 1 the Logit function is used, to represent 2 values at the output.

Logistic regression models the probability of the membership to a particular group or category without directly modeling the values of the dependent variable. If the dependent variable consists of only two categories, logistic regression estimates the odds outcome of the dependent variable given a set of quantitative and/or categorical independent variables.

Logistics regression analysis starts with calculating the “Odds” of the dependent variable, which is the ratio of the probability that an individual is a member of a particular group or category, p(y) divided by the probability that an individual is not a member of the group or category [1– p(y)]. It is represented as follows:

Odds = p(y) / [1 — p(y)]

However, unlike the probability values, ranging from 0 to 1, the values of the odds can theoretically range from 0 to infinity.

In order to establish a linear relationship between the odds and the independent variables in the logistic regression model, the odds need to be transformed to logit (log-odds) by taking the natural logarithm (ln) of odds. The logarithmic transformation creates a continuous dependent variable of the categorical dependent variable

The goal of logistic regression is to determine the probability of an outcome based on a set of inputs.

Modeling with logistic regression

The logistic model with three predictor variables has the following general format:

where P is the probability that the event would occur according to the model and the remaining symbols have the same meaning as in equation 1.

Types of Logistic Regression

There are different types of Logistic Regression based on the number of categories an output variable is characterized.

1. Multinomial Logistic Regression 

It is used to classify subjects into groups based on a categorical range of variables to predict behavior. For example, a survey can be conducted to aid advertising strategy where participants are asked to select one of several competing products as their favorite. It will be based on the profiles of people who are most likely to be interested in a specific product, based on the highest probability of selecting that particular product. It’s used for modeling unordered multi-category (output Y -> more than 2) dependent variable. The model can be described by the following equation:

2. Binary Logistic Regression 

It is most useful when you want to model the event probability for a categorical response variable with two outcomes. For example, its often used in credit analysis in determining the risk whether the next customer is likely to default — or not default — on a loan. Binary Logistic Regression model can be described by the following equation , where Y (output) is binary, with the assumption trials are independent, and the probability of success is the same on any given trial with the same values of X.

Multinomial and Binary Logistic Regression is estimated using Maximum Likelihood Estimation (MLE), unlike linear regression which uses the Ordinary Least Squares (OLS) approach.

• MLE is an iterative procedure, meaning that it starts with a guess as to the best weight for each predictor variable (that is, each coefficient in the model) and then adjusts these coefficients repeatedly until there is no additional improvement in the ability to predict the value of the outcome variable (either 0 or 1) for each case.

• OLS regression can be visualized as the process of finding the line which best fits the data, logistic regression is more similar to cross-tabulation given that the outcome is categorical and the test statistic utilized is the Chi Square.

3. Ordinal Logistic Regression/Proportional odds model 

It is used for modeling ordered multi-category dependent variable. Multinomial regression model discards information about the ordering thats captured by Ordinal Regression and produces an outcome in a particular category or any previous category. For example, for an ordered response variable with three categories, the possible events are defined as:

• being in group 1
• being in group 2 or 1
• being in group 3, 2 or 1.

In the proportional odds model, each outcome has its own intercept but the same regression coefficients. This means:

1. The overall odds of any event can differ, but

2. The effect of the predictors on the odds of an event occurring in every subsequent category is the same for every category. This is an assumption of the model that needs to be checked. It is often violated.

It can be described by the following equation:

Advantages and Disadvantages of Logistic Regression

• Widely used technique due to its simplicity, efficiency, easy interpretation, and usage of limited computational resources.

• Allows easy regularization of outputs to prevent overfitting, yielding probabilities as prediction results.

• Logistic Regression allows easy model updating using stochastic gradient descent.

• Logistic Regression models does not get effected to predict output probabilities on removal of variables uncorrelated to the output or multi-collinear variables.

• Logistic regression’s greatest disadvantage is fails to solve non-linear problems and it underperforms when there are multiple or non-linear decision boundaries. It fails to capture more complex relationships.

• Another important requirement for Logistic Regression to function properly is Feature Engineering as it helps to identify independent variables. Without proper identification of independent variables Logistic Regression fails to perform correctly.

• Logistic Regression can only predict a categorical outcome with discrete probability outcome.

Applications of Logistic Regression

• Logistic regression has been widely used in medical research, in the field of predictive food microbiology, to describe bacterial growth/no growth interface, in the 0–1 format.

• Logistic regression may be used to predict the risk of developing a given disease based on observed characteristics of the patient.

• Logistic regression is widely used in credit risk modeling in identifying potential loan defaulters.

• Logistic regression is used in different predictive models like the Generalized linear model which finds its usage in response modeling problems like Normal, Poisson, binomial responses and other distributions, even including hypothesis testing.

Conclusion

Here in this blog, I have explained about two basic types of Regression Techniques: Linear vs Logistic Regression, advantages and limitations of each of them. You can try out some of the popular python libraries listed here to model Linear/Logistic Regression:

• scikit-learn
• h2O.ai
• Computational Intelligence Packages (CIP) for Mathematica
• Simple Linear Regression

References:

1. https://www.ncbi.nlm.nih.gov/pubmed/11319091
2. https://www.statisticssolutions.com/what-is-logistic-regression/
3. https://www.ibm.com/analytics/learn/logistic-regression
4. https://www.theanalysisfactor.com
5. https://machinelearningmastery.com/logistic-regression-for-machine-learning/
6. https://machinelearning-blog.com/2018/04/23/logistic-regression-101/