A Hybrid Prediction Model for E-Commerce Customer Churn Based on Logistic Regression and Extreme Gradient Boosting Algorithm

Early warning of customer churn has long been a research hotspot in the field of e-commerce. The customers that might be lost should be identified accurately through data mining and data analysis, and retained in time with effective marketing measures. Even if the customer loss is inevitable, a prediction model of customer churn can still help e-commerce enterprises to identify the causes of the loss and reduce similar losses in future.

The traditional prediction models for e-commerce customer churn are simple and rough, due to budget constraint. The customers who have not logged in or purchased any goods are considered lost. Besides, only the data on the previous orders are analyzed by these models, because of the limited capacity of technologies for data collection and storage. Other types of data on customers, e.g. the profile, browsing path, favorites and comments, are rarely taken into account in the prediction process.

The most popular customer churn prediction model is the recency, frequency, and monetary value (FRM) model [1]. The model divides the customers to different categories based on three indices: recency (when did the customer make the last purchase?), frequency (how often the customer makes purchases?) and monetary value (how much the customer spends on each purchase?), and provides e-commerce enterprises corresponding operation measures to retain the target customers and maximize their profits. However, the order data has a certain time delay than the other types of data on customers, especially the data on browsing behavior. Besides, different customers have varied repurchase cycles. Thus, the analysis of the RFM model is too single and too simple.

In this paper, more types of data are introduced to predict the customer churn in e-commerce, making the prediction more comprehensive and accurate. Specifically, a hybrid prediction model for customer churn was established based on logistics regression and extreme gradient boosting (XGBoost) algorithm. Then, multiple indices were selected from various dimensions as the basis for prediction. The hybrid model was verified through a case study on an actual e-commerce platform.

1.jpg

Supervised learning infers a function from labeled training data consisting of a set of training examples, which can correctly determine the class labels for unseen instances. Supervised learning methods can be further divided into classification and prediction methods. The most frequently used supervised learning methods include logistics regression, decision tree, k-nearest neighbor (kNN), Bayes discrimination, etc.

Unsupervised learning is a self-organized learning strategy that looks for previously unknown patterns in dataset without pre-existing labels. It helps to solve the internal relationship and similarity between features in unlabeled data. The common methods of unsupervised learning are association rules, density estimation and clustering analysis.

3.2 Logistics regression

Logistic regression is extended from linear regression. However, the target variable of logistic regression is a discrete variable, rather than a numerical value in linear regression. Logistic regression is a desirable way to deal with binary classification problems. During the regression process, each feature is multiplied by a regression coefficient, then a sigmoid function is introduced, and finally a value is outputted in the interval [0, 1] through linear regression. If the value is greater than 0.5, the label is classified as class 1; otherwise, the label is classified as class 0.

The keys steps of logistic regression include setting up the prediction function $g_{\vartheta}(x)$ to project the judgement based on the input data, constructing the loss function about the deviation of predicted output from the class of training data, and obtaining the regression function with minimal loss.

The sigmoid function plays a significant role in logistic regression. Suppose there exists a binary classification problem, with $y \in\{0,1\}$ as the output of the target variable and $z=\vartheta^{T} x+\vartheta_{0}$ as the predicted output of linear regression (the real value). Then, a step function f(z) is needed to convert z into 0/1:

$f(z)=\left\{\begin{array}{cc}{0} & {\text { if } z<0} \\ {0.5} & {\text { if } z=0} \\ {1} & {\text { if } z>0}\end{array}\right.$                           (1)

However, f(z) is discontinuous rather than differentiable and monotonic. The sigmoid function provides a tool with good differentiability and monotonicity:

$y=\frac{1}{1+e^{-z}}$                                    (2)

If z>0 and y>0.5, y is always positively correlated with z; if z<0 and y<0.5, y is always negatively correlated with z.

According to the sigmoid function, the prediction function can be constructed as:

$g_{\vartheta}(x)=\frac{1}{1+e^{-\vartheta^{T} x}}$                            (3)

The probabilities for an input x to be allocated to class 1 or class 0 can be respectively described as:

$\mathrm{P}\left(y_{i}=1 | x ; \vartheta\right)=g_{\vartheta}(x)$                      (4)

$\mathrm{P}\left(y_{i}=0 | x ; \vartheta\right)=1-g_{\vartheta}(x)$                  (5)

Considering the dichotomy of the dependent variables in logistic regression model, the loss function can be solved by the maximum likelihood estimation.

According to Eqns. (4) and (5), the probability function can be established as:

$\mathrm{P}(\mathrm{y} | \mathrm{x} ; \vartheta)=g_{\vartheta}(x)^{y}\left(1-g_{\vartheta}(x)\right)^{1-y}$                  (6)

Assuming that the samples are independent of each other, the likelihood function can be obtained as:

$L(\vartheta)=\prod_{i=1}^{s} P\left(y_{i} | x_{i} ; \vartheta\right)=\prod_{i=1}^{s} g_{\vartheta}\left(x_{i}\right)^{y_{i}}\left(1-g_{\vartheta}\left(x_{i}\right)\right)^{1-y_{i}}$   (7)

Then, the log likelihood function can be obtained as:

$\begin{aligned} l(\vartheta)=\log (L(\vartheta)) & \\=& \sum_{i=1}^{s}\left(y_{i} \log g_{\vartheta}\left(x_{i}\right)+\left(1-y_{i}\right) \log \left(1-g_{\vartheta}\left(x_{i}\right)\right)\right) \end{aligned}$     (8)

Maximum likelihood estimation aims to maximize l(ϑ). The estimation can be implemented by gradient ascend method, and the ϑ value thus obtained must be the best parameter.

The logistics regression modelling includes the following steps:

Step 1. Initialization: Determine the dependent and independent variables according to the objective, and initialize the regression equation.

Step 2. Coefficient estimation: Estimate the regression coefficient for the model.

Step 3. Equation checking: Test the significance of the regression equation by the F-value and P-value in the analysis of variance (ANOVA) table. If p-value is below the significance level, the equation passes the test; otherwise, the regression equation needs to be rebuilt based on new variables. Note that even if the regression equation passes the significance test, it does not mean that each independent variable has a significant influence on the dependent variable.

Step 4. Variable checking: Test the significance of each independent variable, remove the unimportant and insignificant variables, and rebuild the equation until the whole model and regression variable pass the significance test.

3.3 XGBoost algorithm

The XGBoost is a distributed gradient boosting algorithm based on the CART decision tree. The basic principle is to build a powerful high-precision classifier by improving the iterative performance of weak classification algorithm.

In the CART algorithm, each leaf node is allocated an input attribute, and then output a score. The output scores of all leaves are added up, forming the predicted result for each sample. Suppose K trees are available for prediction. Then, the prediction function can be established as:

$y_{i}=\sum_{k=1}^{K} l_{k}\left(x_{i}\right), l_{k} \in F, i \in[1, n]$                  (9)

where, n is the number of samples; F is the set of all regression trees; l_k is a function of F.

Then, the objective function can be written as:

$\mathrm{Obj}(\vartheta)=\mathrm{L}(\vartheta)+\delta(\vartheta)$                         (10)

where, $L(\vartheta)$ is the loss function to fit training data and evaluate the deviation between the model prediction and sample data in the training set; δ(ϑ) is the regularization term to measure the complexity of the model.

The objective function of XGBoost can be assumed as:

$\mathrm{Obj}(\mathrm{t})=\sum_{i=1}^{N} L\left(y_{i}\right)+\sum_{i=1}^{M} \delta\left(y_{i}\right)$            (11)

This function represents the maximum reduction on the target for a specific tree structure. The function value is called a structure score. The smaller the structure score, the better the tree structure.