Exploring the synergy of guided numeric and text analysis in e-commerce: a comprehensive investigation into univariate and multivariate distributions

E-commerce analysis

E-commerce analysis is a research domain that explores the data and information generated by e-commerce platforms to understand consumer behavior, market trends and various other aspects that affect the success of online businesses. In the context of this research, the focus is on anonymized e-commerce platforms. E-commerce analytics is becoming increasingly crucial due to the increasing popularity of online shopping and the importance of understanding customer preferences and needs. At a basic level, e-commerce analysis involves collecting, processing, and interpreting numerical and text data to identify patterns, trends, and opportunities that can help companies improve customer service and experience.

In this approach, this research utilizes Python and NLP technologies to combine numerical and text analysis in a guided manner. For example, previous research Hernández, Jiménez & Martín (2011), Han, Kim & Lee (2018), Singh & Srivastava (2019), Khan et al. (2021) and Virdi, Kalro & Sharma (2020) can provide insight into how the integration of these analyses can enrich the understanding of consumer behavior and improve the responsiveness of e-commerce platforms. A targeted approach to e-commerce analysis can yield more holistic information, helping to overcome the challenges of approaching multidimensional data.

In this study, univariate distribution is the focus in analyzing numerical trends from customer review data. Previous research from Tian & White (2020), Alzahrani et al. (2022), Singh et al. (2022) and Pandiaraja et al. (2022) shows that univariate analysis can provide substantial insight into product preferences and customer satisfaction. However, this approach can sometimes fall short in understanding the complex relationships between variables. Therefore, this study also explores multivariate distribution to provide a deeper understanding of the interactions between relevant variables in the context of e-commerce customer reviews.

The importance of customer reviews insights in e-commerce analysis cannot be ignored. Previous studies from Virdi, Kalro & Sharma (2020), Asokan-Ajitha (2021), Meng et al. (2021) and Shankar, Jebarajakirthy & Ashaduzzaman (2020) highlighted that customer reviews can be a critical source of information in formulating business strategies. Therefore, by focusing the analysis on customer reviews from the platform, this research seeks to further understand customer preferences, expectations, and needs that can be strategically implemented to improve services and products.

In the context of this research, the findings from numerical and text analysis are expected to provide concrete insights for improving online shopping experience and customer satisfaction. Through the integration of univariate and multivariate distribution, as well as the utilization of customer reviews insights, it is expected that this research can make a significant contribution in filling the knowledge gap in e-commerce analysis, creating a foundation for more effective action plans in the context of e-commerce.

Guided numeric and text exploration

Guided numeric and text exploration refers to a data analysis approach that integrates numeric and text analysis in a targeted manner to gain deeper understanding. In the context of this research, this technique is applied to explore customer reviews from e-commerce platforms holistically. This approach provides a methodological foundation that can provide significant advantages in capturing insights from diverse and complex data sources.

The importance of this directed approach lies in its ability to guide the analysis of numerical and text data to complement each other. For example, previous research Wang (2021), Marichal & Neve (2020) and Zhu & Li (2022) has shown that by using this technique, researchers can identify correlations that might be missed if only analyzed separately. With good guidance, numerical analysis can provide context for text data, and vice versa, creating a fuller understanding of customer dynamics.

Guided numeric and text exploration also has the advantage of overcoming the challenges of multidimensional data analysis. With the help of Python and NLP technology, this research was able to capture the diversity of information from customer reviews and direct the focus of the analysis to more relevant aspects. This is in line with the findings of previous research (Liu et al., 2020) which emphasized that in the face of complex data, the success of analysis depends on the appropriateness of the methods used to guide data exploration.

In the context of this research, there is an attempt to combine numerical and text approaches with a focus on univariate and multivariate distributions. This step is important because previous research Hu et al. (2020), Sun et al. (2020), Alvarez-Garcia & Yaban (2020), Mardanshahi et al. (2020), Siddique (2024), Suryaputra Paramita (2024) and Mu (2024) shows that the integration of numerical and textual analysis in the context of e-commerce has not been comprehensively explored. This research utilizes guidelines to ensure that univariate analysis can provide an in-depth understanding of single variables, while multivariate analysis reveals complex relationships among interrelated variables.

Univariate distribution

Univariate distribution is a statistical concept that explores the distribution of data from a single variable. In this research, the focus on univariate distribution aims to deeply understand the numerical trends of customer reviews on anonymized e-commerce platforms. Univariate distribution can provide an overview of the central characteristics of a variable and help identify patterns that may be contained in the data.

The basic formula of a univariate distribution is a probability density function (PDF) that describes the relative distribution of possible values of a random variable. For continuous random variables, the PDF is represented by a mathematical function, while for discrete random variables, the distribution is described by a probability mass function (PMF). (1)${\displaystyle f\left(x\right)=P\left(X=x\right)\left(Fordiscretevariables\right)}$ (2)${\displaystyle f\left(x\right)\approx Li{m}_{\Delta x\to 0}\frac{P\left(x\le X\le x+\Delta x\right)}{\Delta x}.}$

Univariate distribution analysis enables an understanding of the distribution and frequency of possible values of a particular variable. Xu & Chan (2019), Umar & Gray (2023) and Mohanty, Gopalkrishnan & Mahendra (2021) show that through univariate distributions, researchers can identify specific patterns in customer behavior, such as specific product preferences or trends in review ratings.

In the context of this research, univariate distributions are used to analyze numerical data from customer reviews, identifying trends and variations in product ratings or specific aspects. These distributions can provide a strong picture of customer preferences that may be the basis for product improvement or marketing strategy development.

It is important to note that univariate distributions are the first step in understanding numerical data and do not cover the complex relationships between variables. Therefore, this research also incorporates multivariate distributions to gain a deeper understanding of the interactions between the various factors that influence customer reviews.

In order to assess the impact of univariate distributions in this study, Assad, Cara & Ortega-Mier (2023), Abatzoglou, Dobrowski & Parks (2020) and Eum, Gupta & Dibike (2020) contributed by showing that univariate distribution analysis can help identify outliers that may affect the overall analysis. In conclusion, the univariate distribution became a key element in the initial understanding of numerical data in this study, forming a foundation for further exploration in multivariate distributions.

Multivariate distribution

Multivariate distribution is a statistical concept that extends the notion of univariate distribution into domains involving two or more variables. In the context of this research, multivariate distribution is used to analyze the simultaneous relationships between multiple variables in customer review data from anonymized e-commerce platforms. Multivariate distributions provide an overview of the joint distribution of these variables and can provide insight into the complex patterns of relationships that may exist between them.

The basic formula of multivariate distribution can be represented by the PDF for two continuous random variables. For example, for two random variables X and Y, the probability density function (f(x,y)) gives an idea of their joint distribution. (3)${\displaystyle f\left(x,y\right)=P\left(X=x,Y=y\right)}$ (4)${\displaystyle Cov\left(X,Y\right)=E\left[\left(X-{\mu }_X\right)\left(Y={\mu }_y\right)\right].}$

In multivariate distribution analysis, covariance and correlation are often the main focus. Covariance (Cov(X,Y)) measures the linear relationship between two variables, while correlation (Cov(X,Y)) measures the strength and direction of this relationship. The formula is as follows: (5)${\displaystyle Cov\left(X,Y\right)=\frac{Cov\left(X,Y\right)}{{\sigma }_X{\sigma }_Y}.}$

Umar & Gray (2023), Assad, Cara & Ortega-Mier (2023) and Liboredo et al. (2021) show that through multivariate distribution analysis, researchers can identify relationships that may not be apparent when only univariate analysis is performed. By understanding the covariance and correlation between variables, the research provides insight into how variables interact, helping to identify factors that may influence customer reviews.

In the context of this research, multivariate distributions are used to provide a deeper look into the complexity of the relationships between variables in customer reviews. For example, are certain ratings related to the frequency of certain words in the review? Is there a correlation between ratings of product quality and customer service? Multivariate distributions help answer these kinds of questions.

The importance of multivariate distribution in this research lies not only in understanding the relationship between variables, but also in devising improvement or development strategies based on the information found. Through a deeper understanding of the complexity of the data, this research is expected to make a real contribution to improving the quality of services and products on e-commerce platforms.

Customer reviews insights

Customer review insights play a key role in understanding customer preferences, satisfaction and needs on e-commerce platforms. Customer reviews not only provide direct feedback, but also include contextual understanding that can provide deep insights to companies. In this research, we focus on insights from customer reviews on anonymized e-commerce platforms.

Customer reviews provide a first-hand perception of the customer’s experience with products and services. This information covers both positive and negative aspects, including product preferences, service quality, disappointments, and customer expectations. Through analyzing customer reviews, this research aims to identify common patterns, themes, and trends that can provide strategic insights.

It is important to understand that customer review analysis involves both text and sentiment aspects. NLP plays a key role in extracting meaning from customer reviews. This technology enabled this research to identify key words, major themes, and sentiments that may be contained in customer reviews.

In the context of previous research Salunkhe, Rajan & Kumar (2021), Lai, Wang & Wang (2021) and Bhattacharyya & Dash (2021), similar studies have shown that the analysis of customer reviews can help companies recognize areas where they can improve their services or products. Such research contributes to the understanding of how information from customer reviews can be integrated in business strategy.

Customer review insights are not only useful for evaluation of current products and services but can also shape future product development and marketing strategies. Customer reviews reflect consumer needs and preferences that can serve as a guide for improvement or new development. This research seeks to utilize this information to formulate an action plan that can be implemented at the operational level of e-commerce.

With deep insights from customer reviews, this research can provide strategic information that can help companies to continuously improve service quality and understand market needs. Through text and sentiment analysis, this research is expected to provide a holistic and contextual understanding of customer reviews, creating a strong foundation for informed decision-making in the e-commerce environment.

Data collection

For this study, a comprehensive dataset was collected from Kaggle, a well-known platform for open datasets. The dataset, named “E-Commerce Reviews”, includes a substantial volume of information, with approximately 23k entries. This dataset is a valuable resource for exploring the dynamics of e-commerce customer reviews, providing both numerical and text data for analysis. The presence of numerical features, such as product ratings, provides the basis for univariate distribution analysis, while textual information in customer reviews enables the application of NLP techniques, allowing for deeper exploration of sentiment and insights. The anonymized nature of the platforms in the dataset ensures privacy compliance, in line with ethical considerations regarding data use and confidentiality. More information and access to the dataset can be found at: https://www.kaggle.com/datasets/nicapotato/womens-ecommerce-clothing-reviews. To ensure the integrity of the dataset and readiness for analysis, meticulous data preprocessing was conducted. This included:

1. Handling missing values: Missing values in numerical features were imputed using mean or median values, while missing values in categorical features were filled with the most frequent category or a placeholder value.

2. Text cleaning: Customer reviews were cleaned by removing HTML tags, punctuation, and special characters. Common English stop words were also removed.

3. Tokenization and lemmatization: Text data were tokenized into individual words, and lemmatization was applied to reduce words to their base forms.

4. Standardization of numerical features: Numerical features such as product ratings were standardized to have a mean of zero and a standard deviation of one.

5. Anonymization: Any potentially identifiable information was anonymized to ensure privacy and compliance with ethical standards.

To ensure the integrity of the dataset and readiness for analysis, an important step was taken through meticulous data pre-processing (Ran, 2023; Qi & Cao, 2023; Saputra, Rahayu & Hariguna, 2023; An, 2022). This includes handling of missing values, cleaning of text data, standardization of numerical features, and further, anonymization of possibly identifiable information. This careful pre-processing ensures the quality of the dataset and sets the stage for robust statistical analysis. The size and diversity of the “E-Commerce Reviews” dataset, along with its nature of containing both numerical and text data, makes it a solid foundation for exploring the synergies between numerical and text-guided analysis in the e-commerce domain.

Research steps

This research uses a research flow that is structured from the beginning to the end of the research, the flow of this research can be seen in Fig. 1.

Univariate and multivariate distributions

Multivariate analysis

The multivariate analysis stage in this study provides a deeper understanding of the relationships between variables and the complexity of descriptive statistics (Al-shahrani & Al-garni, 2022; Nordat, Tola & Yasin, 2022). First, attention was focused on comparing the average ratings of products based on recommendations, showing that recommended products tended to receive higher ratings. This analysis was extended to clustering by Clothing ID, revealing that product popularity did not significantly affect their average rating, but there was a strong positive correlation between ratings and recommendations. Furthermore, a correlation analysis was conducted based on product category, showing that there is a positive relationship between the average age of the buyer and the likelihood of the product being recommended. Products in certain categories, such as “Casual bottoms” and ”Chemises”, have a high likelihood of being recommended, strengthening the relationship between age preference and recommendation. These results provide greater insight into the factors that influence consumers’ perceptions of e-commerce products. Figure 2 shows the average ranking by recommendation, while Fig. 3 is a correlation matrix that provides a more detailed visualization of the relationship between variables.

Further analysis was conducted on Clothing ID, highlighting products with low ratings and unfavorable recommendation rates. However, the low frequency of reviews on these products suggests that negative reviews may be outliers and not representative of general consumer opinion. This analysis can provide strategic direction for improving product quality or customer service, especially for products with low recommendation rates. By understanding the complex relationships between these variables, e-commerce businesses can optimize marketing strategies, increase customer satisfaction, and identify potential areas for business performance improvement. Overall, multivariate analysis opens the door to deep insights into consumer dynamics and helps businesses make smarter decisions. Figure 4 illustrates the Clothing ID analysis with an emphasis on products with low ratings and recommendations.

Working with text and sentiment analysis

In the text analysis and sentiment analysis stages, the research focused on an in-depth exploration of customer reviews (Al-Jedibi, 2022; Rakhmansyah et al., 2022). The initial process involved text processing by tidying up the reviews using various techniques such as tokenization, removal of unimportant words, and application of PorterStemmer. This step was essential to clean up the dataset and make it ready for further analysis. Sentiment analysis was performed using the Sentiment Intensity Analyzer, which generates positive, neutral, and negative scores for each review. Reviews were then categorized into three sentiments based on the polarity score, allowing for a better understanding of customer attitudes towards the product. Visualization of the sentiment distribution reveals that most reviews tend to have positive sentiments. Figure 5 is the sentiment distribution on customer reviews.

At a later stage, the relationship between sentiment and other variables was explored. It was found that reviews with positive sentiment were more likely to give high ratings, while reviews with neutral and negative sentiment had more ratings in the middle range. Sentiment analysis was also extended to understand the interactions with recommendation status, department name, and product rating. The correlation matrix presents a more in-depth relationship between variables, highlighting the negative correlation between positive feedback count and positive score. Overall, text and sentiment analysis provides rich insights that can assist companies in improving their understanding of customer perceptions of products and their shopping experience. Figure 6 is the relationship between sentiment and product ratings.

N-grams by recommended feature

N-gram analysis of the non-recommended product reviews revealed key themes that emerged among the negative reviews. Product fit (sizing) was a major highlight, with phrases such as “really wanted to love” and “fit true size” appearing significantly. Customers expressed their disappointment that the product did not meet their expectations, especially in relation to the size and fit not matching their expectations. The phrases “too much fabric” and “looks nothing like” indicate a discrepancy between the online presentation and the actual product.

In contrast, N-gram analysis of recommended product reviews showed predominantly positive sentiments. The themes of product fit that is true to size and the customer’s social experience of wearing the product are positively highlighted. Phrases such as “true size”, ”fit perfectly”, and “received many compliments” reflected customers’ satisfaction with the quality of the product and its fit with their body size.

Intelligible supervised learning

This analysis starts with data preprocessing, where customer reviews and rating labels are converted into tuples. A tokenization stage is performed to convert text to lowercase, separate words, remove stop words, and perform stemming. Feature generation uses one-hot encoding with a focus on the top 2000 most commonly occurring words in the dataset.

Next, the naive Bayes model was applied to make review sentiment predictions. This model proved itself with an accuracy of 82.5%, which shows a satisfactory performance for this analysis. In this step, the features that were most informative in distinguishing recommended and non-recommended reviews were displayed, including words such as “cheap”, ”glad”, and ”perfect”.

A key finding of this analysis is that product recommendations and ratings play different roles. “Recommended” proved to be a strong indicator of positive sentiment in reviews, while more complex ratings with ratings around 3 tended to reflect optimistic reviews that also provided constructive criticism of the product.

In addition to using Naive Bayes, the model implementation was done using logistic regression on product reviews with the help of TF-IDF vectorizer. The model produced excellent accuracy, reaching about 91.3% on training data and 89.2% on validation data. Confusion matrix shows that the model is able to distinguish between recommended and non-recommended reviews well.

With the high accuracy results of the Logistic Regression model, it can be concluded that the model is effective in classifying positive and negative sentiments in product reviews. This analysis provides deep insights into customer preferences and feedback, and the results can serve as a foundation for further improvements and decision-making in enhancing product quality and customer satisfaction. As shown in Fig. 7, the model output using ELI5 demonstrates the interpretability of the logistic regression model, highlighting key features influencing sentiment classification. Additionally, Fig. 8 presents the model output using LGBM and SHAP, offering a detailed explanation of the model’s decision-making process and the contribution of each feature to the predictions.