Overcome. Persevere. Enjoy the journey.

Welcome to my portfolio website!

Jhon Dryden

Clustering of drivers using 6 years of telematic data of drivers + crash prediction + Power BI Dashboard.

**Project:** <https://github.com/octadelsueldo/TFM_Driver_Profiling>

**Authors:**

Octavio del Sueldo 

Jose Lopez

**Supervisor**

Juan Manuel López Zafra

**Abstract:** Traffic accidents represent a high cost for insurance companies as well as for society, in
economic and social terms, because in all cases the costs include medical and rehabilitation ex-
penses, legal and emergency services, property damage and production losses. Thanks to the use of
telematics and data science we may be able to find patterns of behavior that explain the claims.
During this research we will work with a database of more than 95,000 drivers that includes infor-
mation collected over 6 years; for this, we have performed an important work of cleaning and engi-
neering of variables, for finally clustering the drivers through a PAM, being the most representative
variables the intensity of use of the vehicle and the driving experience. In addition, we have made
a prediction based on whether or not they have suffered a crash using a decision tree, obtaining a
72.25% accuracy rate.

> “We highlight Intensity of use and experience as the most decisive variables when it comes to explaining crashes”

1.  **Introduction.**

Insurance is an effective way of protecting individuals against the consequences of
risks. It is based on transferring the risks to an insurer who is responsible for compensat-
ing all or part of the damage caused by the occurrence of an event. A fair price will accu-
rately reflect the actual risk of the insured, otherwise the business may have problems in
the client portfolio, since those clients who have less risks end up supporting those who
are riskier, which causes a client ́s churn with lower loss ratio and an entry of those with
high loss ratio.... \[1-4]

This directly harms insurance companies, policyholders and, consequently, society.
Therefore, actuaries use risk-related information, i.e., they use those factors that model
risk for insurance premium calculation, so that they can construct league tables based on
expected losses.

In the case of automobiles, the traditional variables for determining the risk profile
are personal characteristics, claims history and vehicle characteristics. However, premi-
ums are often inaccurate in practice, as these factors do not have a direct causal relation-
ship with actual driving risk.

Thanks to the development of networks, connectivity, IoT... UBI (Usage Based Insur-
ance) products are increasingly popular within insurance companies. This insurance
product model is based on schemes known as "pay-as-you-drive" (PAYD) and "pay-how-
you-drive" (PHYD); the determination of the premium is based on variables determined
from the actual data of the driver, such as driving time, distance traveled, type of roads
traveled, speeds.... \[5-7]

Thanks to the use of these technologies, the benefit is twofold, since the insured ob-
tains a price that is more in line with his behavior, so that drivers with lower accident
rates will not be penalized, and, on the other hand, insurance companies can effectively
improve the accuracy of insurance prices. In addition, UBI products encourage drivers to drive less or improve their driving habits, as they can benefit from the use of dynamic premiums.

This coupled with the use of data science makes it possible to mine the information
collected by the devices, so that insurers can fine-tune their risk models. Insurers can
model driver behavior and therefore able to predict future claims based on what has hap-
pened in the past.

According to studies on the subject, such as Ryan et al. (1998) \[8-11], young drivers
are those with the highest accident rates, which is why we will focus on this profile of
drivers, specifically, drivers between 18 and 30 years of age.

As we have already mentioned, the main objective of our research is to segment
young drivers based on their driving characteristics, driver information, such as age and
vehicle qualities (power, weight, etc.). This segmentation (unsupervised algorithm) will
help to classify drivers, using a decision tree (supervised algorithm), based on their acci-
dent rate, for which we will use the proxy of crashes (decelerations of more than 4G).

**2. Goals of the Project**

• Database cleaning

• Identification of essential variables.

• Identification of groups of drivers and behavior patterns

• Prediction by classification tree.

• Proposal for a scorecard.

**3. Analysis**

**3.1. Exploratory Data Analysis**

First, we perform an exploratory analysis of the database in order to find possible data quality failures.

**3.2. Feature Engineering**

Once this phase is done, we start with the feature engineering. The first thing will be to perform an imputation of null values ​​(each variable was studied separately). We have also carried out a transformation of variables changing the units of measurement, for example from meters to kilometers. Likewise, we have deleted variables that we do not consider relevant for the analysis, either due to knowledge of the business or due to multicollinearity with other variables. In addition, we have created new variables that have been essential for the explanation of our unsupervised clustering model, such as the intensity of vehicle use. In this section we have also defined the target variable as a dummy variable that takes the value zero if there is no crash and the value one if there is a crash. Finally, given the objectives of the work, we have filtered our database, keeping only those young drivers between 18 and 30 years old, as well as those drivers who had a minimum of 60 registrations and 30 days in the product.

**3.3. Relevant variables**

In a later step, we have carried out an exploratory analysis of the data already filtered by the engineering phase with the aim of obtaining insights on the behavior of these drivers. Thanks to this, we were able to determine that the intensity of vehicle use is related to the number of crashes. Being those drivers with the highest intensity of use the ones that present the greatest crashes. In addition, the experience behind the wheel is another of the variables that explain the number of crashes, so that the less experienced drivers are the ones who present the greatest crashes. After these analyses, we have studied the gender variable and we can conclude that men have a higher accident rate than women.

**3.4. Clustering**

Once the most relevant variables have been determined, we have carried out a clustering through a PAM using the Euclidean distance. To do this, we have determined the optimal number of groups through the elbow rule, resulting in two as the optimal number of groups. Segmenting the dataset, we have obtained a silhouette of 0.25 and two groups that are well differentiated by their characteristics. The first of these corresponds to young drivers who have an average age of less than 25 years and experience behind the wheel of close to 5 years. These drivers use the vehicle more times a day, so they are more exposed to risks, which causes crashes in this group to be higher. On the other hand, the second cluster is made up of the most experienced drivers with an average age close to 29 years and an average experience of more than 9 years. In this case, drivers have fewer crashes due to the learning factor and use the vehicle less times per day, so they are less exposed to crashes.

**3.5. Prediction using a decision tree**

This is where we will take advantage of the groups of drivers and their qualities to predict whether or not they will crash. We have used a decision tree as it is one of the easiest models to interpret for people without technical knowledge of algorithms. Once our environment is initialized in PyCaret, we have defined the number of initial leaves with which we should work the decision tree in such a way that it allows us to obtain the highest Accuracy possible. Likewise, we have tuned our model so that it allows us to better classify those drivers that had crashes within our dataset since, as the classes were unbalanced, our model classified the zeros very well, but left aside the ones of our model, so what we did was assign different weights to each of the classifiers according to the needs of the business. This is where we have assigned a profit of EUR 4000 for those true positives, that is, those that we predict as crash and are effectively crash since if we can identify them we can charge a higher premium to these drivers. On the other hand, we will allocate a reward of EUR 2,500 for those drivers who are identified as not crashing and are effectively not crashing as we will reward them for their good driving, but at the same time we will save capital as they will not give us additional costs for have good driving habits. The false negatives will have a cost of EUR 3,000 since they would be drivers that we have identified as negative when they were positive, that is, we are going to charge them less and they will also crash, so they are the drivers we want to avoid. As for the false positives, we will put a cost of EUR 1000 on them since we will be charging them more than they should and they will also have no incidents driving so it is likely that they can find other alternatives on the market and go to the competition what we would lead to stop charging those premiums.
Once this was done, we achieved superior results compared to our first model. We have decreased about 4 percentage points in Accuracy and also in Precision but we have improved the Recall and the F1 Score since now we are capturing more ones in our model and we go from a Recall of 0.02 to 0.219 while the F1 Score goes from 0.04 to 0.2738 . This leads us to the conclusion that by adding the costs to each of the classifiers according to the relative needs of the business we can outperform our base model.

**4. Conclusions**

After conducting exhaustive research on more than 95,000 drivers and 45 variables, from more than 54 million trips between 2007 and 2013, which determine driving behavior, vehicle characteristics and driver attributes, such as age , experience or sex, we can conclude that there are variables that explain the accident rate for drivers between 18 and 30 years old. Among these variables we highlight the experience behind the wheel or the intensity of use and the fact is that the greater the intensity of use, the greater the accident rate, this makes sense since the insured is more exposed to accidents, so by probability he should have a greater number of crashes, Experience is also decisive when it comes to explaining crashes, and the lesser the experience, the greater the number of crashes. In both cases, the relationship between sex was also studied, and men are always more prone to accidents than women.
Thanks to the identification of these variables, we have carried out a segmentation of the conductors using a Partitioning Around Medoids (PAM). After studying the optimal number of groups, we determined that 2 is the one that maximizes the silhouette of the cluster using the elbow method. Using the Euclidean distance we obtain a mean silhouette of 0.25 and two groups of similar size:

• First group: They are younger and less qualified drivers; the average and maximum speed are slightly lower than those of the first group; however, the intensity of use is higher than average, almost 3 trips a day. Lastly, they have a higher than average accident rate

• Second group: These are the most experienced and older drivers, with higher average and maximum speeds; a possible explanation comes from the fact that they tend to drive on open roads with a higher intensity than in the urban cycle. The intensity of use (defined as the number of times the driver uses the car per day, as stated above) is slightly lower than that of the other group. Finally, the accident rate (measured in terms of collisions) is lower in this group than in the second.

Using the information provided by the unsupervised algorithm, we make a classificatory prediction of users who will or will not crash. For this, we use a decision tree, this is because it is one of the easiest models to interpret by people without technical knowledge. After optimizing the algorithm we have obtained the following metrics

• Accuracy: 72.25

• AUC: 61.64

• Recall: 21.9

• Accuracy: 36.40

• F1Score: 27.38

To achieve these results, we have assigned different weights to each of the possible classifiers in our algorithm, that is, the False Negative will not have the same weight as the True Positive, since they will be those that we treat as low-risk drivers, charging them a low premium when they will crash in the future. Beyond the results, one of the most important conclusions, which reinforces our initial hypotheses, is that the most significant variable for the tree is the intensity of vehicle use.

**References**

1\. Zhang, H., Xu, L., Cheng, X., Chen, W., & Zhao, X. (2017). Big Data Research on Driving Behavior Model and Auto Insurance Pricing Factors Based on UBI. Lecture Notes in Electrical Engineering, 404–411. <https://doi.org/10.1007/978-981-10-7521-6_49>

2\. Ma, Y. L., Zhu, X., Hu, X., & Chiu, Y. C. (2018). The use of context-sensitive insurance telematics data in auto insurance rate making. Transportation Research Part A: Policy and Practice, 113, 243–258. <https://doi.org/10.1016/j.tra.2018.04.013>

3\. Sarabia, J. M., Prieto, F., Jordá, V., & Sperlich, S. (2020). A Note on Combining Machine Learning with Statistical Modeling for Financial Data Analysis. Risks, 8(2), 32. <https://doi.org/10.3390/risks8020032>

4\. Guillen, M., & Pesantez-Narvaez, J. MACHINE LEARNING AND PREDICTIVE MODELING FOR AUTOMOBILE INSURANCE PRICING MACHINE LEARNING Y MODELIZACIÓN PREDICTIVA PARA LA TARIFICACIÓN EN EL SEGURO DE AUTOMÓVILES.

5\. Huang, Y., & Meng, S. (2019). Automobile insurance classification ratemaking based on telematics driving data. Decision Support Systems, 127, 113156. <https://doi.org/10.1016/j.dss.2019.113156>

6\. Bordoff, J., & Noel, P. (2010). Pay-as-you-drive auto insurance. Issues of the Day: 100 Commentaries on Climate, Energy, the Environment, Transportation, and Public Health Policy, 150.

7\. Bolderdijk, J., Knockaert, J., Steg, E., & Verhoef, E. (2011). Effects of Pay-As-You-Drive vehicle insurance on young drivers’ speed choice: Results of a Dutch field experiment. Accident Analysis & Prevention, 43(3), 1181–1186. <https://doi.org/10.1016/j.aap.2010.12.032>

8\. Ryan, G., Legge, M., & Rosman, D. (1998). Age related changes in drivers’ crash risk and crash type. Accident Analysis & Prevention, 30(3), 379–387. [https://doi.org/10.1016/s0001-4575(97)00098-5](https://doi.org/10.1016/s0001-4575%2897%2900098-5)

9\. Lifestyle and accidents among young drivers. (1994, 1 junio). ScienceDirect. <https://linkinghub.elsevier.com/retrieve/pii/0001457594900035>

10\. Machin, M. A., & Sankey, K. S. (2008). Relationships between young drivers’ personality characteristics, risk perceptions, and driving behaviour. Accident Analysis & Prevention, 40(2), 541–547. <https://doi.org/10.1016/j.aap.2007.08.010>

11\. Tränkle, U., Gelau, C., & Metker, T. (1990). Risk perception and age-specific accidents of young drivers. Accident Analysis & Prevention, 22(2), 119–125. [https://doi.org/10.1016/0001-4575(90)90063-q](https://doi.org/10.1016/0001-4575%2890%2990063-q)

12\. Patil, P. (2018, July 7). *What is Exploratory Data Analysis? — Towards Data Science*. Medium. <https://towardsdatascience.com/exploratory-data-analysis-8fc1cb20fd15>

13\. Reid Turner, C., Fuggetta, A., Lavazza, L., & Wolf, A. L. (1999b). A conceptual basis for feature engineering. Journal of Systems and Software, 49(1), 3–15. [https://doi.org/10.1016/s0164-1212(99)00062-x](https://doi.org/10.1016/s0164-1212%2899%2900062-x)

14\. Rençberoğlu, E. (2019, 3 abril). Fundamental Techniques of Feature Engineering for Machine Learning. Medium. <https://towardsdatascience.com/feature-engineering-for-machine-learning-3a5e293a5114>

15\. Kalsoom, R., & Halim, Z. (2013). Clustering the driving features based on data streams. INMIC. Published. <https://doi.org/10.1109/inmic.2013.6731330>

16\. Higgs, B., & Abbas, M. (2013, October). A two-step segmentation algorithm for behavioral clustering of naturalistic driving styles. In 16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013) (pp. 857–862). IEEE.

17\. Hanafy, M., & Ming, R. (2021). Machine learning approaches for auto insurance big data. Risks, 9(2), 42.

18\. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. (1987, 1 november). ScienceDirect. <https://www.sciencedirect.com/science/article/pii/0377042787901257>

19\. Model-based evaluation of clustering validation measures. (2007, 1 march). ScienceDirect. <https://www.sciencedirect.com/science/article/abs/pii/S0031320306003104>

20\. Arora, P., Deepali, & Varshney, S. (2016, 1 January). Analysis of K-Means and K-Medoids Algorithm For Big Data. ScienceDirect, 78. <https://www.sciencedirect.com/science/article/pii/S1877050916000971>

21\. Chakure, A. (2020, 6 November). Decision Tree Classification — The Startup. Medium. <https://medium.com/swlh/decision-tree-classification-de64fc4d5aac>

22\. Moral-García, S., Castellano, J. G., Mantas, C. J., Montella, A., & Abellán, J. (2019). Decision tree ensemble method for analyzing traffic accidents of novice drivers in urban areas. Entropy, 21(4), 360.

23\. Galarnyk, M. (2021, January). Understanding Decision Trees for Classification (Python). Medium. <https://towardsdatascience.com/understanding-decision-trees-for-classification-python-9663d683c952>

24\. Rane, S. (2019, 9 November). SHAP: A reliable way to analyze model interpretability. Medium. <https://towardsdatascience.com/shap-a-reliable-way-to-analyze-your-model-interpretability-874294d30af6>

25\. Dataman. (2021, 2 May). Explain Your Model with the SHAP Values — Towards Data Science. Medium. <https://towardsdatascience.com/explain-your-model-with-the-shap-values-bc36aac4de3d>


Driver Profiling through ML

Credit Risk Prediction using customer information and financial transactions.

Risk Analytics is one of the key areas of data science and business intelligence in finance. With Risk analytics and management, a company is able to take strategic decisions, increase the trustworthiness and security of the company by identifying, assessing, and controlling threats to an organization’s capital and earnings. Risk Analytics is the application of data science knowledge to manage risk in a way that helps in revenue growth with minimal losses associated with fraudulent activities by bad actors.

This was my first project related to Data Science applied to Finance. We obtain the data from the [lending club dataset](https://www.lendingclub.com/investing/peer-to-peer).

> “ I was so amazed by how difficult was to put the ML pipeline in production.”

Visit this [repository](https://github.com/octadelsueldo/lending_club) and check the notebook out to find more information about the steps of the project and the results. A summary here below:

**Objective**

The objective of this exercise was to classify our current and old customers in order to predict if the new clients will pay the loan or not.

**Target**

The target we want to predict is the \`loan_status\` variable.

**Data**

Initially, we have 4 datasets with the information of our clients, to proceed with the data analysis we merged them into one dataset. The shape of this data is 884876 rows × 151 columns.

**EDA**

After performing an Exploratory Data Analysis and selecting the best variables we got 425015 rows × 51 columns dataset.

**Processing data**

In order to perform the next algorithms, we processed the data (scaling, one hot encoding...) and created our TRAIN, VALIDATION, and TEST datasets.

**Predictions**

Algorithms            |Accuracy|ROC-AUC
:-----------------------|:--------|:--------
Logistic Regression    | 76.11 %| 0.72
Support Vector Machine | 58.89 %| 0.51
Random Forest          | 76.25 %| 0.73
XGBoost                | 76.58 %| 0.73
K-Nearest Neighbors    | 73.80 %| 0.59
XGBoost Tuned          | 73.42 %| 0.73

**Conclusions**

The best model is the XGboost tuned, because it has the best accuracy, precisions, recalls, and f1-score. But as we can see the differences between the metrics are very low, so if we have to conclude which is the best one for production would be the Logistic Regression because it is clearly interpretable and faster than the XGboost or Random Forest Classifier.

**Members of the group**

Octavio del Sueldo: <hugo.delsueldo@cunef.edu>

Jose Lopez: <jose.lopez@cunef.edu>


Credit Risk Analytics

Recommendation engine using Alternating Least Squares in PySpark using MovieLens dataset.

#### Project: <https://github.com/octadelsueldo/recommendation_engine>

#### Objective

We will show how to build recommendation engines using Alternating Least Squares in PySpark.

#### Target

We will look forward to the minimum `RMSE` which means, on average, how far is our predictions from the real values.

#### Data

This dataset (ml-latest) describes a 5-star rating and free-text tagging activity from MovieLens, a movie recommendation service. It contains 27,753,444 ratings and 1,108,997 tag applications across 58098 movies. These data were created by 283228 users between January 09, 1995, and September 26, 2018. This dataset was generated on September 26, 2018.

Users were selected at random for inclusion. All selected users had rated at least 1 movie. No demographic information is included. Each user is represented by an id, and no other information is provided.

#### EDA

In this part, we have performed the EDA, so we can see more details of these datasets. First of all, we will look up the first and last rows, get the data shape, schema, dictionary of the variables... Also, we have studied if we have to drop some variables, study NaN and duplicate data. In addition to this, we have performed a genre analysis to know which of the different genres are most rated, better-rated, or their distributions. Also, we studied movies without genres listed. Plus, we studied the most popular movies, movies not rated, the number of movies by year, etc. Finally, we perform an analysis of different users of our interest.

#### Processing data

In order to perform the algorithm, we have processed the data and created our TRAIN, VALIDATION, and TEST datasets.

#### Predictions

Algorithm | RMSE
:---------|:--------
ALS    | 0.81328

#### Conclusions

We learned that matrix factorization can solve “popular bias” and “item cold-start” problems in collaborative filtering. We also leveraged Spark ML to implement distributed recommender system using Alternating Least Square (ALS)

#### Repository structure

*   *README.md* <- The top-level README for developers.

*   *data*

    *   01\_raw <- Immutable input data

    *   02\_models <- trained models

*   *html* <- html of the notebooks.

*   *notebooks* <- Jupyter notebooks.

*   *references* <- Data dictionaries, manuals, etc.

*   *requirements.txt* <- The requirements file for reproducing the analysis environment.

*   *.gitignore* <- Avoids uploading data, credentials, outputs, system files etc

#### Authors:

Octavio del Sueldo hugo.delsueldo@cunef.edu

Jose López Galdón jose.lopez@cunef.edu


Recommendation Engine of movies

Projects:

Stay up-to-date with my words ✍️

In data science, projects usually starts by an inquiry from business people that are sent to Data or IT department to begin with the development of the product that fits the requeriments. After data, the chief of staff talks with their team and communicate the demand to their data scientist. So later, is very common, mostly for junior data scientist to start the development of a funcy machine learning / deep learning model that magically solve the problem that business needs. Actually, this is a hugh mistake. Because in 99% of the cases, this rapid model just don't fit the requirement because it was build without answering the right questions. This is why the scientific method came in place to follow the process of objectively establishing facts through testing and experimentation. The basic process involves making an observation, forming a hypothesis, making a prediction, conducting an experiment and finally analyzing the results.

If we extrapolate the methodology to the life cycle of data science projects, we can establish the following stages:

*   Understand in detail the objectives of the project: this first stage includes meeting with the customers (managers and experts of the company) and discussing and knowing in detail the goals of the project.

*   Collect the available data and information: it can be provided by the client databases, APIs, files or through surveys, etc. In this stage, we look forward to arriving at a clear understanding of all information.

*   Establish the hypotheses to be tested, working models, possible theories, and methodologies to use.

*   Estimate and validate previously established models. Contrast the hypotheses formulated.

*   Obtain preliminary conclusions.

*   Discuss previous conclusions with the client and reconsider the objectives if necessary.

*   Final conclusions and decision-making.

## What is the CRISP-DM methodology?

![](/images/CRISP-DM_Process_Diagram.png)

The CRISP-DM method was developed by IBM (which includes IBM SPSS Statistics and other products) and consists of a set of stages, so that at each stage you can go back to the previous one and improve the process. The procedure takes into account not only the understanding and needs of the client and the business, but also the scientific part of the data analysis. The sequence of the phases is not rigid. Moving back and forth between different phases is always required. The outcome of each phase determines which phase, or particular task of a phase, has to be performed next. The arrows indicate the most important and frequent dependencies between phases.

> “The scientific method trains the brain to examine and observe before making and statement of fact”

The stages are:

*   Business Understanding: In the first step, you need to pay attention to identifying the problem. You should identify the metrics used to measure success over baseline, which is the same as doing nothing. Secondly, the identification of the type of problem we are facing. Thirdly, identify key people within your organization and outside that will help you on the data science project. Fourthly, get the specifications, requirements, priorities, and budgets of the project. You need to know how accurate the solution needs to be. In addition, you should analyze if you should be built internally versus using vendor solutions. If a vendor is a solution, do a vendor comparison and checked their benchmark.
*   Data Understanding: The data understanding phase starts with initial data collection and proceeds with activities that enable you to become familiar with the data, identify data quality problems, discover first insights into the data, and/or detect interesting subsets to form hypotheses regarding hidden information.
*   Data Preparation: The data preparation phase covers all activities needed to construct the final dataset \[data that will be fed into the modeling tool(s)] from the initial raw data. Data preparation tasks are likely to be performed multiple times and not in any prescribed order. Tasks include table, record, and attribute selection, as well as transformation and cleaning of data for modeling tools.
*   Data Modeling: In this phase, various modeling techniques are selected and applied, and their parameters are calibrated to optimal values. Typically, there are several techniques for the same data mining problem type. Some techniques have specific requirements on the form of data. Therefore, going back to the data preparation phase is often necessary.
*   Evaluation: At this stage in the project, you have built a model (or models) that appears to have high quality from a data analysis
    perspective. Before proceeding to final deployment of the model, it is important to thoroughly evaluate it and review
    the steps executed to create it, to be certain the model properly achieves the business objectives. A key objective is
    to determine if there is some important business issue that has not been sufficiently considered. At the end of this
    phase, a decision on the use of the data mining results should be reached.
*   Deployment: Creation of the model is generally not the end of the project. Even if the purpose of the model is to increase knowledge of the data, the knowledge gained will need to be organized and presented in a way that the customer can use it. It
    often involves applying “live” models within an organization’s decision-making processes—for example, real-time personalization of Web pages or repeated scoring of marketing databases. Depending on the requirements, the deployment phase can be as simple as generating a report or as complex as implementing a repeatable data mining process across the enterprise. In many cases, it is the customer, not the data analyst, who carries out the deployment steps. However, even if the analyst will carry out the deployment effort, it is important for the customer to understand up front what actions need to be carried out in order to actually make use of the created models.

![](/images/Screen%20Shot%202022-09-21%20at%2000.01.57.png)

I would like to mention that there are two other basic methodologies: SEMMA and KDD. The methodology called SEMMA was proposed by the SAS Institute, and consists of the stages of Sample, Explore, Modify, Model, and Assess. On the other hand, the KDD (Knowledge Discovery in Databases) methodology refers to the process of finding knowledge in data and emphasizes the high-level application of data mining methods.

References

\[1] Gavin Wright, Scientific Method, <https://www.techtarget.com/whatis/definition/scientific-method#:~:text=The%20scientific%20method%20is%20the,and%20finally%20analyzing%20the%20results.>

\[2] IBM, CRISP-DM v0.1


THE CRISP-DM METHODOLOGY ⌨️

Exploratory data analysis (EDA) is frequently used by data scientists or data analysts to comprehend datasets for use in decision-making and data cleansing procedures. Important facts about the data, including hidden patterns, outliers, variance, covariance, and correlations between characteristics, are revealed by EDA. The data is crucial for designing the hypothesis and building more effective models.

EDA often fits into one of two categories, univariate and multivariate analysis. Analyzing one feature, such as summarizing and identifying feature patterns, is what univariate analysis entails. The univariate exploratory data analysis works with four summary measures:

*   Position

*   Dispersion

*   Shape

*   Inequality



1.  **Positional measures:** they are the typical or representative values ​​of the variable under study.

**Arithmetic average:** The arithmetic mean formula calculates the mean or average of the numbers and is used to measure the central tendency of the data. It can also be defined as the sum of all the given observations to the total number of observations.

![](/images/arithmetic_mean.svg)

It is the most used measure. However, it is very sensitive to outliers and extreme observations that is why, if you have outliers in our data, we should use the median.

**Median:** For the calculation of the median, the data has to be arranged in ascending order, and then the middlemost data point represents the **median of the data**. The basic feature of the median in describing data compared to the mean (often simply described as the "average") is that it is not skewed by a small proportion of extremely large or small values, and therefore provides a better representation of a "typical" value.

![](https://www.gstatic.com/education/formulas2/472522532/en/median_formula.svg)

**Quantile:** quantiles are cut points dividing the range of a probability distribution into continuous intervals with equal probabilities, or dividing the observations in a sample in the same way. *Q*-quantiles are values that partition a finite set of values into *q* subsets of (nearly) equal sizes. There are *q* − 1 partitions of the *q*-quantiles, one for each integer *k* satisfying 0 < *k* < *q*.

Special cases of quantiles are **quartiles** when three values divide the data into four parts (Q1: leaves 25% of the observations to the left; Q2: median: leaves 50% of the observations to the left; Q3: leaves 75% of the observations to the left). **Deciles:** nine values ​​that divide the data into ten parts and **Percentiles** when 99 values ​ divide the data into one hundred parts

![](https://upload.wikimedia.org/wikipedia/commons/5/5e/Iqr_with_quantile.png)

Other positional measures:

**Geometric mean:** the geometric mean is a mean or average, which indicates the central tendency or typical value of a set of numbers by using the product of their values (as opposed to the arithmetic mean which uses their sum). For instance, the geometric mean of two numbers, say 2 and 8, is just the square root of their product, that is, ![](https://wikimedia.org/api/rest_v1/media/math/render/svg/8a8d80ffa0ce79f7200d514844d4becd711f4f0b)

![](https://wikimedia.org/api/rest_v1/media/math/render/svg/026cae6801f672b9858d55935ec7397183dc3a36)

**Harmonic mean:** In mathematics, the harmonic mean is one of several kinds of average, and in particular, one of the Pythagorean means. It is sometimes appropriate for situations when the average rate is desired.

The harmonic mean can be expressed as the reciprocal of the arithmetic mean of the reciprocals of the given set of observations. As a simple example, the harmonic mean of 1, 4, and 4 is

![](https://wikimedia.org/api/rest_v1/media/math/render/svg/eca5906ddf61080e790c0d4df33f47e12da7d019)

The harmonic mean is one of the three Pythagorean means. For all *positive* data sets *containing at least one pair of nonequal values*, the harmonic mean is always the least of the three means, while the arithmetic mean is always the greatest of the three and the geometric mean is always in between.

**Robust means:** it is a trimmed mean by 10%. I mean, it is an arithmetic mean excluding the 10% of the largest values and the 10% of the smallest values)

**2. Dispersion measures:** the extent to which a distribution is stretched or squeezed.

**Variance:** variance is the expectation of the squared deviation of a random variable from its population mean or sample mean. Variance is a measure of dispersion, meaning it is a measure of how far a set of numbers is spread out from their average value.

**Standard deviation:** the standard deviation is a measure of the amount of variation or dispersion of a set of values. A low standard deviation indicates that the values tend to be close to the mean (also called the expected value) of the set, while a high standard deviation indicates that the values are spread out over a wider range. A useful property of the standard deviation is that, unlike the variance, it is expressed in the same unit as the data.

**Coefficient of variation:** the coefficient of variation (CV), also known as relative standard deviation (RSD), is often expressed as a percentage, and is defined as the ratio of the standard deviation ![](https://wikimedia.org/api/rest_v1/media/math/render/svg/8d4ebf618b2ab59428af1b3f2e8e251c78d2f57a) to the mean ![](https://wikimedia.org/api/rest_v1/media/math/render/svg/09bfcd47d25310c5187294b07d55e0954c2685f6) (or its absolute value, ![](https://wikimedia.org/api/rest_v1/media/math/render/svg/f48e9c3009a01ad6bf5f22ce62c842ff99f19521)). The main advantage of this measure is that allows you to compare two variables that are in different units

![](https://wikimedia.org/api/rest_v1/media/math/render/svg/568abd256930542e24daafdd32985b401249c78d)

**Range:**  In statistics, the range of a set of data is the difference between the largest and smallest values, the result of subtracting the sample maximum and minimum. It is expressed in the same units as the data.

In descriptive statistics, the range is the size of the smallest interval which contains all the data and provides an indication of statistical dispersion. Since it only depends on two of the observations, it is most useful in representing the dispersion of small data sets.

**Interquartile range:** In descriptive statistics, the **interquartile range** tells you the spread of the middle half of your distribution.

Quartiles segment any distribution that’s ordered from low to high into four equal parts. The interquartile range (IQR) contains the second and third quartiles, or the middle half of your data set. The interquartile range is an especially useful measure of variability for skewed distributions.

![](https://cdn.scribbr.com/wp-content/uploads/2020/09/iqr_quartiles.png)

**Semi-interquartile range:** The semi-interquartile range is one-half the difference between the first and third quartiles. It is half the distance needed to cover half the scores.  The semi-interquartile range is affected very little by extreme scores.  This makes it a good measure of spread for skewed distributions.

**3. Shape measures:** Measures of shape describe the distribution (or pattern) of the data within a dataset. The distribution shape of quantitative data can be described as there is a logical order of the values, and the 'low' and 'high' end values on the x-axis of the histogram are able to be identified. they are the ones that measure the Kurtosis coefficient and Skewness. **Skewness tells you the amount and direction of skew** (departure from horizontal symmetry), and **kurtosis tells you how tall and sharp the central peak is**, relative to a standard bell curve.

**Skewness:** The **moment coefficient of skewness** of a data set is

skewness: *g*1 = *m*3 / *m*23/2

where

*m*3 = ∑(*x*−*x̅*)3 / *n*   and   

*m*2 = ∑(*x*−*x̅*)2 / *n*

*x̅* is the mean and *n* is the sample size, as usual. *m*3 is called the **third moment** of the data set. *m*2 is the **variance**, the square of the standard deviation.

If g1 = 0, the distribution is symmetric; if g1 < 0, the distribution is negatively skewed and if g1 > 0, the distribution is positively skewed.

**Kurtosis:** The other common measure of shape is called kurtosis. The **moment coefficient of kurtosis** of a data set is computed almost the same way as the coefficient of skewness: just change the exponent 3 to 4 in the formulas:

kurtosis: *a*4 = *m*4 / *m*22   and   excess kurtosis: *g*2 = *a*4−3

where

*m*4 = ∑(*x*−*x̅*)4 / *n*   and   *m*2 = ∑(*x*−*x̅*)2 / *n*

The excess kurtosis is generally used because the excess kurtosis of a normal distribution is 0. *x̅* is the mean and *n* is the sample size, as usual. *m*4 is called the **fourth moment** of the data set. *m*2 is the **variance**, the square of the standard deviation.

if g2 = 0 the ́distribution is Moscurtic; if g2 < 0, the distribution is platykurtic and if g2 > 0, the distribution is leptokurtic.

**4. Concentration measures:** Concentration measures study how the total values ​​of a variable are distributed. The most important concentration measures are the Lorenz curve and the Gini index. They are used in multiple fields like the study of economic inequality.

In economics, the Lorenz curve is a graphical representation of the distribution of income or of wealth. It was developed by Max O. Lorenz in 1905 for representing inequality of the wealth distribution.

The curve is a graph showing the proportion of overall income or wealth assumed by the bottom *x*% of the people, although this is not rigorously true for a finite population (see below). It is often used to represent income distribution, where it shows for the bottom *x*% of households, what percentage (*y*%) of the total income they have. The percentage of households is plotted on the *x*-axis, the percentage of income on the *y*-axis. It can also be used to show distribution of assets. In such use, many economists consider it to be a measure of social inequality.

![](https://upload.wikimedia.org/wikipedia/commons/thumb/5/59/Economics_Gini_coefficient2.svg/260px-Economics_Gini_coefficient2.svg.png)

The **Gini coefficient** measures the inequality among values of a frequency distribution, such as the levels of income. A Gini coefficient of 0 expresses *perfect equality*, where all values are the same, while a Gini coefficient of 1 (or 100%) expresses *maximal inequality* among values. For example, if everyone has the same income, the Gini coefficient will be 0. In contrast, if for a large number of people only one person has all the income or consumption and all others have none, the Gini coefficient will be nearly one.

![](https://upload.wikimedia.org/wikipedia/commons/thumb/c/cc/Map_of_countries_by_GINI_coefficient\_%281990\_to\_2020%29.svg/800px-Map_of_countries_by_GINI_coefficient\_%281990\_to\_2020%29.svg.png)

**Atkinson index:** The Atkinson index (also known as the Atkinson measure or Atkinson inequality measure) is a measure of income inequality developed by British economist Anthony Barnes Atkinson. The measure is useful in determining which end of the distribution contributed most to the observed inequality.  The index can be turned into a normative measure by imposing a coefficient to weight incomes. Greater weight can be placed on changes in a given portion of the income distribution by choosing ![](https://wikimedia.org/api/rest_v1/media/math/render/svg/a30c89172e5b88edbd45d3e2772c7f5e562e5173), the level of "inequality aversion", appropriately. The Atkinson index becomes more sensitive to changes at the lower end of the income distribution as ![](https://wikimedia.org/api/rest_v1/media/math/render/svg/a30c89172e5b88edbd45d3e2772c7f5e562e5173) increases. Conversely, as the level of inequality aversion falls (that is, as ![](https://wikimedia.org/api/rest_v1/media/math/render/svg/a30c89172e5b88edbd45d3e2772c7f5e562e5173) approaches 0) the Atkinson becomes less sensitive to changes in the lower end of the distribution. The Atkinson index is for no value of ![](https://wikimedia.org/api/rest_v1/media/math/render/svg/a30c89172e5b88edbd45d3e2772c7f5e562e5173) highly sensitive to top incomes because of the common restriction that ![](https://wikimedia.org/api/rest_v1/media/math/render/svg/a30c89172e5b88edbd45d3e2772c7f5e562e5173) is nonnegative.

The Atkinson index is defined as:

![](https://wikimedia.org/api/rest_v1/media/math/render/svg/696cc5917249b769a2974a0d5c45c1b92cafbc3b)

where ![](https://wikimedia.org/api/rest_v1/media/math/render/svg/67d30d30b6c2dbe4d6f150d699de040937ecc95f) is individual income (*i* = 1, 2, ..., *N*) and ![](https://wikimedia.org/api/rest_v1/media/math/render/svg/9fd47b2a39f7a7856952afec1f1db72c67af6161) is the mean income.

Entropy:  The formula for general entropy for real values of ![](https://wikimedia.org/api/rest_v1/media/math/render/svg/b79333175c8b3f0840bfb4ec41b8072c83ea88d3) is:

![](https://wikimedia.org/api/rest_v1/media/math/render/svg/844bcf9d016e032d7e03f0de29b7733e36f0b8a9)

where N is the number of cases (e.g., households or families), ![](https://wikimedia.org/api/rest_v1/media/math/render/svg/67d30d30b6c2dbe4d6f150d699de040937ecc95f) is the income for case i and ![](https://wikimedia.org/api/rest_v1/media/math/render/svg/b79333175c8b3f0840bfb4ec41b8072c83ea88d3) is a parameter that regulates the weight given to distances between incomes at different parts of the income distribution. For large ![](https://wikimedia.org/api/rest_v1/media/math/render/svg/b79333175c8b3f0840bfb4ec41b8072c83ea88d3) the index is especially sensitive to the existence of large incomes, whereas for small ![](https://wikimedia.org/api/rest_v1/media/math/render/svg/b79333175c8b3f0840bfb4ec41b8072c83ea88d3) the index is especially sensitive to the existence of small incomes.

**Inequality measures based on quantiles:** statistical measures based on quantiles are frequently applied to the analysis of income distribution as they comprise many popular inequality and poverty indices and indicators. Simple dispersion ratios, defined as the ratios of the income of the richest quantile over that of the poorest quantile, usually utilize deciles and quintiles, but in principle, any quantile of income distribution can be used. These measures do not depend on changes in scale and allow comparison of low, middle, and high income groups.

References:

https://www.cuemath.com/

https://en.wikipedia.org/

https://www.scribbr.com/statistics/interquartile-range/#:~:text=In%20descriptive%20statistics%2C%20the%20interquartile,half%20of%20your%20data%20set.

https://brownmath.com/stat/shape.htm

https://www.researchgate.net/publication/325980614\_Properties_of_Selected_Inequality_Measures_Based_on_Quantiles_and_Their_Application_to_the_Analysis_of_Income_Distribution_in_Poland_by_Macroregion


How to CONDUCT A EDA univariate 🗂️

In the era that we are living in, where we are reading posts, sharing interesting moments, and buying cloth on social media and webs, it is well known that the community is feeding a big amount of data into the databases of those companies.

What do I mean by data? When we talk about data, it means facts that are registered and collected for further treatment or analysis. Let's put this in context. Let's suppose that we are looking for some shirts and jeans in Zara. We just choose our favorite clothes for the next season so then, we need to register or log in on the web and then enter our data to continue with the further payment plus delivery. In the end, the companies send us a ticket that has all the information related to our operation. This invoice represents a summary of the data that was collected such as the clothes that were bought, the price, the quantity, the payment method, and so on. This is the real data that the company collects from us to deliver the purchase, and it is ready to be treated and analyzed in the company's systems.

Currently, we are living in a world that is full of data that we leave not only on retail webs or on social media but also in other places like traffic or facility cameras that collect our movements in order to understand and study them. In addition to this, insurance companies could collect telematics data based on devices connected to the car that learn our behavior while we are riding a car to offer personalized products or services. This data is the raw material, the new gold in this age, and is useful to be treated to fully understand customer behavior.

## About Data Science

> “Data Science combines the techniques and methodologies of statistics, mathematics and computer science, with the aim of understanding, interpreting and inferring conclusions from data and making decisions.”

Now that we know what is data and the potential benefits that it has, we will continue to explain the definition of data science. One of the definitions that I learned during my master's degree was that data science deals with the study of data through the scientific method.  Data science combines the techniques and methodologies of statistics, mathematics, and computer science, with the aim of understanding, interpreting, and inferring conclusions from data and making decisions.  

Based on the definition, we can infer that data science is a multidisciplinary field where domain knowledge, statistics, computer science, and communication skills are used in combination. Furthermore, it is common to listen that data scientists are better statisticians than most programmers and better programmers than most statisticians. Continuing with the definition, we can see that it has different goals like collecting and understanding the data which is called descriptive analytics that answer the question of “*What happened?*”. Another goal is interpreting the data that is part of diagnostic analytics that answer the question of “*Why did this happen?*”. In this kind of analytics, we take the analysis a step further, because we are comparing coexisting trends or movements, uncovering correlations between variables, and determining causal relationships where possible. Finally, it talks about inferring conclusions that we can relate with predictive analytics that is used to make predictions about future trends or events and answers the question, *“What might happen in the future?”*

Without data, people make decisions based on biases and beliefs. So, in order to create value and take decisions with data fundamentals, companies use their data, explore it, analyze and predict future behaviors with them. In this scenario, data scientist takes place in order to extract the data, understand it and apply different techniques to solve problems. But, how do they deliver value to the organization? It is by the following goals:

*   Revenue increase: data science can be applied to find and refine a target customer base to generate more revenue. In sales, specifically, lead management, models can analyze past customers and score leads, resulting in greater sales efficiency. Not only past in-house customer data is used, but also social media interaction for scoring. Companies that applied artificial intelligence lead scoring increased by 50% more appointments and a 60% reduction in call time *\[2]*. Better leads close sales deals faster, which creates more revenue.

*   Cost reduction: In my own personal experience in financial services, we have applied data science concepts to reduce customer turnover and fraud risk. Using predictive models, we predict when customers are likely to get into trouble and offer assistance to prevent breaches of contract. As for fraud prevention, we analyzed client data to detect anomalies that might be fraudulent, which leads to a potential savings of millions per year.

*   More efficient processes: for example data science applied with real-time reporting *\[4]*, customer interactions are more effective by allowing service representatives to better understand consumers while interacting with them. Think about how many times you’ve made a call to a customer service line and waited on hold only to answer a bunch of questions, be put on hold again, and then have to answer all the same questions with everyone else you talk to. With real-time reporting, customer service agents have instant access to the questions callers have already answered, shortening the call and hold times while improving the overall experience.

*   Knowledge discovery: it is related to the extraction of knowledge of the data collected that was previously unknown, and it has useful information

*   More efficient decision-making: With the help of data science, decision-makers are now able to make better-informed choices than in the past by shaping and filtering the data their organizations have collected. By using this data, they are able to formulate predictions of the future based on what would happen if they decided to take their organization in a brand-new direction, for example, or how they would rebuild themselves after a financial disaster, among a variety of other things. For example, in Airbnb, the marketing department thought one of their campaigns was off due to bad metrics *\[3]*. A data scientist found the root cause to be a group in Asia using their site to browse neighborhoods. As a result, the marketing department redirected that group to a travel site, which clarified their marketing metrics and saved resources instead of launching an inefficient marketing initiative.

References

\[1] Lester Leong, How Data Science makes money for businesses (2019), <https://towardsdatascience.com/how-data-science-makes-money-for-businesses-203daba010c1>

\[2] T. Capone, How Data Science Can Help Your Enterprise Generate More Revenue (2018), <https://conceptainc.com/blog/how-data-science-can-help-your-enterprise-generate-more-revenue/>

\[3] N. Patel, How Airbnb Uses Data Science to Improve Their Product and Marketing (2019), <https://neilpatel.com/blog/how-airbnb-uses-data-science/>

\[4] Landon Starr, How Data Science Can Improve Business Efficiency (2018), <https://blogs.oracle.com/ai-and-datascience/post/how-data-science-can-improve-business-efficiency>


Welcome to my portfolio website!

Driver Profiling through ML

Credit Risk Analytics

Recommendation Engine of movies

THE CRISP-DM METHODOLOGY ⌨️

How to CONDUCT A EDA univariate 🗂️

What is DATA SCIENCE? 🚀

Follow me on Medium

Got an interesting project? Tell me more...💬

Home

Welcome to my portfolio website!

Driver Profiling through ML

Credit Risk Analytics

Recommendation Engine of movies

THE CRISP-DM METHODOLOGY ⌨️

How to CONDUCT A EDA univariate 🗂️

What is DATA SCIENCE? 🚀

Follow me on Medium

Got an interesting project? Tell me more...💬