Scrapper quotes from famous authors using the Scrapy and Apache Airflow as an orchestration tool.

Project: [https://github.com/octadelsueldo/scrapy-crawl-airflow.](https://github.com/octadelsueldo/scrapy-crawl-airflow)

This is the repository where you will find a scrapper for the '<http://quotes.toscrape.com/>' website using the Python Scrapy library and Apache Airflow as an orchestration and production tool. Last but not least, the extracted information has been stored in an SQLite database for later analytical treatment.

> "The process of initialize Airflow locally using Docker was, by far, the most dificult part of the project".



Clustering of drivers using 6 years of telematic data of drivers + crash prediction + Power BI Dashboard.

**Project:** <https://github.com/octadelsueldo/TFM_Driver_Profiling>

**Authors:**

Octavio del Sueldo 

Jose Lopez

**Supervisor**

Juan Manuel López Zafra

**Abstract:** Traffic accidents represent a high cost for insurance companies as well as for society, in
economic and social terms, because in all cases the costs include medical and rehabilitation ex-
penses, legal and emergency services, property damage and production losses. Thanks to the use of
telematics and data science we may be able to find patterns of behavior that explain the claims.
During this research we will work with a database of more than 95,000 drivers that includes infor-
mation collected over 6 years; for this, we have performed an important work of cleaning and engi-
neering of variables, for finally clustering the drivers through a PAM, being the most representative
variables the intensity of use of the vehicle and the driving experience. In addition, we have made
a prediction based on whether or not they have suffered a crash using a decision tree, obtaining a
72.25% accuracy rate.

> “We highlight Intensity of use and experience as the most decisive variables when it comes to explaining crashes”

1.  **Introduction.**

Insurance is an effective way of protecting individuals against the consequences of
risks. It is based on transferring the risks to an insurer who is responsible for compensat-
ing all or part of the damage caused by the occurrence of an event. A fair price will accu-
rately reflect the actual risk of the insured, otherwise the business may have problems in
the client portfolio, since those clients who have less risks end up supporting those who
are riskier, which causes a client ́s churn with lower loss ratio and an entry of those with
high loss ratio.... \[1-4]

This directly harms insurance companies, policyholders and, consequently, society.
Therefore, actuaries use risk-related information, i.e., they use those factors that model
risk for insurance premium calculation, so that they can construct league tables based on
expected losses.

In the case of automobiles, the traditional variables for determining the risk profile
are personal characteristics, claims history and vehicle characteristics. However, premi-
ums are often inaccurate in practice, as these factors do not have a direct causal relation-
ship with actual driving risk.

Thanks to the development of networks, connectivity, IoT... UBI (Usage Based Insur-
ance) products are increasingly popular within insurance companies. This insurance
product model is based on schemes known as "pay-as-you-drive" (PAYD) and "pay-how-
you-drive" (PHYD); the determination of the premium is based on variables determined
from the actual data of the driver, such as driving time, distance traveled, type of roads
traveled, speeds.... \[5-7]

Thanks to the use of these technologies, the benefit is twofold, since the insured ob-
tains a price that is more in line with his behavior, so that drivers with lower accident
rates will not be penalized, and, on the other hand, insurance companies can effectively
improve the accuracy of insurance prices. In addition, UBI products encourage drivers to drive less or improve their driving habits, as they can benefit from the use of dynamic premiums.

This coupled with the use of data science makes it possible to mine the information
collected by the devices, so that insurers can fine-tune their risk models. Insurers can
model driver behavior and therefore able to predict future claims based on what has hap-
pened in the past.

According to studies on the subject, such as Ryan et al. (1998) \[8-11], young drivers
are those with the highest accident rates, which is why we will focus on this profile of
drivers, specifically, drivers between 18 and 30 years of age.

As we have already mentioned, the main objective of our research is to segment
young drivers based on their driving characteristics, driver information, such as age and
vehicle qualities (power, weight, etc.). This segmentation (unsupervised algorithm) will
help to classify drivers, using a decision tree (supervised algorithm), based on their acci-
dent rate, for which we will use the proxy of crashes (decelerations of more than 4G).

**2. Goals of the Project**

• Database cleaning

• Identification of essential variables.

• Identification of groups of drivers and behavior patterns

• Prediction by classification tree.

• Proposal for a scorecard.

**3. Analysis**

**3.1. Exploratory Data Analysis**

First, we perform an exploratory analysis of the database in order to find possible data quality failures.

**3.2. Feature Engineering**

Once this phase is done, we start with the feature engineering. The first thing will be to perform an imputation of null values ​​(each variable was studied separately). We have also carried out a transformation of variables changing the units of measurement, for example from meters to kilometers. Likewise, we have deleted variables that we do not consider relevant for the analysis, either due to knowledge of the business or due to multicollinearity with other variables. In addition, we have created new variables that have been essential for the explanation of our unsupervised clustering model, such as the intensity of vehicle use. In this section we have also defined the target variable as a dummy variable that takes the value zero if there is no crash and the value one if there is a crash. Finally, given the objectives of the work, we have filtered our database, keeping only those young drivers between 18 and 30 years old, as well as those drivers who had a minimum of 60 registrations and 30 days in the product.

**3.3. Relevant variables**

In a later step, we have carried out an exploratory analysis of the data already filtered by the engineering phase with the aim of obtaining insights on the behavior of these drivers. Thanks to this, we were able to determine that the intensity of vehicle use is related to the number of crashes. Being those drivers with the highest intensity of use the ones that present the greatest crashes. In addition, the experience behind the wheel is another of the variables that explain the number of crashes, so that the less experienced drivers are the ones who present the greatest crashes. After these analyses, we have studied the gender variable and we can conclude that men have a higher accident rate than women.

**3.4. Clustering**

Once the most relevant variables have been determined, we have carried out a clustering through a PAM using the Euclidean distance. To do this, we have determined the optimal number of groups through the elbow rule, resulting in two as the optimal number of groups. Segmenting the dataset, we have obtained a silhouette of 0.25 and two groups that are well differentiated by their characteristics. The first of these corresponds to young drivers who have an average age of less than 25 years and experience behind the wheel of close to 5 years. These drivers use the vehicle more times a day, so they are more exposed to risks, which causes crashes in this group to be higher. On the other hand, the second cluster is made up of the most experienced drivers with an average age close to 29 years and an average experience of more than 9 years. In this case, drivers have fewer crashes due to the learning factor and use the vehicle less times per day, so they are less exposed to crashes.

**3.5. Prediction using a decision tree**

This is where we will take advantage of the groups of drivers and their qualities to predict whether or not they will crash. We have used a decision tree as it is one of the easiest models to interpret for people without technical knowledge of algorithms. Once our environment is initialized in PyCaret, we have defined the number of initial leaves with which we should work the decision tree in such a way that it allows us to obtain the highest Accuracy possible. Likewise, we have tuned our model so that it allows us to better classify those drivers that had crashes within our dataset since, as the classes were unbalanced, our model classified the zeros very well, but left aside the ones of our model, so what we did was assign different weights to each of the classifiers according to the needs of the business. This is where we have assigned a profit of EUR 4000 for those true positives, that is, those that we predict as crash and are effectively crash since if we can identify them we can charge a higher premium to these drivers. On the other hand, we will allocate a reward of EUR 2,500 for those drivers who are identified as not crashing and are effectively not crashing as we will reward them for their good driving, but at the same time we will save capital as they will not give us additional costs for have good driving habits. The false negatives will have a cost of EUR 3,000 since they would be drivers that we have identified as negative when they were positive, that is, we are going to charge them less and they will also crash, so they are the drivers we want to avoid. As for the false positives, we will put a cost of EUR 1000 on them since we will be charging them more than they should and they will also have no incidents driving so it is likely that they can find other alternatives on the market and go to the competition what we would lead to stop charging those premiums.
Once this was done, we achieved superior results compared to our first model. We have decreased about 4 percentage points in Accuracy and also in Precision but we have improved the Recall and the F1 Score since now we are capturing more ones in our model and we go from a Recall of 0.02 to 0.219 while the F1 Score goes from 0.04 to 0.2738 . This leads us to the conclusion that by adding the costs to each of the classifiers according to the relative needs of the business we can outperform our base model.

**4. Conclusions**

After conducting exhaustive research on more than 95,000 drivers and 45 variables, from more than 54 million trips between 2007 and 2013, which determine driving behavior, vehicle characteristics and driver attributes, such as age , experience or sex, we can conclude that there are variables that explain the accident rate for drivers between 18 and 30 years old. Among these variables we highlight the experience behind the wheel or the intensity of use and the fact is that the greater the intensity of use, the greater the accident rate, this makes sense since the insured is more exposed to accidents, so by probability he should have a greater number of crashes, Experience is also decisive when it comes to explaining crashes, and the lesser the experience, the greater the number of crashes. In both cases, the relationship between sex was also studied, and men are always more prone to accidents than women.
Thanks to the identification of these variables, we have carried out a segmentation of the conductors using a Partitioning Around Medoids (PAM). After studying the optimal number of groups, we determined that 2 is the one that maximizes the silhouette of the cluster using the elbow method. Using the Euclidean distance we obtain a mean silhouette of 0.25 and two groups of similar size:

• First group: They are younger and less qualified drivers; the average and maximum speed are slightly lower than those of the first group; however, the intensity of use is higher than average, almost 3 trips a day. Lastly, they have a higher than average accident rate

• Second group: These are the most experienced and older drivers, with higher average and maximum speeds; a possible explanation comes from the fact that they tend to drive on open roads with a higher intensity than in the urban cycle. The intensity of use (defined as the number of times the driver uses the car per day, as stated above) is slightly lower than that of the other group. Finally, the accident rate (measured in terms of collisions) is lower in this group than in the second.

Using the information provided by the unsupervised algorithm, we make a classificatory prediction of users who will or will not crash. For this, we use a decision tree, this is because it is one of the easiest models to interpret by people without technical knowledge. After optimizing the algorithm we have obtained the following metrics

• Accuracy: 72.25

• AUC: 61.64

• Recall: 21.9

• Accuracy: 36.40

• F1Score: 27.38

To achieve these results, we have assigned different weights to each of the possible classifiers in our algorithm, that is, the False Negative will not have the same weight as the True Positive, since they will be those that we treat as low-risk drivers, charging them a low premium when they will crash in the future. Beyond the results, one of the most important conclusions, which reinforces our initial hypotheses, is that the most significant variable for the tree is the intensity of vehicle use.

**References**

1\. Zhang, H., Xu, L., Cheng, X., Chen, W., & Zhao, X. (2017). Big Data Research on Driving Behavior Model and Auto Insurance Pricing Factors Based on UBI. Lecture Notes in Electrical Engineering, 404–411. <https://doi.org/10.1007/978-981-10-7521-6_49>

2\. Ma, Y. L., Zhu, X., Hu, X., & Chiu, Y. C. (2018). The use of context-sensitive insurance telematics data in auto insurance rate making. Transportation Research Part A: Policy and Practice, 113, 243–258. <https://doi.org/10.1016/j.tra.2018.04.013>

3\. Sarabia, J. M., Prieto, F., Jordá, V., & Sperlich, S. (2020). A Note on Combining Machine Learning with Statistical Modeling for Financial Data Analysis. Risks, 8(2), 32. <https://doi.org/10.3390/risks8020032>

4\. Guillen, M., & Pesantez-Narvaez, J. MACHINE LEARNING AND PREDICTIVE MODELING FOR AUTOMOBILE INSURANCE PRICING MACHINE LEARNING Y MODELIZACIÓN PREDICTIVA PARA LA TARIFICACIÓN EN EL SEGURO DE AUTOMÓVILES.

5\. Huang, Y., & Meng, S. (2019). Automobile insurance classification ratemaking based on telematics driving data. Decision Support Systems, 127, 113156. <https://doi.org/10.1016/j.dss.2019.113156>

6\. Bordoff, J., & Noel, P. (2010). Pay-as-you-drive auto insurance. Issues of the Day: 100 Commentaries on Climate, Energy, the Environment, Transportation, and Public Health Policy, 150.

7\. Bolderdijk, J., Knockaert, J., Steg, E., & Verhoef, E. (2011). Effects of Pay-As-You-Drive vehicle insurance on young drivers’ speed choice: Results of a Dutch field experiment. Accident Analysis & Prevention, 43(3), 1181–1186. <https://doi.org/10.1016/j.aap.2010.12.032>

8\. Ryan, G., Legge, M., & Rosman, D. (1998). Age related changes in drivers’ crash risk and crash type. Accident Analysis & Prevention, 30(3), 379–387. [https://doi.org/10.1016/s0001-4575(97)00098-5](https://doi.org/10.1016/s0001-4575%2897%2900098-5)

9\. Lifestyle and accidents among young drivers. (1994, 1 junio). ScienceDirect. <https://linkinghub.elsevier.com/retrieve/pii/0001457594900035>

10\. Machin, M. A., & Sankey, K. S. (2008). Relationships between young drivers’ personality characteristics, risk perceptions, and driving behaviour. Accident Analysis & Prevention, 40(2), 541–547. <https://doi.org/10.1016/j.aap.2007.08.010>

11\. Tränkle, U., Gelau, C., & Metker, T. (1990). Risk perception and age-specific accidents of young drivers. Accident Analysis & Prevention, 22(2), 119–125. [https://doi.org/10.1016/0001-4575(90)90063-q](https://doi.org/10.1016/0001-4575%2890%2990063-q)

12\. Patil, P. (2018, July 7). *What is Exploratory Data Analysis? — Towards Data Science*. Medium. <https://towardsdatascience.com/exploratory-data-analysis-8fc1cb20fd15>

13\. Reid Turner, C., Fuggetta, A., Lavazza, L., & Wolf, A. L. (1999b). A conceptual basis for feature engineering. Journal of Systems and Software, 49(1), 3–15. [https://doi.org/10.1016/s0164-1212(99)00062-x](https://doi.org/10.1016/s0164-1212%2899%2900062-x)

14\. Rençberoğlu, E. (2019, 3 abril). Fundamental Techniques of Feature Engineering for Machine Learning. Medium. <https://towardsdatascience.com/feature-engineering-for-machine-learning-3a5e293a5114>

15\. Kalsoom, R., & Halim, Z. (2013). Clustering the driving features based on data streams. INMIC. Published. <https://doi.org/10.1109/inmic.2013.6731330>

16\. Higgs, B., & Abbas, M. (2013, October). A two-step segmentation algorithm for behavioral clustering of naturalistic driving styles. In 16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013) (pp. 857–862). IEEE.

17\. Hanafy, M., & Ming, R. (2021). Machine learning approaches for auto insurance big data. Risks, 9(2), 42.

18\. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. (1987, 1 november). ScienceDirect. <https://www.sciencedirect.com/science/article/pii/0377042787901257>

19\. Model-based evaluation of clustering validation measures. (2007, 1 march). ScienceDirect. <https://www.sciencedirect.com/science/article/abs/pii/S0031320306003104>

20\. Arora, P., Deepali, & Varshney, S. (2016, 1 January). Analysis of K-Means and K-Medoids Algorithm For Big Data. ScienceDirect, 78. <https://www.sciencedirect.com/science/article/pii/S1877050916000971>

21\. Chakure, A. (2020, 6 November). Decision Tree Classification — The Startup. Medium. <https://medium.com/swlh/decision-tree-classification-de64fc4d5aac>

22\. Moral-García, S., Castellano, J. G., Mantas, C. J., Montella, A., & Abellán, J. (2019). Decision tree ensemble method for analyzing traffic accidents of novice drivers in urban areas. Entropy, 21(4), 360.

23\. Galarnyk, M. (2021, January). Understanding Decision Trees for Classification (Python). Medium. <https://towardsdatascience.com/understanding-decision-trees-for-classification-python-9663d683c952>

24\. Rane, S. (2019, 9 November). SHAP: A reliable way to analyze model interpretability. Medium. <https://towardsdatascience.com/shap-a-reliable-way-to-analyze-your-model-interpretability-874294d30af6>

25\. Dataman. (2021, 2 May). Explain Your Model with the SHAP Values — Towards Data Science. Medium. <https://towardsdatascience.com/explain-your-model-with-the-shap-values-bc36aac4de3d>


Driver Profiling through ML

Recommendation engine using Alternating Least Squares in PySpark using MovieLens dataset.

#### Project: <https://github.com/octadelsueldo/recommendation_engine>

#### Objective

We will show how to build recommendation engines using Alternating Least Squares in PySpark.

#### Target

We will look forward to the minimum `RMSE` which means, on average, how far is our predictions from the real values.

#### Data

This dataset (ml-latest) describes a 5-star rating and free-text tagging activity from MovieLens, a movie recommendation service. It contains 27,753,444 ratings and 1,108,997 tag applications across 58098 movies. These data were created by 283228 users between January 09, 1995, and September 26, 2018. This dataset was generated on September 26, 2018.

Users were selected at random for inclusion. All selected users had rated at least 1 movie. No demographic information is included. Each user is represented by an id, and no other information is provided.

#### EDA

In this part, we have performed the EDA, so we can see more details of these datasets. First of all, we will look up the first and last rows, get the data shape, schema, dictionary of the variables... Also, we have studied if we have to drop some variables, study NaN and duplicate data. In addition to this, we have performed a genre analysis to know which of the different genres are most rated, better-rated, or their distributions. Also, we studied movies without genres listed. Plus, we studied the most popular movies, movies not rated, the number of movies by year, etc. Finally, we perform an analysis of different users of our interest.

#### Processing data

In order to perform the algorithm, we have processed the data and created our TRAIN, VALIDATION, and TEST datasets.

#### Predictions

Algorithm | RMSE
:---------|:--------
ALS    | 0.81328

#### Conclusions

We learned that matrix factorization can solve “popular bias” and “item cold-start” problems in collaborative filtering. We also leveraged Spark ML to implement distributed recommender system using Alternating Least Square (ALS)

#### Repository structure

*   *README.md* <- The top-level README for developers.

*   *data*

    *   01\_raw <- Immutable input data

    *   02\_models <- trained models

*   *html* <- html of the notebooks.

*   *notebooks* <- Jupyter notebooks.

*   *references* <- Data dictionaries, manuals, etc.

*   *requirements.txt* <- The requirements file for reproducing the analysis environment.

*   *.gitignore* <- Avoids uploading data, credentials, outputs, system files etc

#### Authors:

Octavio del Sueldo hugo.delsueldo@cunef.edu

Jose López Galdón jose.lopez@cunef.edu
