{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import matplotlib.pylab as plt\n",
"import numpy as np\n",
"\n",
"from collections import Counter"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Datasets\n",
"\n",
"Scikit-lego includes several datasets which can be used for testing purposes. Each dataset has different options for returning the data:\n",
"\n",
"- When setting ``as_frame`` to True, the data, including the target, is returned as a dataframe.\n",
"- When setting ``return_X_y`` to True, the data is returned directly as ``(data, target)`` instead of a dict object. \n",
"\n",
"This notebook describes the different sets included in Scikit-lego:\n",
"\n",
"- `sklego.datasets.load_abalone` loads in the abalone dataset\n",
"- `sklego.datasets.load_arrests` loads in a dataset with fairness concerns\n",
"- `sklego.datasets.load_chicken` loads in the joyful chickweight dataset\n",
"- `sklego.datasets.load_heroes` loads a heroes of the storm dataset\n",
"- `sklego.datasets.load_hearts` loads a dataset about hearts\n",
"- `sklego.datasets.load_penguins` loads a lovely dataset about penguins\n",
"- `sklego.datasets.fetch_creditcard` fetch a fraud dataset from openml\n",
"- `sklego.datasets.make_simpleseries` make a simulated timeseries"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Abalone\n",
"\n",
"Loads the abalone dataset where the goal is to predict the gender of the creature."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" sex \n",
" length \n",
" diameter \n",
" height \n",
" whole_weight \n",
" shucked_weight \n",
" viscera_weight \n",
" shell_weight \n",
" rings \n",
" \n",
" \n",
" \n",
" \n",
" 0 \n",
" M \n",
" 0.455 \n",
" 0.365 \n",
" 0.095 \n",
" 0.5140 \n",
" 0.2245 \n",
" 0.1010 \n",
" 0.150 \n",
" 15 \n",
" \n",
" \n",
" 1 \n",
" M \n",
" 0.350 \n",
" 0.265 \n",
" 0.090 \n",
" 0.2255 \n",
" 0.0995 \n",
" 0.0485 \n",
" 0.070 \n",
" 7 \n",
" \n",
" \n",
" 2 \n",
" F \n",
" 0.530 \n",
" 0.420 \n",
" 0.135 \n",
" 0.6770 \n",
" 0.2565 \n",
" 0.1415 \n",
" 0.210 \n",
" 9 \n",
" \n",
" \n",
" 3 \n",
" M \n",
" 0.440 \n",
" 0.365 \n",
" 0.125 \n",
" 0.5160 \n",
" 0.2155 \n",
" 0.1140 \n",
" 0.155 \n",
" 10 \n",
" \n",
" \n",
" 4 \n",
" I \n",
" 0.330 \n",
" 0.255 \n",
" 0.080 \n",
" 0.2050 \n",
" 0.0895 \n",
" 0.0395 \n",
" 0.055 \n",
" 7 \n",
" \n",
" \n",
"
\n",
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"X, y = load_abalone(return_X_y=True)\n",
"\n",
"plt.bar(Counter(y).keys(), Counter(y).values())\n",
"plt.title('Distribution of sex (target)');"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Arrests\n",
"\n",
"Loads the arrests dataset which can serve as a benchmark for fairness. It is data on\n",
"the police treatment of individuals arrested in Toronto for simple possession of small\n",
"quantities of marijuana. The goal is to predict whether or not the arrestee was released\n",
"with a summons while maintaining a degree of fairness."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" released \n",
" colour \n",
" year \n",
" age \n",
" sex \n",
" employed \n",
" citizen \n",
" checks \n",
" \n",
" \n",
" \n",
" \n",
" 0 \n",
" Yes \n",
" White \n",
" 2002 \n",
" 21 \n",
" Male \n",
" Yes \n",
" Yes \n",
" 3 \n",
" \n",
" \n",
" 1 \n",
" No \n",
" Black \n",
" 1999 \n",
" 17 \n",
" Male \n",
" Yes \n",
" Yes \n",
" 3 \n",
" \n",
" \n",
" 2 \n",
" Yes \n",
" White \n",
" 2000 \n",
" 24 \n",
" Male \n",
" Yes \n",
" Yes \n",
" 3 \n",
" \n",
" \n",
" 3 \n",
" No \n",
" Black \n",
" 2000 \n",
" 46 \n",
" Male \n",
" Yes \n",
" Yes \n",
" 1 \n",
" \n",
" \n",
" 4 \n",
" Yes \n",
" Black \n",
" 1999 \n",
" 27 \n",
" Female \n",
" Yes \n",
" Yes \n",
" 1 \n",
" \n",
" \n",
"
\n",
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"X, y = load_arrests(return_X_y=True)\n",
"\n",
"plt.bar(Counter(y).keys(), Counter(y).values())\n",
"plt.title('Distribution of released (target)');"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Chicken\n",
"Loads the chicken dataset. The chicken data has 578 rows and 4 columns\n",
"from an experiment on the effect of diet on early growth of chicks.\n",
"The body weights of the chicks were measured at birth and every second\n",
"day thereafter until day 20. They were also measured on day 21.\n",
"There were four groups on chicks on different protein diets."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" weight \n",
" time \n",
" chick \n",
" diet \n",
" \n",
" \n",
" \n",
" \n",
" 0 \n",
" 42 \n",
" 0 \n",
" 1 \n",
" 1 \n",
" \n",
" \n",
" 1 \n",
" 51 \n",
" 2 \n",
" 1 \n",
" 1 \n",
" \n",
" \n",
" 2 \n",
" 59 \n",
" 4 \n",
" 1 \n",
" 1 \n",
" \n",
" \n",
" 3 \n",
" 64 \n",
" 6 \n",
" 1 \n",
" 1 \n",
" \n",
" \n",
" 4 \n",
" 76 \n",
" 8 \n",
" 1 \n",
" 1 \n",
" \n",
" \n",
"
\n",
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"X, y = load_chicken(return_X_y=True)\n",
"\n",
"plt.hist(y)\n",
"plt.title('Distribution of weight (target)');"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Heroes\n",
"\n",
"A dataset from a video game: \"heroes of the storm\". The goal of the dataset\n",
"is to predict the attack type. Note that the pandas dataset returns more information."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" name \n",
" attack_type \n",
" role \n",
" health \n",
" attack \n",
" attack_spd \n",
" \n",
" \n",
" \n",
" \n",
" 0 \n",
" Artanis \n",
" Melee \n",
" Bruiser \n",
" 2470.0 \n",
" 111.0 \n",
" 1.00 \n",
" \n",
" \n",
" 1 \n",
" Chen \n",
" Melee \n",
" Bruiser \n",
" 2473.0 \n",
" 90.0 \n",
" 1.11 \n",
" \n",
" \n",
" 2 \n",
" Dehaka \n",
" Melee \n",
" Bruiser \n",
" 2434.0 \n",
" 100.0 \n",
" 1.11 \n",
" \n",
" \n",
" 3 \n",
" Imperius \n",
" Melee \n",
" Bruiser \n",
" 2450.0 \n",
" 122.0 \n",
" 0.83 \n",
" \n",
" \n",
" 4 \n",
" Leoric \n",
" Melee \n",
" Bruiser \n",
" 2550.0 \n",
" 109.0 \n",
" 0.77 \n",
" \n",
" \n",
"
\n",
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"X, y = load_heroes(return_X_y=True)\n",
"\n",
"plt.bar(Counter(y).keys(), Counter(y).values())\n",
"plt.title('Distribution of attack_type (target)');"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Hearts\n",
"\n",
"Loads the Cleveland Heart Diseases dataset. The goal is to predict the presence of a heart disease (target values 1, 2, 3, and 4).\n",
"The data originates from research to heart diseases by four institutions and originally contains 76 attributes.\n",
"Yet, all published experiments refer to using a subset of 13 features and one target.\n",
"This implementation loads the Cleveland dataset of the research which is the only set used by ML researchers to this date."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" age \n",
" sex \n",
" cp \n",
" trestbps \n",
" chol \n",
" fbs \n",
" restecg \n",
" thalach \n",
" exang \n",
" oldpeak \n",
" slope \n",
" ca \n",
" thal \n",
" target \n",
" \n",
" \n",
" \n",
" \n",
" 0 \n",
" 63 \n",
" 1 \n",
" 1 \n",
" 145 \n",
" 233 \n",
" 1 \n",
" 2 \n",
" 150 \n",
" 0 \n",
" 2.3 \n",
" 3 \n",
" 0 \n",
" fixed \n",
" 0 \n",
" \n",
" \n",
" 1 \n",
" 67 \n",
" 1 \n",
" 4 \n",
" 160 \n",
" 286 \n",
" 0 \n",
" 2 \n",
" 108 \n",
" 1 \n",
" 1.5 \n",
" 2 \n",
" 3 \n",
" normal \n",
" 1 \n",
" \n",
" \n",
" 2 \n",
" 67 \n",
" 1 \n",
" 4 \n",
" 120 \n",
" 229 \n",
" 0 \n",
" 2 \n",
" 129 \n",
" 1 \n",
" 2.6 \n",
" 2 \n",
" 2 \n",
" reversible \n",
" 0 \n",
" \n",
" \n",
" 3 \n",
" 37 \n",
" 1 \n",
" 3 \n",
" 130 \n",
" 250 \n",
" 0 \n",
" 0 \n",
" 187 \n",
" 0 \n",
" 3.5 \n",
" 3 \n",
" 0 \n",
" normal \n",
" 0 \n",
" \n",
" \n",
" 4 \n",
" 41 \n",
" 0 \n",
" 2 \n",
" 130 \n",
" 204 \n",
" 0 \n",
" 2 \n",
" 172 \n",
" 0 \n",
" 1.4 \n",
" 1 \n",
" 0 \n",
" normal \n",
" 0 \n",
" \n",
" \n",
"
\n",
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"X, y = load_hearts(return_X_y=True)\n",
"\n",
"y_counted = Counter([str(integer) for integer in y])\n",
"\n",
"plt.bar(y_counted.keys(), y_counted.values())\n",
"plt.title('Distribution of target (presence of heart disease)');"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load penguins\n",
"\n",
"Loads the penguins dataset, which is a lovely alternative for the iris dataset. We've\n",
"added this dataset for educational use.\n",
"\n",
"Data were collected and made available by Dr. Kristen Gorman and the Palmer Station,\n",
"Antarctica LTER, a member of the Long Term Ecological Research Network. The goal\n",
"of the dataset is to predict which species of penguin a penguin belongs to."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" species \n",
" island \n",
" bill_length_mm \n",
" bill_depth_mm \n",
" flipper_length_mm \n",
" body_mass_g \n",
" sex \n",
" \n",
" \n",
" \n",
" \n",
" 0 \n",
" Adelie \n",
" Torgersen \n",
" 39.1 \n",
" 18.7 \n",
" 181.0 \n",
" 3750.0 \n",
" male \n",
" \n",
" \n",
" 1 \n",
" Adelie \n",
" Torgersen \n",
" 39.5 \n",
" 17.4 \n",
" 186.0 \n",
" 3800.0 \n",
" female \n",
" \n",
" \n",
" 2 \n",
" Adelie \n",
" Torgersen \n",
" 40.3 \n",
" 18.0 \n",
" 195.0 \n",
" 3250.0 \n",
" female \n",
" \n",
" \n",
" 3 \n",
" Adelie \n",
" Torgersen \n",
" NaN \n",
" NaN \n",
" NaN \n",
" NaN \n",
" NaN \n",
" \n",
" \n",
" 4 \n",
" Adelie \n",
" Torgersen \n",
" 36.7 \n",
" 19.3 \n",
" 193.0 \n",
" 3450.0 \n",
" female \n",
" \n",
" \n",
"
\n",
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"X, y = load_penguins(return_X_y=True)\n",
"\n",
"plt.bar(Counter(y).keys(), Counter(y).values())\n",
"plt.title('Distribution of species (target)');"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Fetch creditcard\n",
"\n",
"Load the creditcard dataset. Download it if necessary.\n",
"\n",
"Note that internally this is using `fetch_openml` from scikit-learn, which is experimental.\n",
"\n",
"```\n",
"============== ==============\n",
"Samples total 284807\n",
"Dimensionality 29\n",
"Features real\n",
"Target int 0, 1\n",
"============== ==============\n",
"```\n",
"The datasets contains transactions made by credit cards in September 2013 by european\n",
"cardholders. This dataset present transactions that occurred in two days, where we have\n",
"492 frauds out of 284,807 transactions. The dataset is highly unbalanced, the positive\n",
"class (frauds) account for 0.172% of all transactions.\n",
"\n",
"Please cite:\n",
" Andrea Dal Pozzolo, Olivier Caelen, Reid A. Johnson and Gianluca Bontempi.\n",
" Calibrating Probability with Undersampling for Unbalanced Classification.\n",
" In Symposium on Computational Intelligence and Data Mining (CIDM), IEEE, 2015"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" V1 \n",
" V2 \n",
" V3 \n",
" V4 \n",
" V5 \n",
" V6 \n",
" V7 \n",
" V8 \n",
" V9 \n",
" V10 \n",
" ... \n",
" V21 \n",
" V22 \n",
" V23 \n",
" V24 \n",
" V25 \n",
" V26 \n",
" V27 \n",
" V28 \n",
" Amount \n",
" Class \n",
" \n",
" \n",
" \n",
" \n",
" 0 \n",
" -1.359807 \n",
" -0.072781 \n",
" 2.536347 \n",
" 1.378155 \n",
" -0.338321 \n",
" 0.462388 \n",
" 0.239599 \n",
" 0.098698 \n",
" 0.363787 \n",
" 0.090794 \n",
" ... \n",
" -0.018307 \n",
" 0.277838 \n",
" -0.110474 \n",
" 0.066928 \n",
" 0.128539 \n",
" -0.189115 \n",
" 0.133558 \n",
" -0.021053 \n",
" 149.62 \n",
" 0 \n",
" \n",
" \n",
" 1 \n",
" 1.191857 \n",
" 0.266151 \n",
" 0.166480 \n",
" 0.448154 \n",
" 0.060018 \n",
" -0.082361 \n",
" -0.078803 \n",
" 0.085102 \n",
" -0.255425 \n",
" -0.166974 \n",
" ... \n",
" -0.225775 \n",
" -0.638672 \n",
" 0.101288 \n",
" -0.339846 \n",
" 0.167170 \n",
" 0.125895 \n",
" -0.008983 \n",
" 0.014724 \n",
" 2.69 \n",
" 0 \n",
" \n",
" \n",
" 2 \n",
" -1.358354 \n",
" -1.340163 \n",
" 1.773209 \n",
" 0.379780 \n",
" -0.503198 \n",
" 1.800499 \n",
" 0.791461 \n",
" 0.247676 \n",
" -1.514654 \n",
" 0.207643 \n",
" ... \n",
" 0.247998 \n",
" 0.771679 \n",
" 0.909412 \n",
" -0.689281 \n",
" -0.327642 \n",
" -0.139097 \n",
" -0.055353 \n",
" -0.059752 \n",
" 378.66 \n",
" 0 \n",
" \n",
" \n",
" 3 \n",
" -0.966272 \n",
" -0.185226 \n",
" 1.792993 \n",
" -0.863291 \n",
" -0.010309 \n",
" 1.247203 \n",
" 0.237609 \n",
" 0.377436 \n",
" -1.387024 \n",
" -0.054952 \n",
" ... \n",
" -0.108300 \n",
" 0.005274 \n",
" -0.190321 \n",
" -1.175575 \n",
" 0.647376 \n",
" -0.221929 \n",
" 0.062723 \n",
" 0.061458 \n",
" 123.50 \n",
" 0 \n",
" \n",
" \n",
" 4 \n",
" -1.158233 \n",
" 0.877737 \n",
" 1.548718 \n",
" 0.403034 \n",
" -0.407193 \n",
" 0.095921 \n",
" 0.592941 \n",
" -0.270533 \n",
" 0.817739 \n",
" 0.753074 \n",
" ... \n",
" -0.009431 \n",
" 0.798278 \n",
" -0.137458 \n",
" 0.141267 \n",
" -0.206010 \n",
" 0.502292 \n",
" 0.219422 \n",
" 0.215153 \n",
" 69.99 \n",
" 0 \n",
" \n",
" \n",
"
\n",
"
5 rows × 30 columns
\n",
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"X, y = dict_creditcard['data'], dict_creditcard['target']\n",
"\n",
"plt.bar(Counter(y).keys(), Counter(y).values())\n",
"plt.title('Distribution of fraud (target)');"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Simple time series\n",
"\n",
"Generate a very simple timeseries dataset to play with. The generator assumes to generate daily data with a season, trend and noise."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" yt \n",
" \n",
" \n",
" \n",
" \n",
" 0 \n",
" -1.066620 \n",
" \n",
" \n",
" 1 \n",
" 0.303379 \n",
" \n",
" \n",
" 2 \n",
" -0.412812 \n",
" \n",
" \n",
" 3 \n",
" -0.227176 \n",
" \n",
" \n",
" 4 \n",
" 0.105581 \n",
" \n",
" \n",
"
\n",
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plt.plot(df_simpleseries['yt'])\n",
"plt.title('Timeseries yt');"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.7"
},
"toc": {
"base_numbering": 1,
"nav_menu": {},
"number_sections": true,
"sideBar": true,
"skip_h1_title": false,
"title_cell": "Table of Contents",
"title_sidebar": "Contents",
"toc_cell": false,
"toc_position": {},
"toc_section_display": true,
"toc_window_display": true
}
},
"nbformat": 4,
"nbformat_minor": 4
}