{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Data School's top 25 pandas tricks ([video](https://www.youtube.com/watch?v=RlIiVeig3hc))\n", "\n", "- Watch the [complete pandas video series](https://www.dataschool.io/easier-data-analysis-with-pandas/)\n", "- Connect on [Twitter](https://twitter.com/justmarkham), [Facebook](https://www.facebook.com/DataScienceSchool/), and [LinkedIn](https://www.linkedin.com/in/justmarkham/)\n", "- Subscribe on [YouTube](https://www.youtube.com/dataschool?sub_confirmation=1)\n", "- Join the [email newsletter](https://www.dataschool.io/subscribe/)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Table of contents\n", "\n", "1. Show installed versions\n", "2. Create an example DataFrame\n", "3. Rename columns\n", "4. Reverse row order\n", "5. Reverse column order\n", "6. Select columns by data type\n", "7. Convert strings to numbers\n", "8. Reduce DataFrame size\n", "9. Build a DataFrame from multiple files (row-wise)\n", "10. Build a DataFrame from multiple files (column-wise)\n", "11. Create a DataFrame from the clipboard\n", "12. Split a DataFrame into two random subsets\n", "13. Filter a DataFrame by multiple categories\n", "14. Filter a DataFrame by largest categories\n", "15. Handle missing values\n", "16. Split a string into multiple columns\n", "17. Expand a Series of lists into a DataFrame\n", "18. Aggregate by multiple functions\n", "19. Combine the output of an aggregation with a DataFrame\n", "20. Select a slice of rows and columns\n", "21. Reshape a MultiIndexed Series\n", "22. Create a pivot table\n", "23. Convert continuous data into categorical data\n", "24. Change display options\n", "25. Style a DataFrame\n", "26. Bonus trick: Profile a DataFrame" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Load example datasets" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "drinks = pd.read_csv('http://bit.ly/drinksbycountry')\n", "movies = pd.read_csv('http://bit.ly/imdbratings')\n", "orders = pd.read_csv('http://bit.ly/chiporders', sep='\\t')\n", "orders['item_price'] = orders.item_price.str.replace('$', '').astype('float')\n", "stocks = pd.read_csv('http://bit.ly/smallstocks', parse_dates=['Date'])\n", "titanic = pd.read_csv('http://bit.ly/kaggletrain')\n", "ufo = pd.read_csv('http://bit.ly/uforeports', parse_dates=['Time'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Show installed versions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Sometimes you need to know the pandas version you're using, especially when reading the pandas documentation. You can show the pandas version by typing:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'0.24.2'" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.__version__" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "But if you also need to know the versions of pandas' dependencies, you can use the `show_versions()` function:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "scrolled": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "INSTALLED VERSIONS\n", "------------------\n", "commit: None\n", "python: 3.7.3.final.0\n", "python-bits: 64\n", "OS: Darwin\n", "OS-release: 18.6.0\n", "machine: x86_64\n", "processor: i386\n", "byteorder: little\n", "LC_ALL: None\n", "LANG: en_US.UTF-8\n", "LOCALE: en_US.UTF-8\n", "\n", "pandas: 0.24.2\n", "pytest: None\n", "pip: 19.1.1\n", "setuptools: 41.0.1\n", "Cython: None\n", "numpy: 1.16.4\n", "scipy: None\n", "pyarrow: None\n", "xarray: None\n", "IPython: 7.5.0\n", "sphinx: None\n", "patsy: None\n", "dateutil: 2.8.0\n", "pytz: 2019.1\n", "blosc: None\n", "bottleneck: None\n", "tables: None\n", "numexpr: None\n", "feather: None\n", "matplotlib: 3.1.0\n", "openpyxl: None\n", "xlrd: None\n", "xlwt: None\n", "xlsxwriter: None\n", "lxml.etree: None\n", "bs4: None\n", "html5lib: None\n", "sqlalchemy: None\n", "pymysql: None\n", "psycopg2: None\n", "jinja2: 2.10.1\n", "s3fs: None\n", "fastparquet: None\n", "pandas_gbq: None\n", "pandas_datareader: None\n", "gcsfs: None\n" ] } ], "source": [ "pd.show_versions()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can see the versions of Python, pandas, NumPy, matplotlib, and more." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Create an example DataFrame" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's say that you want to demonstrate some pandas code. You need an example DataFrame to work with.\n", "\n", "There are many ways to do this, but my favorite way is to pass a dictionary to the DataFrame constructor, in which the dictionary keys are the column names and the dictionary values are lists of column values:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
col onecol two
0100300
1200400
\n", "
" ], "text/plain": [ " col one col two\n", "0 100 300\n", "1 200 400" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = pd.DataFrame({'col one':[100, 200], 'col two':[300, 400]})\n", "df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now if you need a much larger DataFrame, the above method will require way too much typing. In that case, you can use NumPy's `random.rand()` function, tell it the number of rows and columns, and pass that to the DataFrame constructor:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
01234567
00.7650500.6724380.6585160.5152310.3145630.7596570.8388040.154178
10.5267860.2588710.0325770.6352550.0083150.8277650.5743180.781200
20.1140550.7951560.1442480.1617380.6248360.2232520.4922550.274132
30.0140800.0973080.4226320.0989520.4710070.3075620.5030400.317663
\n", "
" ], "text/plain": [ " 0 1 2 3 4 5 6 \\\n", "0 0.765050 0.672438 0.658516 0.515231 0.314563 0.759657 0.838804 \n", "1 0.526786 0.258871 0.032577 0.635255 0.008315 0.827765 0.574318 \n", "2 0.114055 0.795156 0.144248 0.161738 0.624836 0.223252 0.492255 \n", "3 0.014080 0.097308 0.422632 0.098952 0.471007 0.307562 0.503040 \n", "\n", " 7 \n", "0 0.154178 \n", "1 0.781200 \n", "2 0.274132 \n", "3 0.317663 " ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.DataFrame(np.random.rand(4, 8))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "That's pretty good, but if you also want non-numeric column names, you can coerce a string of letters to a list and then pass that list to the columns parameter:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
abcdefgh
00.9291560.6656030.9348040.4983390.5981480.7172800.3044520.311813
10.3087360.4183610.7582430.7335210.1452160.8229320.3696320.470175
20.9646710.4391960.3775380.5476040.1381130.7899900.6153330.540587
30.1080640.8341340.3670980.1320730.6087100.7836280.3475940.836521
\n", "
" ], "text/plain": [ " a b c d e f g \\\n", "0 0.929156 0.665603 0.934804 0.498339 0.598148 0.717280 0.304452 \n", "1 0.308736 0.418361 0.758243 0.733521 0.145216 0.822932 0.369632 \n", "2 0.964671 0.439196 0.377538 0.547604 0.138113 0.789990 0.615333 \n", "3 0.108064 0.834134 0.367098 0.132073 0.608710 0.783628 0.347594 \n", "\n", " h \n", "0 0.311813 \n", "1 0.470175 \n", "2 0.540587 \n", "3 0.836521 " ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.DataFrame(np.random.rand(4, 8), columns=list('abcdefgh'))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you might guess, your string will need to have the same number of characters as there are columns." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Rename columns" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's take a look at the example DataFrame we created in the last trick:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
col onecol two
0100300
1200400
\n", "
" ], "text/plain": [ " col one col two\n", "0 100 300\n", "1 200 400" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "I prefer to use dot notation to select pandas columns, but that won't work since the column names have spaces. Let's fix this.\n", "\n", "The most flexible method for renaming columns is the `rename()` method. You pass it a dictionary in which the keys are the old names and the values are the new names, and you also specify the axis:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "df = df.rename({'col one':'col_one', 'col two':'col_two'}, axis='columns')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The best thing about this method is that you can use it to rename any number of columns, whether it be just one column or all columns.\n", "\n", "Now if you're going to rename all of the columns at once, a simpler method is just to overwrite the columns attribute of the DataFrame:" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "df.columns = ['col_one', 'col_two']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now if the only thing you're doing is replacing spaces with underscores, an even better method is to use the `str.replace()` method, since you don't have to type out all of the column names:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "df.columns = df.columns.str.replace(' ', '_')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "All three of these methods have the same result, which is to rename the columns so that they don't have any spaces:" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
col_onecol_two
0100300
1200400
\n", "
" ], "text/plain": [ " col_one col_two\n", "0 100 300\n", "1 200 400" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally, if you just need to add a prefix or suffix to all of your column names, you can use the `add_prefix()` method..." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X_col_oneX_col_two
0100300
1200400
\n", "
" ], "text/plain": [ " X_col_one X_col_two\n", "0 100 300\n", "1 200 400" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.add_prefix('X_')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "...or the `add_suffix()` method:" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
col_one_Ycol_two_Y
0100300
1200400
\n", "
" ], "text/plain": [ " col_one_Y col_two_Y\n", "0 100 300\n", "1 200 400" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.add_suffix('_Y')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Reverse row order" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's take a look at the drinks DataFrame:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countrybeer_servingsspirit_servingswine_servingstotal_litres_of_pure_alcoholcontinent
0Afghanistan0000.0Asia
1Albania89132544.9Europe
2Algeria250140.7Africa
3Andorra24513831212.4Europe
4Angola21757455.9Africa
\n", "
" ], "text/plain": [ " country beer_servings spirit_servings wine_servings \\\n", "0 Afghanistan 0 0 0 \n", "1 Albania 89 132 54 \n", "2 Algeria 25 0 14 \n", "3 Andorra 245 138 312 \n", "4 Angola 217 57 45 \n", "\n", " total_litres_of_pure_alcohol continent \n", "0 0.0 Asia \n", "1 4.9 Europe \n", "2 0.7 Africa \n", "3 12.4 Europe \n", "4 5.9 Africa " ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "drinks.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is a dataset of average alcohol consumption by country. What if you wanted to reverse the order of the rows?\n", "\n", "The most straightforward method is to use the `loc` accessor and pass it `::-1`, which is the same slicing notation used to reverse a Python list:" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countrybeer_servingsspirit_servingswine_servingstotal_litres_of_pure_alcoholcontinent
192Zimbabwe641844.7Africa
191Zambia321942.5Africa
190Yemen6000.1Asia
189Vietnam111212.0Asia
188Venezuela33310037.7South America
\n", "
" ], "text/plain": [ " country beer_servings spirit_servings wine_servings \\\n", "192 Zimbabwe 64 18 4 \n", "191 Zambia 32 19 4 \n", "190 Yemen 6 0 0 \n", "189 Vietnam 111 2 1 \n", "188 Venezuela 333 100 3 \n", "\n", " total_litres_of_pure_alcohol continent \n", "192 4.7 Africa \n", "191 2.5 Africa \n", "190 0.1 Asia \n", "189 2.0 Asia \n", "188 7.7 South America " ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "drinks.loc[::-1].head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "What if you also wanted to reset the index so that it starts at zero?\n", "\n", "You would use the `reset_index()` method and tell it to drop the old index entirely:" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countrybeer_servingsspirit_servingswine_servingstotal_litres_of_pure_alcoholcontinent
0Zimbabwe641844.7Africa
1Zambia321942.5Africa
2Yemen6000.1Asia
3Vietnam111212.0Asia
4Venezuela33310037.7South America
\n", "
" ], "text/plain": [ " country beer_servings spirit_servings wine_servings \\\n", "0 Zimbabwe 64 18 4 \n", "1 Zambia 32 19 4 \n", "2 Yemen 6 0 0 \n", "3 Vietnam 111 2 1 \n", "4 Venezuela 333 100 3 \n", "\n", " total_litres_of_pure_alcohol continent \n", "0 4.7 Africa \n", "1 2.5 Africa \n", "2 0.1 Asia \n", "3 2.0 Asia \n", "4 7.7 South America " ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "drinks.loc[::-1].reset_index(drop=True).head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you can see, the rows are in reverse order but the index has been reset to the default integer index." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Reverse column order" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Similar to the previous trick, you can also use `loc` to reverse the left-to-right order of your columns:" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
continenttotal_litres_of_pure_alcoholwine_servingsspirit_servingsbeer_servingscountry
0Asia0.0000Afghanistan
1Europe4.95413289Albania
2Africa0.714025Algeria
3Europe12.4312138245Andorra
4Africa5.94557217Angola
\n", "
" ], "text/plain": [ " continent total_litres_of_pure_alcohol wine_servings spirit_servings \\\n", "0 Asia 0.0 0 0 \n", "1 Europe 4.9 54 132 \n", "2 Africa 0.7 14 0 \n", "3 Europe 12.4 312 138 \n", "4 Africa 5.9 45 57 \n", "\n", " beer_servings country \n", "0 0 Afghanistan \n", "1 89 Albania \n", "2 25 Algeria \n", "3 245 Andorra \n", "4 217 Angola " ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "drinks.loc[:, ::-1].head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The colon before the comma means \"select all rows\", and the `::-1` after the comma means \"reverse the columns\", which is why \"country\" is now on the right side." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 6. Select columns by data type" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here are the data types of the drinks DataFrame:" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "country object\n", "beer_servings int64\n", "spirit_servings int64\n", "wine_servings int64\n", "total_litres_of_pure_alcohol float64\n", "continent object\n", "dtype: object" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "drinks.dtypes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's say you need to select only the numeric columns. You can use the `select_dtypes()` method:" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
beer_servingsspirit_servingswine_servingstotal_litres_of_pure_alcohol
00000.0
189132544.9
2250140.7
324513831212.4
421757455.9
\n", "
" ], "text/plain": [ " beer_servings spirit_servings wine_servings total_litres_of_pure_alcohol\n", "0 0 0 0 0.0\n", "1 89 132 54 4.9\n", "2 25 0 14 0.7\n", "3 245 138 312 12.4\n", "4 217 57 45 5.9" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "drinks.select_dtypes(include='number').head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This includes both int and float columns.\n", "\n", "You could also use this method to select just the object columns:" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countrycontinent
0AfghanistanAsia
1AlbaniaEurope
2AlgeriaAfrica
3AndorraEurope
4AngolaAfrica
\n", "
" ], "text/plain": [ " country continent\n", "0 Afghanistan Asia\n", "1 Albania Europe\n", "2 Algeria Africa\n", "3 Andorra Europe\n", "4 Angola Africa" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "drinks.select_dtypes(include='object').head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can tell it to include multiple data types by passing a list:" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countrybeer_servingsspirit_servingswine_servingstotal_litres_of_pure_alcoholcontinent
0Afghanistan0000.0Asia
1Albania89132544.9Europe
2Algeria250140.7Africa
3Andorra24513831212.4Europe
4Angola21757455.9Africa
\n", "
" ], "text/plain": [ " country beer_servings spirit_servings wine_servings \\\n", "0 Afghanistan 0 0 0 \n", "1 Albania 89 132 54 \n", "2 Algeria 25 0 14 \n", "3 Andorra 245 138 312 \n", "4 Angola 217 57 45 \n", "\n", " total_litres_of_pure_alcohol continent \n", "0 0.0 Asia \n", "1 4.9 Europe \n", "2 0.7 Africa \n", "3 12.4 Europe \n", "4 5.9 Africa " ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "drinks.select_dtypes(include=['number', 'object', 'category', 'datetime']).head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can also tell it to exclude certain data types:" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countrycontinent
0AfghanistanAsia
1AlbaniaEurope
2AlgeriaAfrica
3AndorraEurope
4AngolaAfrica
\n", "
" ], "text/plain": [ " country continent\n", "0 Afghanistan Asia\n", "1 Albania Europe\n", "2 Algeria Africa\n", "3 Andorra Europe\n", "4 Angola Africa" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "drinks.select_dtypes(exclude='number').head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 7. Convert strings to numbers" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's create another example DataFrame:" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
col_onecol_twocol_three
01.14.47.7
12.25.58.8
23.36.6-
\n", "
" ], "text/plain": [ " col_one col_two col_three\n", "0 1.1 4.4 7.7\n", "1 2.2 5.5 8.8\n", "2 3.3 6.6 -" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = pd.DataFrame({'col_one':['1.1', '2.2', '3.3'],\n", " 'col_two':['4.4', '5.5', '6.6'],\n", " 'col_three':['7.7', '8.8', '-']})\n", "df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "These numbers are actually stored as strings, which results in object columns:" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "col_one object\n", "col_two object\n", "col_three object\n", "dtype: object" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.dtypes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In order to do mathematical operations on these columns, we need to convert the data types to numeric. You can use the `astype()` method on the first two columns:" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "col_one float64\n", "col_two float64\n", "col_three object\n", "dtype: object" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.astype({'col_one':'float', 'col_two':'float'}).dtypes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "However, this would have resulted in an error if you tried to use it on the third column, because that column contains a dash to represent zero and pandas doesn't understand how to handle it.\n", "\n", "Instead, you can use the `to_numeric()` function on the third column and tell it to convert any invalid input into `NaN` values:" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 7.7\n", "1 8.8\n", "2 NaN\n", "Name: col_three, dtype: float64" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.to_numeric(df.col_three, errors='coerce')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you know that the `NaN` values actually represent zeros, you can fill them with zeros using the `fillna()` method:" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 7.7\n", "1 8.8\n", "2 0.0\n", "Name: col_three, dtype: float64" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.to_numeric(df.col_three, errors='coerce').fillna(0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally, you can apply this function to the entire DataFrame all at once by using the `apply()` method:" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
col_onecol_twocol_three
01.14.47.7
12.25.58.8
23.36.60.0
\n", "
" ], "text/plain": [ " col_one col_two col_three\n", "0 1.1 4.4 7.7\n", "1 2.2 5.5 8.8\n", "2 3.3 6.6 0.0" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = df.apply(pd.to_numeric, errors='coerce').fillna(0)\n", "df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This one line of code accomplishes our goal, because all of the data types have now been converted to float:" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "col_one float64\n", "col_two float64\n", "col_three float64\n", "dtype: object" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.dtypes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8. Reduce DataFrame size" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "pandas DataFrames are designed to fit into memory, and so sometimes you need to reduce the DataFrame size in order to work with it on your system.\n", "\n", "Here's the size of the drinks DataFrame:" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "RangeIndex: 193 entries, 0 to 192\n", "Data columns (total 6 columns):\n", "country 193 non-null object\n", "beer_servings 193 non-null int64\n", "spirit_servings 193 non-null int64\n", "wine_servings 193 non-null int64\n", "total_litres_of_pure_alcohol 193 non-null float64\n", "continent 193 non-null object\n", "dtypes: float64(1), int64(3), object(2)\n", "memory usage: 30.4 KB\n" ] } ], "source": [ "drinks.info(memory_usage='deep')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can see that it currently uses 30.4 KB.\n", "\n", "If you're having performance problems with your DataFrame, or you can't even read it into memory, there are two easy steps you can take during the file reading process to reduce the DataFrame size.\n", "\n", "The first step is to only read in the columns that you actually need, which we specify with the \"usecols\" parameter:" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "RangeIndex: 193 entries, 0 to 192\n", "Data columns (total 2 columns):\n", "beer_servings 193 non-null int64\n", "continent 193 non-null object\n", "dtypes: int64(1), object(1)\n", "memory usage: 13.6 KB\n" ] } ], "source": [ "cols = ['beer_servings', 'continent']\n", "small_drinks = pd.read_csv('http://bit.ly/drinksbycountry', usecols=cols)\n", "small_drinks.info(memory_usage='deep')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By only reading in these two columns, we've reduced the DataFrame size to 13.6 KB.\n", "\n", "The second step is to convert any object columns containing categorical data to the category data type, which we specify with the \"dtype\" parameter:" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "RangeIndex: 193 entries, 0 to 192\n", "Data columns (total 2 columns):\n", "beer_servings 193 non-null int64\n", "continent 193 non-null category\n", "dtypes: category(1), int64(1)\n", "memory usage: 2.3 KB\n" ] } ], "source": [ "dtypes = {'continent':'category'}\n", "smaller_drinks = pd.read_csv('http://bit.ly/drinksbycountry', usecols=cols, dtype=dtypes)\n", "smaller_drinks.info(memory_usage='deep')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By reading in the continent column as the category data type, we've further reduced the DataFrame size to 2.3 KB.\n", "\n", "Keep in mind that the category data type will only reduce memory usage if you have a small number of categories relative to the number of rows." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 9. Build a DataFrame from multiple files (row-wise)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's say that your dataset is spread across multiple files, but you want to read the dataset into a single DataFrame.\n", "\n", "For example, I have a small dataset of stock data in which each CSV file only includes a single day. Here's the first day:" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
DateCloseVolumeSymbol
02016-10-0331.5014070500CSCO
12016-10-03112.5221701800AAPL
22016-10-0357.4219189500MSFT
\n", "
" ], "text/plain": [ " Date Close Volume Symbol\n", "0 2016-10-03 31.50 14070500 CSCO\n", "1 2016-10-03 112.52 21701800 AAPL\n", "2 2016-10-03 57.42 19189500 MSFT" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.read_csv('data/stocks1.csv')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here's the second day:" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
DateCloseVolumeSymbol
02016-10-04113.0029736800AAPL
12016-10-0457.2420085900MSFT
22016-10-0431.3518460400CSCO
\n", "
" ], "text/plain": [ " Date Close Volume Symbol\n", "0 2016-10-04 113.00 29736800 AAPL\n", "1 2016-10-04 57.24 20085900 MSFT\n", "2 2016-10-04 31.35 18460400 CSCO" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.read_csv('data/stocks2.csv')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And here's the third day:" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
DateCloseVolumeSymbol
02016-10-0557.6416726400MSFT
12016-10-0531.5911808600CSCO
22016-10-05113.0521453100AAPL
\n", "
" ], "text/plain": [ " Date Close Volume Symbol\n", "0 2016-10-05 57.64 16726400 MSFT\n", "1 2016-10-05 31.59 11808600 CSCO\n", "2 2016-10-05 113.05 21453100 AAPL" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.read_csv('data/stocks3.csv')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You could read each CSV file into its own DataFrame, combine them together, and then delete the original DataFrames, but that would be memory inefficient and require a lot of code.\n", "\n", "A better solution is to use the built-in glob module:" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [], "source": [ "from glob import glob" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can pass a pattern to `glob()`, including wildcard characters, and it will return a list of all files that match that pattern.\n", "\n", "In this case, glob is looking in the \"data\" subdirectory for all CSV files that start with the word \"stocks\":" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['data/stocks1.csv', 'data/stocks2.csv', 'data/stocks3.csv']" ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "stock_files = sorted(glob('data/stocks*.csv'))\n", "stock_files" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "glob returns filenames in an arbitrary order, which is why we sorted the list using Python's built-in `sorted()` function.\n", "\n", "We can then use a generator expression to read each of the files using `read_csv()` and pass the results to the `concat()` function, which will concatenate the rows into a single DataFrame:" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
DateCloseVolumeSymbol
02016-10-0331.5014070500CSCO
12016-10-03112.5221701800AAPL
22016-10-0357.4219189500MSFT
02016-10-04113.0029736800AAPL
12016-10-0457.2420085900MSFT
22016-10-0431.3518460400CSCO
02016-10-0557.6416726400MSFT
12016-10-0531.5911808600CSCO
22016-10-05113.0521453100AAPL
\n", "
" ], "text/plain": [ " Date Close Volume Symbol\n", "0 2016-10-03 31.50 14070500 CSCO\n", "1 2016-10-03 112.52 21701800 AAPL\n", "2 2016-10-03 57.42 19189500 MSFT\n", "0 2016-10-04 113.00 29736800 AAPL\n", "1 2016-10-04 57.24 20085900 MSFT\n", "2 2016-10-04 31.35 18460400 CSCO\n", "0 2016-10-05 57.64 16726400 MSFT\n", "1 2016-10-05 31.59 11808600 CSCO\n", "2 2016-10-05 113.05 21453100 AAPL" ] }, "execution_count": 39, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.concat((pd.read_csv(file) for file in stock_files))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Unfortunately, there are now duplicate values in the index. To avoid that, we can tell the `concat()` function to ignore the index and instead use the default integer index:" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
DateCloseVolumeSymbol
02016-10-0331.5014070500CSCO
12016-10-03112.5221701800AAPL
22016-10-0357.4219189500MSFT
32016-10-04113.0029736800AAPL
42016-10-0457.2420085900MSFT
52016-10-0431.3518460400CSCO
62016-10-0557.6416726400MSFT
72016-10-0531.5911808600CSCO
82016-10-05113.0521453100AAPL
\n", "
" ], "text/plain": [ " Date Close Volume Symbol\n", "0 2016-10-03 31.50 14070500 CSCO\n", "1 2016-10-03 112.52 21701800 AAPL\n", "2 2016-10-03 57.42 19189500 MSFT\n", "3 2016-10-04 113.00 29736800 AAPL\n", "4 2016-10-04 57.24 20085900 MSFT\n", "5 2016-10-04 31.35 18460400 CSCO\n", "6 2016-10-05 57.64 16726400 MSFT\n", "7 2016-10-05 31.59 11808600 CSCO\n", "8 2016-10-05 113.05 21453100 AAPL" ] }, "execution_count": 40, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.concat((pd.read_csv(file) for file in stock_files), ignore_index=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 10. Build a DataFrame from multiple files (column-wise)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The previous trick is useful when each file contains rows from your dataset. But what if each file instead contains columns from your dataset?\n", "\n", "Here's an example in which the drinks dataset has been split into two CSV files, and each file contains three columns:" ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countrybeer_servingsspirit_servings
0Afghanistan00
1Albania89132
2Algeria250
3Andorra245138
4Angola21757
\n", "
" ], "text/plain": [ " country beer_servings spirit_servings\n", "0 Afghanistan 0 0\n", "1 Albania 89 132\n", "2 Algeria 25 0\n", "3 Andorra 245 138\n", "4 Angola 217 57" ] }, "execution_count": 41, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.read_csv('data/drinks1.csv').head()" ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
wine_servingstotal_litres_of_pure_alcoholcontinent
000.0Asia
1544.9Europe
2140.7Africa
331212.4Europe
4455.9Africa
\n", "
" ], "text/plain": [ " wine_servings total_litres_of_pure_alcohol continent\n", "0 0 0.0 Asia\n", "1 54 4.9 Europe\n", "2 14 0.7 Africa\n", "3 312 12.4 Europe\n", "4 45 5.9 Africa" ] }, "execution_count": 42, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.read_csv('data/drinks2.csv').head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Similar to the previous trick, we'll start by using `glob()`:" ] }, { "cell_type": "code", "execution_count": 43, "metadata": {}, "outputs": [], "source": [ "drink_files = sorted(glob('data/drinks*.csv'))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And this time, we'll tell the `concat()` function to concatenate along the columns axis:" ] }, { "cell_type": "code", "execution_count": 44, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countrybeer_servingsspirit_servingswine_servingstotal_litres_of_pure_alcoholcontinent
0Afghanistan0000.0Asia
1Albania89132544.9Europe
2Algeria250140.7Africa
3Andorra24513831212.4Europe
4Angola21757455.9Africa
\n", "
" ], "text/plain": [ " country beer_servings spirit_servings wine_servings \\\n", "0 Afghanistan 0 0 0 \n", "1 Albania 89 132 54 \n", "2 Algeria 25 0 14 \n", "3 Andorra 245 138 312 \n", "4 Angola 217 57 45 \n", "\n", " total_litres_of_pure_alcohol continent \n", "0 0.0 Asia \n", "1 4.9 Europe \n", "2 0.7 Africa \n", "3 12.4 Europe \n", "4 5.9 Africa " ] }, "execution_count": 44, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.concat((pd.read_csv(file) for file in drink_files), axis='columns').head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now our DataFrame has all six columns." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 11. Create a DataFrame from the clipboard" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's say that you have some data stored in an Excel spreadsheet or a [Google Sheet](https://docs.google.com/spreadsheets/d/1ipv_HAykbky8OXUubs9eLL-LQ1rAkexXG61-B4jd0Rc/edit?usp=sharing), and you want to get it into a DataFrame as quickly as possible.\n", "\n", "Just select the data and copy it to the clipboard. Then, you can use the `read_clipboard()` function to read it into a DataFrame:" ] }, { "cell_type": "code", "execution_count": 45, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Column AColumn BColumn C
014.4seven
125.5eight
236.6nine
\n", "
" ], "text/plain": [ " Column A Column B Column C\n", "0 1 4.4 seven\n", "1 2 5.5 eight\n", "2 3 6.6 nine" ] }, "execution_count": 45, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = pd.read_clipboard()\n", "df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Just like the `read_csv()` function, `read_clipboard()` automatically detects the correct data type for each column:" ] }, { "cell_type": "code", "execution_count": 46, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Column A int64\n", "Column B float64\n", "Column C object\n", "dtype: object" ] }, "execution_count": 46, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.dtypes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's copy one other dataset to the clipboard:" ] }, { "cell_type": "code", "execution_count": 47, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
LeftRight
Alice1040
Bob2050
Charlie3060
\n", "
" ], "text/plain": [ " Left Right\n", "Alice 10 40\n", "Bob 20 50\n", "Charlie 30 60" ] }, "execution_count": 47, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = pd.read_clipboard()\n", "df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Amazingly, pandas has even identified the first column as the index:" ] }, { "cell_type": "code", "execution_count": 48, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Index(['Alice', 'Bob', 'Charlie'], dtype='object')" ] }, "execution_count": 48, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.index" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Keep in mind that if you want your work to be reproducible in the future, `read_clipboard()` is not the recommended approach." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 12. Split a DataFrame into two random subsets" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's say that you want to split a DataFrame into two parts, randomly assigning 75% of the rows to one DataFrame and the other 25% to a second DataFrame.\n", "\n", "For example, we have a DataFrame of movie ratings with 979 rows:" ] }, { "cell_type": "code", "execution_count": 49, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "979" ] }, "execution_count": 49, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(movies)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can use the `sample()` method to randomly select 75% of the rows and assign them to the \"movies_1\" DataFrame:" ] }, { "cell_type": "code", "execution_count": 50, "metadata": {}, "outputs": [], "source": [ "movies_1 = movies.sample(frac=0.75, random_state=1234)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then we can use the `drop()` method to drop all rows that are in \"movies_1\" and assign the remaining rows to \"movies_2\":" ] }, { "cell_type": "code", "execution_count": 51, "metadata": {}, "outputs": [], "source": [ "movies_2 = movies.drop(movies_1.index)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can see that the total number of rows is correct:" ] }, { "cell_type": "code", "execution_count": 52, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "979" ] }, "execution_count": 52, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(movies_1) + len(movies_2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And you can see from the index that every movie is in either \"movies_1\":" ] }, { "cell_type": "code", "execution_count": 53, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Int64Index([ 0, 2, 5, 6, 7, 8, 9, 11, 13, 16,\n", " ...\n", " 966, 967, 969, 971, 972, 974, 975, 976, 977, 978],\n", " dtype='int64', length=734)" ] }, "execution_count": 53, "metadata": {}, "output_type": "execute_result" } ], "source": [ "movies_1.index.sort_values()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "...or \"movies_2\":" ] }, { "cell_type": "code", "execution_count": 54, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Int64Index([ 1, 3, 4, 10, 12, 14, 15, 18, 26, 30,\n", " ...\n", " 931, 934, 937, 941, 950, 954, 960, 968, 970, 973],\n", " dtype='int64', length=245)" ] }, "execution_count": 54, "metadata": {}, "output_type": "execute_result" } ], "source": [ "movies_2.index.sort_values()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Keep in mind that this approach will not work if your index values are not unique." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 13. Filter a DataFrame by multiple categories" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's take a look at the movies DataFrame:" ] }, { "cell_type": "code", "execution_count": 55, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
star_ratingtitlecontent_ratinggenredurationactors_list
09.3The Shawshank RedemptionRCrime142[u'Tim Robbins', u'Morgan Freeman', u'Bob Gunt...
19.2The GodfatherRCrime175[u'Marlon Brando', u'Al Pacino', u'James Caan']
29.1The Godfather: Part IIRCrime200[u'Al Pacino', u'Robert De Niro', u'Robert Duv...
39.0The Dark KnightPG-13Action152[u'Christian Bale', u'Heath Ledger', u'Aaron E...
48.9Pulp FictionRCrime154[u'John Travolta', u'Uma Thurman', u'Samuel L....
\n", "
" ], "text/plain": [ " star_rating title content_rating genre duration \\\n", "0 9.3 The Shawshank Redemption R Crime 142 \n", "1 9.2 The Godfather R Crime 175 \n", "2 9.1 The Godfather: Part II R Crime 200 \n", "3 9.0 The Dark Knight PG-13 Action 152 \n", "4 8.9 Pulp Fiction R Crime 154 \n", "\n", " actors_list \n", "0 [u'Tim Robbins', u'Morgan Freeman', u'Bob Gunt... \n", "1 [u'Marlon Brando', u'Al Pacino', u'James Caan'] \n", "2 [u'Al Pacino', u'Robert De Niro', u'Robert Duv... \n", "3 [u'Christian Bale', u'Heath Ledger', u'Aaron E... \n", "4 [u'John Travolta', u'Uma Thurman', u'Samuel L.... " ] }, "execution_count": 55, "metadata": {}, "output_type": "execute_result" } ], "source": [ "movies.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "One of the columns is genre:" ] }, { "cell_type": "code", "execution_count": 56, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array(['Crime', 'Action', 'Drama', 'Western', 'Adventure', 'Biography',\n", " 'Comedy', 'Animation', 'Mystery', 'Horror', 'Film-Noir', 'Sci-Fi',\n", " 'History', 'Thriller', 'Family', 'Fantasy'], dtype=object)" ] }, "execution_count": 56, "metadata": {}, "output_type": "execute_result" } ], "source": [ "movies.genre.unique()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If we wanted to filter the DataFrame to only show movies with the genre Action or Drama or Western, we could use multiple conditions separated by the \"or\" operator:" ] }, { "cell_type": "code", "execution_count": 57, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
star_ratingtitlecontent_ratinggenredurationactors_list
39.0The Dark KnightPG-13Action152[u'Christian Bale', u'Heath Ledger', u'Aaron E...
58.912 Angry MenNOT RATEDDrama96[u'Henry Fonda', u'Lee J. Cobb', u'Martin Bals...
68.9The Good, the Bad and the UglyNOT RATEDWestern161[u'Clint Eastwood', u'Eli Wallach', u'Lee Van ...
98.9Fight ClubRDrama139[u'Brad Pitt', u'Edward Norton', u'Helena Bonh...
118.8InceptionPG-13Action148[u'Leonardo DiCaprio', u'Joseph Gordon-Levitt'...
\n", "
" ], "text/plain": [ " star_rating title content_rating genre \\\n", "3 9.0 The Dark Knight PG-13 Action \n", "5 8.9 12 Angry Men NOT RATED Drama \n", "6 8.9 The Good, the Bad and the Ugly NOT RATED Western \n", "9 8.9 Fight Club R Drama \n", "11 8.8 Inception PG-13 Action \n", "\n", " duration actors_list \n", "3 152 [u'Christian Bale', u'Heath Ledger', u'Aaron E... \n", "5 96 [u'Henry Fonda', u'Lee J. Cobb', u'Martin Bals... \n", "6 161 [u'Clint Eastwood', u'Eli Wallach', u'Lee Van ... \n", "9 139 [u'Brad Pitt', u'Edward Norton', u'Helena Bonh... \n", "11 148 [u'Leonardo DiCaprio', u'Joseph Gordon-Levitt'... " ] }, "execution_count": 57, "metadata": {}, "output_type": "execute_result" } ], "source": [ "movies[(movies.genre == 'Action') |\n", " (movies.genre == 'Drama') |\n", " (movies.genre == 'Western')].head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "However, you can actually rewrite this code more clearly by using the `isin()` method and passing it a list of genres:" ] }, { "cell_type": "code", "execution_count": 58, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
star_ratingtitlecontent_ratinggenredurationactors_list
39.0The Dark KnightPG-13Action152[u'Christian Bale', u'Heath Ledger', u'Aaron E...
58.912 Angry MenNOT RATEDDrama96[u'Henry Fonda', u'Lee J. Cobb', u'Martin Bals...
68.9The Good, the Bad and the UglyNOT RATEDWestern161[u'Clint Eastwood', u'Eli Wallach', u'Lee Van ...
98.9Fight ClubRDrama139[u'Brad Pitt', u'Edward Norton', u'Helena Bonh...
118.8InceptionPG-13Action148[u'Leonardo DiCaprio', u'Joseph Gordon-Levitt'...
\n", "
" ], "text/plain": [ " star_rating title content_rating genre \\\n", "3 9.0 The Dark Knight PG-13 Action \n", "5 8.9 12 Angry Men NOT RATED Drama \n", "6 8.9 The Good, the Bad and the Ugly NOT RATED Western \n", "9 8.9 Fight Club R Drama \n", "11 8.8 Inception PG-13 Action \n", "\n", " duration actors_list \n", "3 152 [u'Christian Bale', u'Heath Ledger', u'Aaron E... \n", "5 96 [u'Henry Fonda', u'Lee J. Cobb', u'Martin Bals... \n", "6 161 [u'Clint Eastwood', u'Eli Wallach', u'Lee Van ... \n", "9 139 [u'Brad Pitt', u'Edward Norton', u'Helena Bonh... \n", "11 148 [u'Leonardo DiCaprio', u'Joseph Gordon-Levitt'... " ] }, "execution_count": 58, "metadata": {}, "output_type": "execute_result" } ], "source": [ "movies[movies.genre.isin(['Action', 'Drama', 'Western'])].head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And if you want to reverse this filter, so that you are excluding (rather than including) those three genres, you can put a tilde in front of the condition:" ] }, { "cell_type": "code", "execution_count": 59, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
star_ratingtitlecontent_ratinggenredurationactors_list
09.3The Shawshank RedemptionRCrime142[u'Tim Robbins', u'Morgan Freeman', u'Bob Gunt...
19.2The GodfatherRCrime175[u'Marlon Brando', u'Al Pacino', u'James Caan']
29.1The Godfather: Part IIRCrime200[u'Al Pacino', u'Robert De Niro', u'Robert Duv...
48.9Pulp FictionRCrime154[u'John Travolta', u'Uma Thurman', u'Samuel L....
78.9The Lord of the Rings: The Return of the KingPG-13Adventure201[u'Elijah Wood', u'Viggo Mortensen', u'Ian McK...
\n", "
" ], "text/plain": [ " star_rating title content_rating \\\n", "0 9.3 The Shawshank Redemption R \n", "1 9.2 The Godfather R \n", "2 9.1 The Godfather: Part II R \n", "4 8.9 Pulp Fiction R \n", "7 8.9 The Lord of the Rings: The Return of the King PG-13 \n", "\n", " genre duration actors_list \n", "0 Crime 142 [u'Tim Robbins', u'Morgan Freeman', u'Bob Gunt... \n", "1 Crime 175 [u'Marlon Brando', u'Al Pacino', u'James Caan'] \n", "2 Crime 200 [u'Al Pacino', u'Robert De Niro', u'Robert Duv... \n", "4 Crime 154 [u'John Travolta', u'Uma Thurman', u'Samuel L.... \n", "7 Adventure 201 [u'Elijah Wood', u'Viggo Mortensen', u'Ian McK... " ] }, "execution_count": 59, "metadata": {}, "output_type": "execute_result" } ], "source": [ "movies[~movies.genre.isin(['Action', 'Drama', 'Western'])].head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This works because tilde is the \"not\" operator in Python." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 14. Filter a DataFrame by largest categories" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's say that you needed to filter the movies DataFrame by genre, but only include the 3 largest genres.\n", "\n", "We'll start by taking the `value_counts()` of genre and saving it as a Series called counts:" ] }, { "cell_type": "code", "execution_count": 60, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Drama 278\n", "Comedy 156\n", "Action 136\n", "Crime 124\n", "Biography 77\n", "Adventure 75\n", "Animation 62\n", "Horror 29\n", "Mystery 16\n", "Western 9\n", "Sci-Fi 5\n", "Thriller 5\n", "Film-Noir 3\n", "Family 2\n", "Fantasy 1\n", "History 1\n", "Name: genre, dtype: int64" ] }, "execution_count": 60, "metadata": {}, "output_type": "execute_result" } ], "source": [ "counts = movies.genre.value_counts()\n", "counts" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The Series method `nlargest()` makes it easy to select the 3 largest values in this Series:" ] }, { "cell_type": "code", "execution_count": 61, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Drama 278\n", "Comedy 156\n", "Action 136\n", "Name: genre, dtype: int64" ] }, "execution_count": 61, "metadata": {}, "output_type": "execute_result" } ], "source": [ "counts.nlargest(3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And all we actually need from this Series is the index:" ] }, { "cell_type": "code", "execution_count": 62, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Index(['Drama', 'Comedy', 'Action'], dtype='object')" ] }, "execution_count": 62, "metadata": {}, "output_type": "execute_result" } ], "source": [ "counts.nlargest(3).index" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally, we can pass the index object to `isin()`, and it will be treated like a list of genres:" ] }, { "cell_type": "code", "execution_count": 63, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
star_ratingtitlecontent_ratinggenredurationactors_list
39.0The Dark KnightPG-13Action152[u'Christian Bale', u'Heath Ledger', u'Aaron E...
58.912 Angry MenNOT RATEDDrama96[u'Henry Fonda', u'Lee J. Cobb', u'Martin Bals...
98.9Fight ClubRDrama139[u'Brad Pitt', u'Edward Norton', u'Helena Bonh...
118.8InceptionPG-13Action148[u'Leonardo DiCaprio', u'Joseph Gordon-Levitt'...
128.8Star Wars: Episode V - The Empire Strikes BackPGAction124[u'Mark Hamill', u'Harrison Ford', u'Carrie Fi...
\n", "
" ], "text/plain": [ " star_rating title \\\n", "3 9.0 The Dark Knight \n", "5 8.9 12 Angry Men \n", "9 8.9 Fight Club \n", "11 8.8 Inception \n", "12 8.8 Star Wars: Episode V - The Empire Strikes Back \n", "\n", " content_rating genre duration \\\n", "3 PG-13 Action 152 \n", "5 NOT RATED Drama 96 \n", "9 R Drama 139 \n", "11 PG-13 Action 148 \n", "12 PG Action 124 \n", "\n", " actors_list \n", "3 [u'Christian Bale', u'Heath Ledger', u'Aaron E... \n", "5 [u'Henry Fonda', u'Lee J. Cobb', u'Martin Bals... \n", "9 [u'Brad Pitt', u'Edward Norton', u'Helena Bonh... \n", "11 [u'Leonardo DiCaprio', u'Joseph Gordon-Levitt'... \n", "12 [u'Mark Hamill', u'Harrison Ford', u'Carrie Fi... " ] }, "execution_count": 63, "metadata": {}, "output_type": "execute_result" } ], "source": [ "movies[movies.genre.isin(counts.nlargest(3).index)].head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Thus, only Drama and Comedy and Action movies remain in the DataFrame." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 15. Handle missing values" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's look at a dataset of UFO sightings:" ] }, { "cell_type": "code", "execution_count": 64, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
CityColors ReportedShape ReportedStateTime
0IthacaNaNTRIANGLENY1930-06-01 22:00:00
1WillingboroNaNOTHERNJ1930-06-30 20:00:00
2HolyokeNaNOVALCO1931-02-15 14:00:00
3AbileneNaNDISKKS1931-06-01 13:00:00
4New York Worlds FairNaNLIGHTNY1933-04-18 19:00:00
\n", "
" ], "text/plain": [ " City Colors Reported Shape Reported State \\\n", "0 Ithaca NaN TRIANGLE NY \n", "1 Willingboro NaN OTHER NJ \n", "2 Holyoke NaN OVAL CO \n", "3 Abilene NaN DISK KS \n", "4 New York Worlds Fair NaN LIGHT NY \n", "\n", " Time \n", "0 1930-06-01 22:00:00 \n", "1 1930-06-30 20:00:00 \n", "2 1931-02-15 14:00:00 \n", "3 1931-06-01 13:00:00 \n", "4 1933-04-18 19:00:00 " ] }, "execution_count": 64, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ufo.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You'll notice that some of the values are missing.\n", "\n", "To find out how many values are missing in each column, you can use the `isna()` method and then take the `sum()`:" ] }, { "cell_type": "code", "execution_count": 65, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "City 25\n", "Colors Reported 15359\n", "Shape Reported 2644\n", "State 0\n", "Time 0\n", "dtype: int64" ] }, "execution_count": 65, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ufo.isna().sum()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`isna()` generated a DataFrame of True and False values, and `sum()` converted all of the True values to 1 and added them up.\n", "\n", "Similarly, you can find out the percentage of values that are missing by taking the `mean()` of `isna()`:" ] }, { "cell_type": "code", "execution_count": 66, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "City 0.001371\n", "Colors Reported 0.842004\n", "Shape Reported 0.144948\n", "State 0.000000\n", "Time 0.000000\n", "dtype: float64" ] }, "execution_count": 66, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ufo.isna().mean()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you want to drop the columns that have any missing values, you can use the `dropna()` method:" ] }, { "cell_type": "code", "execution_count": 67, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
StateTime
0NY1930-06-01 22:00:00
1NJ1930-06-30 20:00:00
2CO1931-02-15 14:00:00
3KS1931-06-01 13:00:00
4NY1933-04-18 19:00:00
\n", "
" ], "text/plain": [ " State Time\n", "0 NY 1930-06-01 22:00:00\n", "1 NJ 1930-06-30 20:00:00\n", "2 CO 1931-02-15 14:00:00\n", "3 KS 1931-06-01 13:00:00\n", "4 NY 1933-04-18 19:00:00" ] }, "execution_count": 67, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ufo.dropna(axis='columns').head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Or if you want to drop columns in which more than 10% of the values are missing, you can set a threshold for `dropna()`:" ] }, { "cell_type": "code", "execution_count": 68, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
CityStateTime
0IthacaNY1930-06-01 22:00:00
1WillingboroNJ1930-06-30 20:00:00
2HolyokeCO1931-02-15 14:00:00
3AbileneKS1931-06-01 13:00:00
4New York Worlds FairNY1933-04-18 19:00:00
\n", "
" ], "text/plain": [ " City State Time\n", "0 Ithaca NY 1930-06-01 22:00:00\n", "1 Willingboro NJ 1930-06-30 20:00:00\n", "2 Holyoke CO 1931-02-15 14:00:00\n", "3 Abilene KS 1931-06-01 13:00:00\n", "4 New York Worlds Fair NY 1933-04-18 19:00:00" ] }, "execution_count": 68, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ufo.dropna(thresh=len(ufo)*0.9, axis='columns').head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`len(ufo)` returns the total number of rows, and then we multiply that by 0.9 to tell pandas to only keep columns in which at least 90% of the values are not missing." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 16. Split a string into multiple columns" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's create another example DataFrame:" ] }, { "cell_type": "code", "execution_count": 69, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
namelocation
0John Arthur DoeLos Angeles, CA
1Jane Ann SmithWashington, DC
\n", "
" ], "text/plain": [ " name location\n", "0 John Arthur Doe Los Angeles, CA\n", "1 Jane Ann Smith Washington, DC" ] }, "execution_count": 69, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = pd.DataFrame({'name':['John Arthur Doe', 'Jane Ann Smith'],\n", " 'location':['Los Angeles, CA', 'Washington, DC']})\n", "df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "What if we wanted to split the \"name\" column into three separate columns, for first, middle, and last name? We would use the `str.split()` method and tell it to split on a space character and expand the results into a DataFrame:" ] }, { "cell_type": "code", "execution_count": 70, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
012
0JohnArthurDoe
1JaneAnnSmith
\n", "
" ], "text/plain": [ " 0 1 2\n", "0 John Arthur Doe\n", "1 Jane Ann Smith" ] }, "execution_count": 70, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.name.str.split(' ', expand=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "These three columns can actually be saved to the original DataFrame in a single assignment statement:" ] }, { "cell_type": "code", "execution_count": 71, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
namelocationfirstmiddlelast
0John Arthur DoeLos Angeles, CAJohnArthurDoe
1Jane Ann SmithWashington, DCJaneAnnSmith
\n", "
" ], "text/plain": [ " name location first middle last\n", "0 John Arthur Doe Los Angeles, CA John Arthur Doe\n", "1 Jane Ann Smith Washington, DC Jane Ann Smith" ] }, "execution_count": 71, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[['first', 'middle', 'last']] = df.name.str.split(' ', expand=True)\n", "df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "What if we wanted to split a string, but only keep one of the resulting columns? For example, let's split the location column on \"comma space\":" ] }, { "cell_type": "code", "execution_count": 72, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
01
0Los AngelesCA
1WashingtonDC
\n", "
" ], "text/plain": [ " 0 1\n", "0 Los Angeles CA\n", "1 Washington DC" ] }, "execution_count": 72, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.location.str.split(', ', expand=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If we only cared about saving the city name in column 0, we can just select that column and save it to the DataFrame:" ] }, { "cell_type": "code", "execution_count": 73, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
namelocationfirstmiddlelastcity
0John Arthur DoeLos Angeles, CAJohnArthurDoeLos Angeles
1Jane Ann SmithWashington, DCJaneAnnSmithWashington
\n", "
" ], "text/plain": [ " name location first middle last city\n", "0 John Arthur Doe Los Angeles, CA John Arthur Doe Los Angeles\n", "1 Jane Ann Smith Washington, DC Jane Ann Smith Washington" ] }, "execution_count": 73, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df['city'] = df.location.str.split(', ', expand=True)[0]\n", "df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 17. Expand a Series of lists into a DataFrame" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's create another example DataFrame:" ] }, { "cell_type": "code", "execution_count": 74, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
col_onecol_two
0a[10, 40]
1b[20, 50]
2c[30, 60]
\n", "
" ], "text/plain": [ " col_one col_two\n", "0 a [10, 40]\n", "1 b [20, 50]\n", "2 c [30, 60]" ] }, "execution_count": 74, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = pd.DataFrame({'col_one':['a', 'b', 'c'], 'col_two':[[10, 40], [20, 50], [30, 60]]})\n", "df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are two columns, and the second column contains regular Python lists of integers.\n", "\n", "If we wanted to expand the second column into its own DataFrame, we can use the `apply()` method on that column and pass it the Series constructor:" ] }, { "cell_type": "code", "execution_count": 75, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
01
01040
12050
23060
\n", "
" ], "text/plain": [ " 0 1\n", "0 10 40\n", "1 20 50\n", "2 30 60" ] }, "execution_count": 75, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_new = df.col_two.apply(pd.Series)\n", "df_new" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And by using the `concat()` function, you can combine the original DataFrame with the new DataFrame:" ] }, { "cell_type": "code", "execution_count": 76, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
col_onecol_two01
0a[10, 40]1040
1b[20, 50]2050
2c[30, 60]3060
\n", "
" ], "text/plain": [ " col_one col_two 0 1\n", "0 a [10, 40] 10 40\n", "1 b [20, 50] 20 50\n", "2 c [30, 60] 30 60" ] }, "execution_count": 76, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.concat([df, df_new], axis='columns')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 18. Aggregate by multiple functions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's look at a DataFrame of orders from the Chipotle restaurant chain:" ] }, { "cell_type": "code", "execution_count": 77, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
order_idquantityitem_namechoice_descriptionitem_price
011Chips and Fresh Tomato SalsaNaN2.39
111Izze[Clementine]3.39
211Nantucket Nectar[Apple]3.39
311Chips and Tomatillo-Green Chili SalsaNaN2.39
422Chicken Bowl[Tomatillo-Red Chili Salsa (Hot), [Black Beans...16.98
531Chicken Bowl[Fresh Tomato Salsa (Mild), [Rice, Cheese, Sou...10.98
631Side of ChipsNaN1.69
741Steak Burrito[Tomatillo Red Chili Salsa, [Fajita Vegetables...11.75
841Steak Soft Tacos[Tomatillo Green Chili Salsa, [Pinto Beans, Ch...9.25
951Steak Burrito[Fresh Tomato Salsa, [Rice, Black Beans, Pinto...9.25
\n", "
" ], "text/plain": [ " order_id quantity item_name \\\n", "0 1 1 Chips and Fresh Tomato Salsa \n", "1 1 1 Izze \n", "2 1 1 Nantucket Nectar \n", "3 1 1 Chips and Tomatillo-Green Chili Salsa \n", "4 2 2 Chicken Bowl \n", "5 3 1 Chicken Bowl \n", "6 3 1 Side of Chips \n", "7 4 1 Steak Burrito \n", "8 4 1 Steak Soft Tacos \n", "9 5 1 Steak Burrito \n", "\n", " choice_description item_price \n", "0 NaN 2.39 \n", "1 [Clementine] 3.39 \n", "2 [Apple] 3.39 \n", "3 NaN 2.39 \n", "4 [Tomatillo-Red Chili Salsa (Hot), [Black Beans... 16.98 \n", "5 [Fresh Tomato Salsa (Mild), [Rice, Cheese, Sou... 10.98 \n", "6 NaN 1.69 \n", "7 [Tomatillo Red Chili Salsa, [Fajita Vegetables... 11.75 \n", "8 [Tomatillo Green Chili Salsa, [Pinto Beans, Ch... 9.25 \n", "9 [Fresh Tomato Salsa, [Rice, Black Beans, Pinto... 9.25 " ] }, "execution_count": 77, "metadata": {}, "output_type": "execute_result" } ], "source": [ "orders.head(10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Each order has an order_id and consists of one or more rows. To figure out the total price of an order, you sum the item_price for that order_id. For example, here's the total price of order number 1:" ] }, { "cell_type": "code", "execution_count": 78, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "11.56" ] }, "execution_count": 78, "metadata": {}, "output_type": "execute_result" } ], "source": [ "orders[orders.order_id == 1].item_price.sum()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you wanted to calculate the total price of every order, you would `groupby()` order_id and then take the sum of item_price for each group:" ] }, { "cell_type": "code", "execution_count": 79, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "order_id\n", "1 11.56\n", "2 16.98\n", "3 12.67\n", "4 21.00\n", "5 13.70\n", "Name: item_price, dtype: float64" ] }, "execution_count": 79, "metadata": {}, "output_type": "execute_result" } ], "source": [ "orders.groupby('order_id').item_price.sum().head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "However, you're not actually limited to aggregating by a single function such as `sum()`. To aggregate by multiple functions, you use the `agg()` method and pass it a list of functions such as `sum()` and `count()`:" ] }, { "cell_type": "code", "execution_count": 80, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
sumcount
order_id
111.564
216.981
312.672
421.002
513.702
\n", "
" ], "text/plain": [ " sum count\n", "order_id \n", "1 11.56 4\n", "2 16.98 1\n", "3 12.67 2\n", "4 21.00 2\n", "5 13.70 2" ] }, "execution_count": 80, "metadata": {}, "output_type": "execute_result" } ], "source": [ "orders.groupby('order_id').item_price.agg(['sum', 'count']).head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "That gives us the total price of each order as well as the number of items in each order." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 19. Combine the output of an aggregation with a DataFrame" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's take another look at the orders DataFrame:" ] }, { "cell_type": "code", "execution_count": 81, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
order_idquantityitem_namechoice_descriptionitem_price
011Chips and Fresh Tomato SalsaNaN2.39
111Izze[Clementine]3.39
211Nantucket Nectar[Apple]3.39
311Chips and Tomatillo-Green Chili SalsaNaN2.39
422Chicken Bowl[Tomatillo-Red Chili Salsa (Hot), [Black Beans...16.98
531Chicken Bowl[Fresh Tomato Salsa (Mild), [Rice, Cheese, Sou...10.98
631Side of ChipsNaN1.69
741Steak Burrito[Tomatillo Red Chili Salsa, [Fajita Vegetables...11.75
841Steak Soft Tacos[Tomatillo Green Chili Salsa, [Pinto Beans, Ch...9.25
951Steak Burrito[Fresh Tomato Salsa, [Rice, Black Beans, Pinto...9.25
\n", "
" ], "text/plain": [ " order_id quantity item_name \\\n", "0 1 1 Chips and Fresh Tomato Salsa \n", "1 1 1 Izze \n", "2 1 1 Nantucket Nectar \n", "3 1 1 Chips and Tomatillo-Green Chili Salsa \n", "4 2 2 Chicken Bowl \n", "5 3 1 Chicken Bowl \n", "6 3 1 Side of Chips \n", "7 4 1 Steak Burrito \n", "8 4 1 Steak Soft Tacos \n", "9 5 1 Steak Burrito \n", "\n", " choice_description item_price \n", "0 NaN 2.39 \n", "1 [Clementine] 3.39 \n", "2 [Apple] 3.39 \n", "3 NaN 2.39 \n", "4 [Tomatillo-Red Chili Salsa (Hot), [Black Beans... 16.98 \n", "5 [Fresh Tomato Salsa (Mild), [Rice, Cheese, Sou... 10.98 \n", "6 NaN 1.69 \n", "7 [Tomatillo Red Chili Salsa, [Fajita Vegetables... 11.75 \n", "8 [Tomatillo Green Chili Salsa, [Pinto Beans, Ch... 9.25 \n", "9 [Fresh Tomato Salsa, [Rice, Black Beans, Pinto... 9.25 " ] }, "execution_count": 81, "metadata": {}, "output_type": "execute_result" } ], "source": [ "orders.head(10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "What if we wanted to create a new column listing the total price of each order? Recall that we calculated the total price using the `sum()` method:" ] }, { "cell_type": "code", "execution_count": 82, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "order_id\n", "1 11.56\n", "2 16.98\n", "3 12.67\n", "4 21.00\n", "5 13.70\n", "Name: item_price, dtype: float64" ] }, "execution_count": 82, "metadata": {}, "output_type": "execute_result" } ], "source": [ "orders.groupby('order_id').item_price.sum().head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`sum()` is an aggregation function, which means that it returns a reduced version of the input data.\n", "\n", "In other words, the output of the `sum()` function:" ] }, { "cell_type": "code", "execution_count": 83, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "1834" ] }, "execution_count": 83, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(orders.groupby('order_id').item_price.sum())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "...is smaller than the input to the function:" ] }, { "cell_type": "code", "execution_count": 84, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "4622" ] }, "execution_count": 84, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(orders.item_price)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The solution is to use the `transform()` method, which performs the same calculation but returns output data that is the same shape as the input data:" ] }, { "cell_type": "code", "execution_count": 85, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "4622" ] }, "execution_count": 85, "metadata": {}, "output_type": "execute_result" } ], "source": [ "total_price = orders.groupby('order_id').item_price.transform('sum')\n", "len(total_price)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll store the results in a new DataFrame column called total_price:" ] }, { "cell_type": "code", "execution_count": 86, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
order_idquantityitem_namechoice_descriptionitem_pricetotal_price
011Chips and Fresh Tomato SalsaNaN2.3911.56
111Izze[Clementine]3.3911.56
211Nantucket Nectar[Apple]3.3911.56
311Chips and Tomatillo-Green Chili SalsaNaN2.3911.56
422Chicken Bowl[Tomatillo-Red Chili Salsa (Hot), [Black Beans...16.9816.98
531Chicken Bowl[Fresh Tomato Salsa (Mild), [Rice, Cheese, Sou...10.9812.67
631Side of ChipsNaN1.6912.67
741Steak Burrito[Tomatillo Red Chili Salsa, [Fajita Vegetables...11.7521.00
841Steak Soft Tacos[Tomatillo Green Chili Salsa, [Pinto Beans, Ch...9.2521.00
951Steak Burrito[Fresh Tomato Salsa, [Rice, Black Beans, Pinto...9.2513.70
\n", "
" ], "text/plain": [ " order_id quantity item_name \\\n", "0 1 1 Chips and Fresh Tomato Salsa \n", "1 1 1 Izze \n", "2 1 1 Nantucket Nectar \n", "3 1 1 Chips and Tomatillo-Green Chili Salsa \n", "4 2 2 Chicken Bowl \n", "5 3 1 Chicken Bowl \n", "6 3 1 Side of Chips \n", "7 4 1 Steak Burrito \n", "8 4 1 Steak Soft Tacos \n", "9 5 1 Steak Burrito \n", "\n", " choice_description item_price total_price \n", "0 NaN 2.39 11.56 \n", "1 [Clementine] 3.39 11.56 \n", "2 [Apple] 3.39 11.56 \n", "3 NaN 2.39 11.56 \n", "4 [Tomatillo-Red Chili Salsa (Hot), [Black Beans... 16.98 16.98 \n", "5 [Fresh Tomato Salsa (Mild), [Rice, Cheese, Sou... 10.98 12.67 \n", "6 NaN 1.69 12.67 \n", "7 [Tomatillo Red Chili Salsa, [Fajita Vegetables... 11.75 21.00 \n", "8 [Tomatillo Green Chili Salsa, [Pinto Beans, Ch... 9.25 21.00 \n", "9 [Fresh Tomato Salsa, [Rice, Black Beans, Pinto... 9.25 13.70 " ] }, "execution_count": 86, "metadata": {}, "output_type": "execute_result" } ], "source": [ "orders['total_price'] = total_price\n", "orders.head(10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you can see, the total price of each order is now listed on every single line.\n", "\n", "That makes it easy to calculate the percentage of the total order price that each line represents:" ] }, { "cell_type": "code", "execution_count": 87, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
order_idquantityitem_namechoice_descriptionitem_pricetotal_pricepercent_of_total
011Chips and Fresh Tomato SalsaNaN2.3911.560.206747
111Izze[Clementine]3.3911.560.293253
211Nantucket Nectar[Apple]3.3911.560.293253
311Chips and Tomatillo-Green Chili SalsaNaN2.3911.560.206747
422Chicken Bowl[Tomatillo-Red Chili Salsa (Hot), [Black Beans...16.9816.981.000000
531Chicken Bowl[Fresh Tomato Salsa (Mild), [Rice, Cheese, Sou...10.9812.670.866614
631Side of ChipsNaN1.6912.670.133386
741Steak Burrito[Tomatillo Red Chili Salsa, [Fajita Vegetables...11.7521.000.559524
841Steak Soft Tacos[Tomatillo Green Chili Salsa, [Pinto Beans, Ch...9.2521.000.440476
951Steak Burrito[Fresh Tomato Salsa, [Rice, Black Beans, Pinto...9.2513.700.675182
\n", "
" ], "text/plain": [ " order_id quantity item_name \\\n", "0 1 1 Chips and Fresh Tomato Salsa \n", "1 1 1 Izze \n", "2 1 1 Nantucket Nectar \n", "3 1 1 Chips and Tomatillo-Green Chili Salsa \n", "4 2 2 Chicken Bowl \n", "5 3 1 Chicken Bowl \n", "6 3 1 Side of Chips \n", "7 4 1 Steak Burrito \n", "8 4 1 Steak Soft Tacos \n", "9 5 1 Steak Burrito \n", "\n", " choice_description item_price total_price \\\n", "0 NaN 2.39 11.56 \n", "1 [Clementine] 3.39 11.56 \n", "2 [Apple] 3.39 11.56 \n", "3 NaN 2.39 11.56 \n", "4 [Tomatillo-Red Chili Salsa (Hot), [Black Beans... 16.98 16.98 \n", "5 [Fresh Tomato Salsa (Mild), [Rice, Cheese, Sou... 10.98 12.67 \n", "6 NaN 1.69 12.67 \n", "7 [Tomatillo Red Chili Salsa, [Fajita Vegetables... 11.75 21.00 \n", "8 [Tomatillo Green Chili Salsa, [Pinto Beans, Ch... 9.25 21.00 \n", "9 [Fresh Tomato Salsa, [Rice, Black Beans, Pinto... 9.25 13.70 \n", "\n", " percent_of_total \n", "0 0.206747 \n", "1 0.293253 \n", "2 0.293253 \n", "3 0.206747 \n", "4 1.000000 \n", "5 0.866614 \n", "6 0.133386 \n", "7 0.559524 \n", "8 0.440476 \n", "9 0.675182 " ] }, "execution_count": 87, "metadata": {}, "output_type": "execute_result" } ], "source": [ "orders['percent_of_total'] = orders.item_price / orders.total_price\n", "orders.head(10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 20. Select a slice of rows and columns" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's take a look at another dataset:" ] }, { "cell_type": "code", "execution_count": 88, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarked
0103Braund, Mr. Owen Harrismale22.010A/5 211717.2500NaNS
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833C85C
2313Heikkinen, Miss. Lainafemale26.000STON/O2. 31012827.9250NaNS
3411Futrelle, Mrs. Jacques Heath (Lily May Peel)female35.01011380353.1000C123S
4503Allen, Mr. William Henrymale35.0003734508.0500NaNS
\n", "
" ], "text/plain": [ " PassengerId Survived Pclass \\\n", "0 1 0 3 \n", "1 2 1 1 \n", "2 3 1 3 \n", "3 4 1 1 \n", "4 5 0 3 \n", "\n", " Name Sex Age SibSp \\\n", "0 Braund, Mr. Owen Harris male 22.0 1 \n", "1 Cumings, Mrs. John Bradley (Florence Briggs Th... female 38.0 1 \n", "2 Heikkinen, Miss. Laina female 26.0 0 \n", "3 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.0 1 \n", "4 Allen, Mr. William Henry male 35.0 0 \n", "\n", " Parch Ticket Fare Cabin Embarked \n", "0 0 A/5 21171 7.2500 NaN S \n", "1 0 PC 17599 71.2833 C85 C \n", "2 0 STON/O2. 3101282 7.9250 NaN S \n", "3 0 113803 53.1000 C123 S \n", "4 0 373450 8.0500 NaN S " ] }, "execution_count": 88, "metadata": {}, "output_type": "execute_result" } ], "source": [ "titanic.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is the famous Titanic dataset, which shows information about passengers on the Titanic and whether or not they survived.\n", "\n", "If you wanted a numerical summary of the dataset, you would use the `describe()` method:" ] }, { "cell_type": "code", "execution_count": 89, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PassengerIdSurvivedPclassAgeSibSpParchFare
count891.000000891.000000891.000000714.000000891.000000891.000000891.000000
mean446.0000000.3838382.30864229.6991180.5230080.38159432.204208
std257.3538420.4865920.83607114.5264971.1027430.80605749.693429
min1.0000000.0000001.0000000.4200000.0000000.0000000.000000
25%223.5000000.0000002.00000020.1250000.0000000.0000007.910400
50%446.0000000.0000003.00000028.0000000.0000000.00000014.454200
75%668.5000001.0000003.00000038.0000001.0000000.00000031.000000
max891.0000001.0000003.00000080.0000008.0000006.000000512.329200
\n", "
" ], "text/plain": [ " PassengerId Survived Pclass Age SibSp \\\n", "count 891.000000 891.000000 891.000000 714.000000 891.000000 \n", "mean 446.000000 0.383838 2.308642 29.699118 0.523008 \n", "std 257.353842 0.486592 0.836071 14.526497 1.102743 \n", "min 1.000000 0.000000 1.000000 0.420000 0.000000 \n", "25% 223.500000 0.000000 2.000000 20.125000 0.000000 \n", "50% 446.000000 0.000000 3.000000 28.000000 0.000000 \n", "75% 668.500000 1.000000 3.000000 38.000000 1.000000 \n", "max 891.000000 1.000000 3.000000 80.000000 8.000000 \n", "\n", " Parch Fare \n", "count 891.000000 891.000000 \n", "mean 0.381594 32.204208 \n", "std 0.806057 49.693429 \n", "min 0.000000 0.000000 \n", "25% 0.000000 7.910400 \n", "50% 0.000000 14.454200 \n", "75% 0.000000 31.000000 \n", "max 6.000000 512.329200 " ] }, "execution_count": 89, "metadata": {}, "output_type": "execute_result" } ], "source": [ "titanic.describe()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "However, the resulting DataFrame might be displaying more information than you need.\n", "\n", "If you wanted to filter it to only show the \"five-number summary\", you can use the `loc` accessor and pass it a slice of the \"min\" through the \"max\" row labels:" ] }, { "cell_type": "code", "execution_count": 90, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PassengerIdSurvivedPclassAgeSibSpParchFare
min1.00.01.00.4200.00.00.0000
25%223.50.02.020.1250.00.07.9104
50%446.00.03.028.0000.00.014.4542
75%668.51.03.038.0001.00.031.0000
max891.01.03.080.0008.06.0512.3292
\n", "
" ], "text/plain": [ " PassengerId Survived Pclass Age SibSp Parch Fare\n", "min 1.0 0.0 1.0 0.420 0.0 0.0 0.0000\n", "25% 223.5 0.0 2.0 20.125 0.0 0.0 7.9104\n", "50% 446.0 0.0 3.0 28.000 0.0 0.0 14.4542\n", "75% 668.5 1.0 3.0 38.000 1.0 0.0 31.0000\n", "max 891.0 1.0 3.0 80.000 8.0 6.0 512.3292" ] }, "execution_count": 90, "metadata": {}, "output_type": "execute_result" } ], "source": [ "titanic.describe().loc['min':'max']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And if you're not interested in all of the columns, you can also pass it a slice of column labels:" ] }, { "cell_type": "code", "execution_count": 91, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PclassAgeSibSpParch
min1.00.4200.00.0
25%2.020.1250.00.0
50%3.028.0000.00.0
75%3.038.0001.00.0
max3.080.0008.06.0
\n", "
" ], "text/plain": [ " Pclass Age SibSp Parch\n", "min 1.0 0.420 0.0 0.0\n", "25% 2.0 20.125 0.0 0.0\n", "50% 3.0 28.000 0.0 0.0\n", "75% 3.0 38.000 1.0 0.0\n", "max 3.0 80.000 8.0 6.0" ] }, "execution_count": 91, "metadata": {}, "output_type": "execute_result" } ], "source": [ "titanic.describe().loc['min':'max', 'Pclass':'Parch']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 21. Reshape a MultiIndexed Series" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The Titanic dataset has a \"Survived\" column made up of ones and zeros, so you can calculate the overall survival rate by taking a mean of that column:" ] }, { "cell_type": "code", "execution_count": 92, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.3838383838383838" ] }, "execution_count": 92, "metadata": {}, "output_type": "execute_result" } ], "source": [ "titanic.Survived.mean()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you wanted to calculate the survival rate by a single category such as \"Sex\", you would use a `groupby()`:" ] }, { "cell_type": "code", "execution_count": 93, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Sex\n", "female 0.742038\n", "male 0.188908\n", "Name: Survived, dtype: float64" ] }, "execution_count": 93, "metadata": {}, "output_type": "execute_result" } ], "source": [ "titanic.groupby('Sex').Survived.mean()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And if you wanted to calculate the survival rate across two different categories at once, you would `groupby()` both of those categories:" ] }, { "cell_type": "code", "execution_count": 94, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Sex Pclass\n", "female 1 0.968085\n", " 2 0.921053\n", " 3 0.500000\n", "male 1 0.368852\n", " 2 0.157407\n", " 3 0.135447\n", "Name: Survived, dtype: float64" ] }, "execution_count": 94, "metadata": {}, "output_type": "execute_result" } ], "source": [ "titanic.groupby(['Sex', 'Pclass']).Survived.mean()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This shows the survival rate for every combination of Sex and Passenger Class. It's stored as a MultiIndexed Series, meaning that it has multiple index levels to the left of the actual data.\n", "\n", "It can be hard to read and interact with data in this format, so it's often more convenient to reshape a MultiIndexed Series into a DataFrame by using the `unstack()` method:" ] }, { "cell_type": "code", "execution_count": 95, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Pclass123
Sex
female0.9680850.9210530.500000
male0.3688520.1574070.135447
\n", "
" ], "text/plain": [ "Pclass 1 2 3\n", "Sex \n", "female 0.968085 0.921053 0.500000\n", "male 0.368852 0.157407 0.135447" ] }, "execution_count": 95, "metadata": {}, "output_type": "execute_result" } ], "source": [ "titanic.groupby(['Sex', 'Pclass']).Survived.mean().unstack()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This DataFrame contains the same exact data as the MultiIndexed Series, except that now you can interact with it using familiar DataFrame methods." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 22. Create a pivot table" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you often create DataFrames like the one above, you might find it more convenient to use the `pivot_table()` method instead:" ] }, { "cell_type": "code", "execution_count": 96, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Pclass123
Sex
female0.9680850.9210530.500000
male0.3688520.1574070.135447
\n", "
" ], "text/plain": [ "Pclass 1 2 3\n", "Sex \n", "female 0.968085 0.921053 0.500000\n", "male 0.368852 0.157407 0.135447" ] }, "execution_count": 96, "metadata": {}, "output_type": "execute_result" } ], "source": [ "titanic.pivot_table(index='Sex', columns='Pclass', values='Survived', aggfunc='mean')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With a pivot table, you directly specify the index, the columns, the values, and the aggregation function.\n", "\n", "An added benefit of a pivot table is that you can easily add row and column totals by setting `margins=True`:" ] }, { "cell_type": "code", "execution_count": 97, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Pclass123All
Sex
female0.9680850.9210530.5000000.742038
male0.3688520.1574070.1354470.188908
All0.6296300.4728260.2423630.383838
\n", "
" ], "text/plain": [ "Pclass 1 2 3 All\n", "Sex \n", "female 0.968085 0.921053 0.500000 0.742038\n", "male 0.368852 0.157407 0.135447 0.188908\n", "All 0.629630 0.472826 0.242363 0.383838" ] }, "execution_count": 97, "metadata": {}, "output_type": "execute_result" } ], "source": [ "titanic.pivot_table(index='Sex', columns='Pclass', values='Survived', aggfunc='mean',\n", " margins=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This shows the overall survival rate as well as the survival rate by Sex and Passenger Class.\n", "\n", "Finally, you can create a cross-tabulation just by changing the aggregation function from \"mean\" to \"count\":" ] }, { "cell_type": "code", "execution_count": 98, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Pclass123All
Sex
female9476144314
male122108347577
All216184491891
\n", "
" ], "text/plain": [ "Pclass 1 2 3 All\n", "Sex \n", "female 94 76 144 314\n", "male 122 108 347 577\n", "All 216 184 491 891" ] }, "execution_count": 98, "metadata": {}, "output_type": "execute_result" } ], "source": [ "titanic.pivot_table(index='Sex', columns='Pclass', values='Survived', aggfunc='count',\n", " margins=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This shows the number of records that appear in each combination of categories." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 23. Convert continuous data into categorical data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's take a look at the Age column from the Titanic dataset:" ] }, { "cell_type": "code", "execution_count": 99, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 22.0\n", "1 38.0\n", "2 26.0\n", "3 35.0\n", "4 35.0\n", "5 NaN\n", "6 54.0\n", "7 2.0\n", "8 27.0\n", "9 14.0\n", "Name: Age, dtype: float64" ] }, "execution_count": 99, "metadata": {}, "output_type": "execute_result" } ], "source": [ "titanic.Age.head(10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It's currently continuous data, but what if you wanted to convert it into categorical data?\n", "\n", "One solution would be to label the age ranges, such as \"child\", \"young adult\", and \"adult\". The best way to do this is by using the `cut()` function:" ] }, { "cell_type": "code", "execution_count": 100, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 young adult\n", "1 adult\n", "2 adult\n", "3 adult\n", "4 adult\n", "5 NaN\n", "6 adult\n", "7 child\n", "8 adult\n", "9 child\n", "Name: Age, dtype: category\n", "Categories (3, object): [child < young adult < adult]" ] }, "execution_count": 100, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.cut(titanic.Age, bins=[0, 18, 25, 99], labels=['child', 'young adult', 'adult']).head(10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This assigned each value to a bin with a label. Ages 0 to 18 were assigned the label \"child\", ages 18 to 25 were assigned the label \"young adult\", and ages 25 to 99 were assigned the label \"adult\".\n", "\n", "Notice that the data type is now \"category\", and the categories are automatically ordered." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 24. Change display options" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's take another look at the Titanic dataset:" ] }, { "cell_type": "code", "execution_count": 101, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarked
0103Braund, Mr. Owen Harrismale22.010A/5 211717.2500NaNS
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833C85C
2313Heikkinen, Miss. Lainafemale26.000STON/O2. 31012827.9250NaNS
3411Futrelle, Mrs. Jacques Heath (Lily May Peel)female35.01011380353.1000C123S
4503Allen, Mr. William Henrymale35.0003734508.0500NaNS
\n", "
" ], "text/plain": [ " PassengerId Survived Pclass \\\n", "0 1 0 3 \n", "1 2 1 1 \n", "2 3 1 3 \n", "3 4 1 1 \n", "4 5 0 3 \n", "\n", " Name Sex Age SibSp \\\n", "0 Braund, Mr. Owen Harris male 22.0 1 \n", "1 Cumings, Mrs. John Bradley (Florence Briggs Th... female 38.0 1 \n", "2 Heikkinen, Miss. Laina female 26.0 0 \n", "3 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.0 1 \n", "4 Allen, Mr. William Henry male 35.0 0 \n", "\n", " Parch Ticket Fare Cabin Embarked \n", "0 0 A/5 21171 7.2500 NaN S \n", "1 0 PC 17599 71.2833 C85 C \n", "2 0 STON/O2. 3101282 7.9250 NaN S \n", "3 0 113803 53.1000 C123 S \n", "4 0 373450 8.0500 NaN S " ] }, "execution_count": 101, "metadata": {}, "output_type": "execute_result" } ], "source": [ "titanic.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Notice that the Age column has 1 decimal place and the Fare column has 4 decimal places. What if you wanted to standardize the display to use 2 decimal places?\n", "\n", "You can use the `set_option()` function:" ] }, { "cell_type": "code", "execution_count": 102, "metadata": {}, "outputs": [], "source": [ "pd.set_option('display.float_format', '{:.2f}'.format)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The first argument is the name of the option, and the second argument is a Python format string." ] }, { "cell_type": "code", "execution_count": 103, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarked
0103Braund, Mr. Owen Harrismale22.0010A/5 211717.25NaNS
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.0010PC 1759971.28C85C
2313Heikkinen, Miss. Lainafemale26.0000STON/O2. 31012827.92NaNS
3411Futrelle, Mrs. Jacques Heath (Lily May Peel)female35.001011380353.10C123S
4503Allen, Mr. William Henrymale35.00003734508.05NaNS
\n", "
" ], "text/plain": [ " PassengerId Survived Pclass \\\n", "0 1 0 3 \n", "1 2 1 1 \n", "2 3 1 3 \n", "3 4 1 1 \n", "4 5 0 3 \n", "\n", " Name Sex Age SibSp \\\n", "0 Braund, Mr. Owen Harris male 22.00 1 \n", "1 Cumings, Mrs. John Bradley (Florence Briggs Th... female 38.00 1 \n", "2 Heikkinen, Miss. Laina female 26.00 0 \n", "3 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.00 1 \n", "4 Allen, Mr. William Henry male 35.00 0 \n", "\n", " Parch Ticket Fare Cabin Embarked \n", "0 0 A/5 21171 7.25 NaN S \n", "1 0 PC 17599 71.28 C85 C \n", "2 0 STON/O2. 3101282 7.92 NaN S \n", "3 0 113803 53.10 C123 S \n", "4 0 373450 8.05 NaN S " ] }, "execution_count": 103, "metadata": {}, "output_type": "execute_result" } ], "source": [ "titanic.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can see that Age and Fare are now using 2 decimal places. Note that this did not change the underlying data, only the display of the data.\n", "\n", "You can also reset any option back to its default:" ] }, { "cell_type": "code", "execution_count": 104, "metadata": {}, "outputs": [], "source": [ "pd.reset_option('display.float_format')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are many more options you can specify is a similar way." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 25. Style a DataFrame" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The previous trick is useful if you want to change the display of your entire notebook. However, a more flexible and powerful approach is to define the style of a particular DataFrame.\n", "\n", "Let's return to the stocks DataFrame:" ] }, { "cell_type": "code", "execution_count": 105, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
DateCloseVolumeSymbol
02016-10-0331.5014070500CSCO
12016-10-03112.5221701800AAPL
22016-10-0357.4219189500MSFT
32016-10-04113.0029736800AAPL
42016-10-0457.2420085900MSFT
52016-10-0431.3518460400CSCO
62016-10-0557.6416726400MSFT
72016-10-0531.5911808600CSCO
82016-10-05113.0521453100AAPL
\n", "
" ], "text/plain": [ " Date Close Volume Symbol\n", "0 2016-10-03 31.50 14070500 CSCO\n", "1 2016-10-03 112.52 21701800 AAPL\n", "2 2016-10-03 57.42 19189500 MSFT\n", "3 2016-10-04 113.00 29736800 AAPL\n", "4 2016-10-04 57.24 20085900 MSFT\n", "5 2016-10-04 31.35 18460400 CSCO\n", "6 2016-10-05 57.64 16726400 MSFT\n", "7 2016-10-05 31.59 11808600 CSCO\n", "8 2016-10-05 113.05 21453100 AAPL" ] }, "execution_count": 105, "metadata": {}, "output_type": "execute_result" } ], "source": [ "stocks" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can create a dictionary of format strings that specifies how each column should be formatted:" ] }, { "cell_type": "code", "execution_count": 106, "metadata": {}, "outputs": [], "source": [ "format_dict = {'Date':'{:%m/%d/%y}', 'Close':'${:.2f}', 'Volume':'{:,}'}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And then we can pass it to the DataFrame's `style.format()` method:" ] }, { "cell_type": "code", "execution_count": 107, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Date Close Volume Symbol
010/03/16$31.5014,070,500CSCO
110/03/16$112.5221,701,800AAPL
210/03/16$57.4219,189,500MSFT
310/04/16$113.0029,736,800AAPL
410/04/16$57.2420,085,900MSFT
510/04/16$31.3518,460,400CSCO
610/05/16$57.6416,726,400MSFT
710/05/16$31.5911,808,600CSCO
810/05/16$113.0521,453,100AAPL
" ], "text/plain": [ "" ] }, "execution_count": 107, "metadata": {}, "output_type": "execute_result" } ], "source": [ "stocks.style.format(format_dict)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Notice that the Date is now in month-day-year format, the closing price has a dollar sign, and the Volume has commas.\n", "\n", "We can apply more styling by chaining additional methods:" ] }, { "cell_type": "code", "execution_count": 108, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Date Close Volume Symbol
10/03/16$31.5014,070,500CSCO
10/03/16$112.5221,701,800AAPL
10/03/16$57.4219,189,500MSFT
10/04/16$113.0029,736,800AAPL
10/04/16$57.2420,085,900MSFT
10/04/16$31.3518,460,400CSCO
10/05/16$57.6416,726,400MSFT
10/05/16$31.5911,808,600CSCO
10/05/16$113.0521,453,100AAPL
" ], "text/plain": [ "" ] }, "execution_count": 108, "metadata": {}, "output_type": "execute_result" } ], "source": [ "(stocks.style.format(format_dict)\n", " .hide_index()\n", " .highlight_min('Close', color='red')\n", " .highlight_max('Close', color='lightgreen')\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We've now hidden the index, highlighted the minimum Close value in red, and highlighted the maximum Close value in green.\n", "\n", "Here's another example of DataFrame styling:" ] }, { "cell_type": "code", "execution_count": 109, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Date Close Volume Symbol
10/03/16$31.5014,070,500CSCO
10/03/16$112.5221,701,800AAPL
10/03/16$57.4219,189,500MSFT
10/04/16$113.0029,736,800AAPL
10/04/16$57.2420,085,900MSFT
10/04/16$31.3518,460,400CSCO
10/05/16$57.6416,726,400MSFT
10/05/16$31.5911,808,600CSCO
10/05/16$113.0521,453,100AAPL
" ], "text/plain": [ "" ] }, "execution_count": 109, "metadata": {}, "output_type": "execute_result" } ], "source": [ "(stocks.style.format(format_dict)\n", " .hide_index()\n", " .background_gradient(subset='Volume', cmap='Blues')\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The Volume column now has a background gradient to help you easily identify high and low values.\n", "\n", "And here's one final example:" ] }, { "cell_type": "code", "execution_count": 110, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Stock Prices from October 2016
Date Close Volume Symbol
10/03/16$31.5014,070,500CSCO
10/03/16$112.5221,701,800AAPL
10/03/16$57.4219,189,500MSFT
10/04/16$113.0029,736,800AAPL
10/04/16$57.2420,085,900MSFT
10/04/16$31.3518,460,400CSCO
10/05/16$57.6416,726,400MSFT
10/05/16$31.5911,808,600CSCO
10/05/16$113.0521,453,100AAPL
" ], "text/plain": [ "" ] }, "execution_count": 110, "metadata": {}, "output_type": "execute_result" } ], "source": [ "(stocks.style.format(format_dict)\n", " .hide_index()\n", " .bar('Volume', color='lightblue', align='zero')\n", " .set_caption('Stock Prices from October 2016')\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There's now a bar chart within the Volume column and a caption above the DataFrame.\n", "\n", "Note that there are many more options for how you can style your DataFrame." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Bonus: Profile a DataFrame" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's say that you've got a new dataset, and you want to quickly explore it without too much work. There's a separate package called [pandas-profiling](https://github.com/pandas-profiling/pandas-profiling) that is designed for this purpose.\n", "\n", "First you have to install it using conda or pip. Once that's done, you import `pandas_profiling`:" ] }, { "cell_type": "code", "execution_count": 111, "metadata": {}, "outputs": [], "source": [ "import pandas_profiling" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then, simply run the `ProfileReport()` function and pass it any DataFrame. It returns an interactive HTML report:\n", "\n", "- The first section is an overview of the dataset and a list of possible issues with the data.\n", "- The next section gives a summary of each column. You can click \"toggle details\" for even more information.\n", "- The third section shows a heatmap of the correlation between columns.\n", "- And the fourth section shows the head of the dataset." ] }, { "cell_type": "code", "execution_count": 112, "metadata": { "scrolled": false }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "
\n", "
\n", "

Overview

\n", "
\n", "
\n", "
\n", "

Dataset info

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Number of variables12
Number of observations891
Total Missing (%)8.1%
Total size in memory83.6 KiB
Average record size in memory96.1 B
\n", "
\n", "
\n", "

Variables types

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Numeric6
Categorical4
Boolean1
Date0
Text (Unique)1
Rejected0
Unsupported0
\n", "
\n", "
\n", " \n", "

Warnings

\n", "
  • Age has 177 / 19.9% missing values Missing
  • Cabin has 687 / 77.1% missing values Missing
  • Cabin has a high cardinality: 148 distinct values Warning
  • Fare has 15 / 1.7% zeros Zeros
  • Parch has 678 / 76.1% zeros Zeros
  • SibSp has 608 / 68.2% zeros Zeros
  • Ticket has a high cardinality: 681 distinct values Warning
\n", "
\n", "
\n", "
\n", "

Variables

\n", "
\n", "
\n", "
\n", "

Age
\n", " Numeric\n", "

\n", "
\n", "
\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Distinct count89
Unique (%)10.0%
Missing (%)19.9%
Missing (n)177
Infinite (%)0.0%
Infinite (n)0
\n", "\n", "
\n", "
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Mean29.699
Minimum0.42
Maximum80
Zeros (%)0.0%
\n", "
\n", "
\n", "
\n", "
\n", " \n", "\n", "
\n", "
\n", " \n", " Toggle details\n", " \n", "
\n", "
\n", " \n", "\n", "
\n", "
\n", "
\n", "

Quantile statistics

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Minimum0.42
5-th percentile4
Q120.125
Median28
Q338
95-th percentile56
Maximum80
Range79.58
Interquartile range17.875
\n", "
\n", "
\n", "

Descriptive statistics

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Standard deviation14.526
Coef of variation0.48912
Kurtosis0.17827
Mean29.699
MAD11.323
Skewness0.38911
Sum21205
Variance211.02
Memory size7.0 KiB
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", "
ValueCountFrequency (%) 
24.0303.4%\n", "
 
\n", "
22.0273.0%\n", "
 
\n", "
18.0262.9%\n", "
 
\n", "
28.0252.8%\n", "
 
\n", "
19.0252.8%\n", "
 
\n", "
30.0252.8%\n", "
 
\n", "
21.0242.7%\n", "
 
\n", "
25.0232.6%\n", "
 
\n", "
36.0222.5%\n", "
 
\n", "
29.0202.2%\n", "
 
\n", "
Other values (78)46752.4%\n", "
 
\n", "
(Missing)17719.9%\n", "
 
\n", "
\n", "
\n", "
\n", "

Minimum 5 values

\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", "
ValueCountFrequency (%) 
0.4210.1%\n", "
 
\n", "
0.6710.1%\n", "
 
\n", "
0.7520.2%\n", "
 
\n", "
0.8320.2%\n", "
 
\n", "
0.9210.1%\n", "
 
\n", "
\n", "

Maximum 5 values

\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", "
ValueCountFrequency (%) 
70.020.2%\n", "
 
\n", "
70.510.1%\n", "
 
\n", "
71.020.2%\n", "
 
\n", "
74.010.1%\n", "
 
\n", "
80.010.1%\n", "
 
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "

Cabin
\n", " Categorical\n", "

\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Distinct count148
Unique (%)16.6%
Missing (%)77.1%
Missing (n)687
\n", "
\n", "
\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", "\n", "
G6\n", "
\n", "  \n", "
\n", " 4\n", "
C23 C25 C27\n", "
\n", "  \n", "
\n", " 4\n", "
B96 B98\n", "
\n", "  \n", "
\n", " 4\n", "
Other values (144)\n", "
\n", " 192\n", "
\n", " \n", "
(Missing)\n", "
\n", " 687\n", "
\n", " \n", "
\n", "
\n", "
\n", " \n", " Toggle details\n", " \n", "
\n", "
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", "
ValueCountFrequency (%) 
G640.4%\n", "
 
\n", "
C23 C25 C2740.4%\n", "
 
\n", "
B96 B9840.4%\n", "
 
\n", "
D30.3%\n", "
 
\n", "
F230.3%\n", "
 
\n", "
F3330.3%\n", "
 
\n", "
E10130.3%\n", "
 
\n", "
C22 C2630.3%\n", "
 
\n", "
C12420.2%\n", "
 
\n", "
D3520.2%\n", "
 
\n", "
Other values (137)17319.4%\n", "
 
\n", "
(Missing)68777.1%\n", "
 
\n", "
\n", "
\n", "
\n", "
\n", "

Embarked
\n", " Categorical\n", "

\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Distinct count4
Unique (%)0.4%
Missing (%)0.2%
Missing (n)2
\n", "
\n", "
\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", "\n", "
S\n", "
\n", " 644\n", "
\n", " \n", "
C\n", "
\n", " 168\n", "
\n", " \n", "
Q\n", "
\n", "  \n", "
\n", " 77\n", "
(Missing)\n", "
\n", "  \n", "
\n", " 2\n", "
\n", "
\n", "
\n", " \n", " Toggle details\n", " \n", "
\n", "
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", "
ValueCountFrequency (%) 
S64472.3%\n", "
 
\n", "
C16818.9%\n", "
 
\n", "
Q778.6%\n", "
 
\n", "
(Missing)20.2%\n", "
 
\n", "
\n", "
\n", "
\n", "
\n", "

Fare
\n", " Numeric\n", "

\n", "
\n", "
\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Distinct count248
Unique (%)27.8%
Missing (%)0.0%
Missing (n)0
Infinite (%)0.0%
Infinite (n)0
\n", "\n", "
\n", "
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Mean32.204
Minimum0
Maximum512.33
Zeros (%)1.7%
\n", "
\n", "
\n", "
\n", "
\n", " \n", "\n", "
\n", "
\n", " \n", " Toggle details\n", " \n", "
\n", "
\n", " \n", "\n", "
\n", "
\n", "
\n", "

Quantile statistics

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Minimum0
5-th percentile7.225
Q17.9104
Median14.454
Q331
95-th percentile112.08
Maximum512.33
Range512.33
Interquartile range23.09
\n", "
\n", "
\n", "

Descriptive statistics

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Standard deviation49.693
Coef of variation1.5431
Kurtosis33.398
Mean32.204
MAD28.164
Skewness4.7873
Sum28694
Variance2469.4
Memory size7.0 KiB
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", "
ValueCountFrequency (%) 
8.05434.8%\n", "
 
\n", "
13.0424.7%\n", "
 
\n", "
7.8958384.3%\n", "
 
\n", "
7.75343.8%\n", "
 
\n", "
26.0313.5%\n", "
 
\n", "
10.5242.7%\n", "
 
\n", "
7.925182.0%\n", "
 
\n", "
7.775161.8%\n", "
 
\n", "
26.55151.7%\n", "
 
\n", "
0.0151.7%\n", "
 
\n", "
Other values (238)61569.0%\n", "
 
\n", "
\n", "
\n", "
\n", "

Minimum 5 values

\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", "
ValueCountFrequency (%) 
0.0151.7%\n", "
 
\n", "
4.012510.1%\n", "
 
\n", "
5.010.1%\n", "
 
\n", "
6.237510.1%\n", "
 
\n", "
6.437510.1%\n", "
 
\n", "
\n", "

Maximum 5 values

\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", "
ValueCountFrequency (%) 
227.52540.4%\n", "
 
\n", "
247.520820.2%\n", "
 
\n", "
262.37520.2%\n", "
 
\n", "
263.040.4%\n", "
 
\n", "
512.329230.3%\n", "
 
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "

Name
\n", " Categorical, Unique\n", "

\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
First 3 values
Mockler, Miss. Helen Mary \"Ellie\"
Baclini, Miss. Eugenie
Mayne, Mlle. Berthe Antonine (\"Mrs de Villiers\")
\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Last 3 values
Hoyt, Mrs. Frederick Maxfield (Jane Anne Forby)
Gustafsson, Mr. Karl Gideon
Dowdell, Miss. Elizabeth
\n", "
\n", " \n", " Toggle details\n", " \n", "
\n", "
\n", "

First 10 values

\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", "
ValueCountFrequency (%) 
Abbing, Mr. Anthony10.1%\n", "
 
\n", "
Abbott, Mr. Rossmore Edward10.1%\n", "
 
\n", "
Abbott, Mrs. Stanton (Rosa Hunt)10.1%\n", "
 
\n", "
Abelson, Mr. Samuel10.1%\n", "
 
\n", "
Abelson, Mrs. Samuel (Hannah Wizosky)10.1%\n", "
 
\n", "
\n", "

Last 10 values

\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", "
ValueCountFrequency (%) 
de Mulder, Mr. Theodore10.1%\n", "
 
\n", "
de Pelsmaeker, Mr. Alfons10.1%\n", "
 
\n", "
del Carlo, Mr. Sebastiano10.1%\n", "
 
\n", "
van Billiard, Mr. Austin Blyler10.1%\n", "
 
\n", "
van Melkebeke, Mr. Philemon10.1%\n", "
 
\n", "
\n", "
\n", "
\n", "
\n", "

Parch
\n", " Numeric\n", "

\n", "
\n", "
\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Distinct count7
Unique (%)0.8%
Missing (%)0.0%
Missing (n)0
Infinite (%)0.0%
Infinite (n)0
\n", "\n", "
\n", "
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Mean0.38159
Minimum0
Maximum6
Zeros (%)76.1%
\n", "
\n", "
\n", "
\n", "
\n", " \n", "\n", "
\n", "
\n", " \n", " Toggle details\n", " \n", "
\n", "
\n", " \n", "\n", "
\n", "
\n", "
\n", "

Quantile statistics

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Minimum0
5-th percentile0
Q10
Median0
Q30
95-th percentile2
Maximum6
Range6
Interquartile range0
\n", "
\n", "
\n", "

Descriptive statistics

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Standard deviation0.80606
Coef of variation2.1123
Kurtosis9.7781
Mean0.38159
MAD0.58074
Skewness2.7491
Sum340
Variance0.64973
Memory size7.0 KiB
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", "
ValueCountFrequency (%) 
067876.1%\n", "
 
\n", "
111813.2%\n", "
 
\n", "
2809.0%\n", "
 
\n", "
550.6%\n", "
 
\n", "
350.6%\n", "
 
\n", "
440.4%\n", "
 
\n", "
610.1%\n", "
 
\n", "
\n", "
\n", "
\n", "

Minimum 5 values

\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", "
ValueCountFrequency (%) 
067876.1%\n", "
 
\n", "
111813.2%\n", "
 
\n", "
2809.0%\n", "
 
\n", "
350.6%\n", "
 
\n", "
440.4%\n", "
 
\n", "
\n", "

Maximum 5 values

\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", "
ValueCountFrequency (%) 
2809.0%\n", "
 
\n", "
350.6%\n", "
 
\n", "
440.4%\n", "
 
\n", "
550.6%\n", "
 
\n", "
610.1%\n", "
 
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "

PassengerId
\n", " Numeric\n", "

\n", "
\n", "
\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Distinct count891
Unique (%)100.0%
Missing (%)0.0%
Missing (n)0
Infinite (%)0.0%
Infinite (n)0
\n", "\n", "
\n", "
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Mean446
Minimum1
Maximum891
Zeros (%)0.0%
\n", "
\n", "
\n", "
\n", "
\n", " \n", "\n", "
\n", "
\n", " \n", " Toggle details\n", " \n", "
\n", "
\n", " \n", "\n", "
\n", "
\n", "
\n", "

Quantile statistics

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Minimum1
5-th percentile45.5
Q1223.5
Median446
Q3668.5
95-th percentile846.5
Maximum891
Range890
Interquartile range445
\n", "
\n", "
\n", "

Descriptive statistics

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Standard deviation257.35
Coef of variation0.57703
Kurtosis-1.2
Mean446
MAD222.75
Skewness0
Sum397386
Variance66231
Memory size7.0 KiB
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", "
ValueCountFrequency (%) 
89110.1%\n", "
 
\n", "
29310.1%\n", "
 
\n", "
30410.1%\n", "
 
\n", "
30310.1%\n", "
 
\n", "
30210.1%\n", "
 
\n", "
30110.1%\n", "
 
\n", "
30010.1%\n", "
 
\n", "
29910.1%\n", "
 
\n", "
29810.1%\n", "
 
\n", "
29710.1%\n", "
 
\n", "
Other values (881)88198.9%\n", "
 
\n", "
\n", "
\n", "
\n", "

Minimum 5 values

\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", "
ValueCountFrequency (%) 
110.1%\n", "
 
\n", "
210.1%\n", "
 
\n", "
310.1%\n", "
 
\n", "
410.1%\n", "
 
\n", "
510.1%\n", "
 
\n", "
\n", "

Maximum 5 values

\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", "
ValueCountFrequency (%) 
88710.1%\n", "
 
\n", "
88810.1%\n", "
 
\n", "
88910.1%\n", "
 
\n", "
89010.1%\n", "
 
\n", "
89110.1%\n", "
 
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "

Pclass
\n", " Numeric\n", "

\n", "
\n", "
\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Distinct count3
Unique (%)0.3%
Missing (%)0.0%
Missing (n)0
Infinite (%)0.0%
Infinite (n)0
\n", "\n", "
\n", "
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Mean2.3086
Minimum1
Maximum3
Zeros (%)0.0%
\n", "
\n", "
\n", "
\n", "
\n", " \n", "\n", "
\n", "
\n", " \n", " Toggle details\n", " \n", "
\n", "
\n", " \n", "\n", "
\n", "
\n", "
\n", "

Quantile statistics

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Minimum1
5-th percentile1
Q12
Median3
Q33
95-th percentile3
Maximum3
Range2
Interquartile range1
\n", "
\n", "
\n", "

Descriptive statistics

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Standard deviation0.83607
Coef of variation0.36215
Kurtosis-1.28
Mean2.3086
MAD0.76197
Skewness-0.63055
Sum2057
Variance0.69902
Memory size7.0 KiB
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", "
ValueCountFrequency (%) 
349155.1%\n", "
 
\n", "
121624.2%\n", "
 
\n", "
218420.7%\n", "
 
\n", "
\n", "
\n", "
\n", "

Minimum 5 values

\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", "
ValueCountFrequency (%) 
121624.2%\n", "
 
\n", "
218420.7%\n", "
 
\n", "
349155.1%\n", "
 
\n", "
\n", "

Maximum 5 values

\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", "
ValueCountFrequency (%) 
121624.2%\n", "
 
\n", "
218420.7%\n", "
 
\n", "
349155.1%\n", "
 
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "

Sex
\n", " Categorical\n", "

\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Distinct count2
Unique (%)0.2%
Missing (%)0.0%
Missing (n)0
\n", "
\n", "
\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", "
male\n", "
\n", " 577\n", "
\n", " \n", "
female\n", "
\n", " 314\n", "
\n", " \n", "
\n", "
\n", "
\n", " \n", " Toggle details\n", " \n", "
\n", "
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", "
ValueCountFrequency (%) 
male57764.8%\n", "
 
\n", "
female31435.2%\n", "
 
\n", "
\n", "
\n", "
\n", "
\n", "

SibSp
\n", " Numeric\n", "

\n", "
\n", "
\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Distinct count7
Unique (%)0.8%
Missing (%)0.0%
Missing (n)0
Infinite (%)0.0%
Infinite (n)0
\n", "\n", "
\n", "
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Mean0.52301
Minimum0
Maximum8
Zeros (%)68.2%
\n", "
\n", "
\n", "
\n", "
\n", " \n", "\n", "
\n", "
\n", " \n", " Toggle details\n", " \n", "
\n", "
\n", " \n", "\n", "
\n", "
\n", "
\n", "

Quantile statistics

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Minimum0
5-th percentile0
Q10
Median0
Q31
95-th percentile3
Maximum8
Range8
Interquartile range1
\n", "
\n", "
\n", "

Descriptive statistics

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Standard deviation1.1027
Coef of variation2.1085
Kurtosis17.88
Mean0.52301
MAD0.71378
Skewness3.6954
Sum466
Variance1.216
Memory size7.0 KiB
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", "
ValueCountFrequency (%) 
060868.2%\n", "
 
\n", "
120923.5%\n", "
 
\n", "
2283.1%\n", "
 
\n", "
4182.0%\n", "
 
\n", "
3161.8%\n", "
 
\n", "
870.8%\n", "
 
\n", "
550.6%\n", "
 
\n", "
\n", "
\n", "
\n", "

Minimum 5 values

\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", "
ValueCountFrequency (%) 
060868.2%\n", "
 
\n", "
120923.5%\n", "
 
\n", "
2283.1%\n", "
 
\n", "
3161.8%\n", "
 
\n", "
4182.0%\n", "
 
\n", "
\n", "

Maximum 5 values

\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", "
ValueCountFrequency (%) 
2283.1%\n", "
 
\n", "
3161.8%\n", "
 
\n", "
4182.0%\n", "
 
\n", "
550.6%\n", "
 
\n", "
870.8%\n", "
 
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "
\n", "

Survived
\n", " Boolean\n", "

\n", "
\n", "
\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Distinct count2
Unique (%)0.2%
Missing (%)0.0%
Missing (n)0
\n", "
\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
Mean0.38384
\n", "
\n", "
\n", "
\n", "
\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", "
0\n", "
\n", " 549\n", "
\n", " \n", "
1\n", "
\n", " 342\n", "
\n", " \n", "
\n", "
\n", "
\n", " \n", " Toggle details\n", " \n", "
\n", "
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", "
ValueCountFrequency (%) 
054961.6%\n", "
 
\n", "
134238.4%\n", "
 
\n", "
\n", "
\n", "
\n", "
\n", "

Ticket
\n", " Categorical\n", "

\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Distinct count681
Unique (%)76.4%
Missing (%)0.0%
Missing (n)0
\n", "
\n", "
\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " \n", "\n", "
347082\n", "
\n", "  \n", "
\n", " 7\n", "
1601\n", "
\n", "  \n", "
\n", " 7\n", "
CA. 2343\n", "
\n", "  \n", "
\n", " 7\n", "
Other values (678)\n", "
\n", " 870\n", "
\n", " \n", "
\n", "
\n", "
\n", " \n", " Toggle details\n", " \n", "
\n", "
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", "
ValueCountFrequency (%) 
34708270.8%\n", "
 
\n", "
160170.8%\n", "
 
\n", "
CA. 234370.8%\n", "
 
\n", "
CA 214460.7%\n", "
 
\n", "
34708860.7%\n", "
 
\n", "
310129560.7%\n", "
 
\n", "
S.O.C. 1487950.6%\n", "
 
\n", "
38265250.6%\n", "
 
\n", "
34990940.4%\n", "
 
\n", "
LINE40.4%\n", "
 
\n", "
Other values (671)83493.6%\n", "
 
\n", "
\n", "
\n", "
\n", "
\n", "

Correlations

\n", "
\n", "
\n", " \n", " \n", "
\n", "
\n", "

Sample

\n", "
\n", "
\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarked
0103Braund, Mr. Owen Harrismale22.010A/5 211717.2500NaNS
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833C85C
2313Heikkinen, Miss. Lainafemale26.000STON/O2. 31012827.9250NaNS
3411Futrelle, Mrs. Jacques Heath (Lily May Peel)female35.01011380353.1000C123S
4503Allen, Mr. William Henrymale35.0003734508.0500NaNS
\n", "
\n", "
\n", "
" ], "text/plain": [ "" ] }, "execution_count": 112, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pandas_profiling.ProfileReport(titanic)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.3" } }, "nbformat": 4, "nbformat_minor": 2 }