2 - Trabalhando estruturas tabulares e arquivos.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2 - Trabalhando estruturas tabulares e arquivos\n",
    "\n",
    "Agora, iremos trabalhar com dois conceitos extremamente importantes para a ciência de dados: Series e Dataframes. Tratam-se de estruturas tabulares e que são comuns em várias atividades do dia a dia do cientista de dados. \n",
    "\n",
    "Para trabalhar com essas estruturas, utilizaremos o Pandas. O Pandas é uma biblioteca opensource, em Python, para análise de dados, e é uma das mais importantes ferramentas para quem deseja ser um cientista de dados.\n",
    "\n",
    "Além disso, iremos realizar as primeiras análises estatísticas descritivas e aprenderemos como importar e trabalhar com arquivos, incluindo alguns procedimentos básico de limpeza de dados."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Importando o módulo Pandas\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 Séries\n",
    "\n",
    "Quando trabalhamos com um vetor de uma única dimensão, devemos transformá-lo para uma Serie do Pandas.\n",
    "Uma série é um objeto unidemensional similar a um vetor, lista, ou coluna numa matriz ou tabela."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[23, 25, 24, 25, 15, 25, 27, 25]\n",
      "<class 'list'>\n"
     ]
    }
   ],
   "source": [
    "# Criando um vetor de idades\n",
    "idade = [23, 25, 24, 25, 15, 25, 27, 25]\n",
    "print (idade) # Imprime o vetor\n",
    "print (type(idade)) # Imprime o tipo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Soma: 189\n",
      "Valor máximo: 27\n",
      "Velor mínimo: 15\n"
     ]
    }
   ],
   "source": [
    "# Realizando algumas operações com o vetor criado\n",
    "print ('Soma:', sum(idade))\n",
    "print ('Valor máximo:', max(idade))\n",
    "print ('Velor mínimo:', min(idade))\n",
    "\n",
    "# Se você desejar realizar mais operações matemáticas, importe o módulo math (import math)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    23\n",
      "1    25\n",
      "2    24\n",
      "3    25\n",
      "4    15\n",
      "5    25\n",
      "6    27\n",
      "7    25\n",
      "Name: idades, dtype: int64\n",
      "<class 'pandas.core.series.Series'>\n"
     ]
    }
   ],
   "source": [
    "# Transformando o vetor em uma série\n",
    "idade_pd = pd.Series(idade, name='idades') # O primeiro parâmetro é o vetor que será convertido. O segundo, o nome da série\n",
    "print (idade_pd)\n",
    "print (type(idade_pd))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Quantidade: 8\n",
      "Soma: 189\n",
      "Máximo: 27\n",
      "Mínimo: 15\n",
      "Média: 23.6250\n",
      "Moda: 25\n",
      "Mediana: 25.0000\n",
      "Variância: 13.4107\n",
      "Desvio Padrão: 3.6621\n",
      "Sumário: \n",
      " count     8.000000\n",
      "mean     23.625000\n",
      "std       3.662064\n",
      "min      15.000000\n",
      "25%      23.750000\n",
      "50%      25.000000\n",
      "75%      25.000000\n",
      "max      27.000000\n",
      "Name: idades, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# Análise descritiva\n",
    "print ('Quantidade: %.d' % idade_pd.count())\n",
    "print('Soma: %.d' % idade_pd.sum())\n",
    "print('Máximo: %.d' % idade_pd.max())\n",
    "print('Mínimo: %.d' % idade_pd.min())\n",
    "print('Média: %.4f' % idade_pd.mean())\n",
    "print('Moda: %.d' % idade_pd.mode())\n",
    "print('Mediana: %.4f' % idade_pd.median())\n",
    "print ('Variância: %.4f' % idade_pd.var())\n",
    "print ('Desvio Padrão: %.4f' % idade_pd.std())\n",
    "print ('Sumário: \\n', idade_pd.describe())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 DataFrame\n",
    "\n",
    "DataFrame é uma estrutura de dados tabular de duas dimensões  composta de colunas e linhas indexadas, similar a uma tabela de excel. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[56, 78, 86, 88, 66, 77, 89, 90]\n",
      "[(23, 56), (25, 78), (24, 86), (25, 88), (15, 66), (25, 77), (27, 89), (25, 90)]\n"
     ]
    }
   ],
   "source": [
    "# Utilizarei o vetor idade, criado no início do script e criarei um vetor peso\n",
    "peso = [56, 78, 86, 88, 66, 77, 89, 90]\n",
    "print (peso)\n",
    "# Criando um dataset\n",
    "dados = list(zip(idade,peso)) # a função zip realiza um merge entre os vetores\n",
    "print (dados)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   Idades  Pesos\n",
      "0      23     56\n",
      "1      25     78\n",
      "2      24     86\n",
      "3      25     88\n",
      "4      15     66\n",
      "5      25     77\n",
      "6      27     89\n",
      "7      25     90\n"
     ]
    }
   ],
   "source": [
    "# Criando um dataframe (dados) com os vetores de idade e peso\n",
    "\n",
    "df_pd = pd.DataFrame(data=dados, \n",
    "                     columns=['Idades', 'Pesos'])\n",
    "print (df_pd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Idades</th>\n",
       "      <th>Pesos</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>8.000000</td>\n",
       "      <td>8.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>23.625000</td>\n",
       "      <td>78.750000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>3.662064</td>\n",
       "      <td>12.267844</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>15.000000</td>\n",
       "      <td>56.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>23.750000</td>\n",
       "      <td>74.250000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>25.000000</td>\n",
       "      <td>82.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>25.000000</td>\n",
       "      <td>88.250000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>27.000000</td>\n",
       "      <td>90.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Idades      Pesos\n",
       "count   8.000000   8.000000\n",
       "mean   23.625000  78.750000\n",
       "std     3.662064  12.267844\n",
       "min    15.000000  56.000000\n",
       "25%    23.750000  74.250000\n",
       "50%    25.000000  82.000000\n",
       "75%    25.000000  88.250000\n",
       "max    27.000000  90.000000"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# As mesmas operações descritivas podem ser realizadas no dataframe...\n",
    "df_pd.describe() # Para todos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "23.625"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Para analisar uma coluna, em especial, faça:\n",
    "df_pd['Idades'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   Idades  Pesos  Alturas\n",
      "0      23     56     1.56\n",
      "1      25     78     1.78\n",
      "2      24     86     1.89\n",
      "3      25     88     1.67\n",
      "4      15     66     1.58\n",
      "5      25     77     1.90\n",
      "6      27     89     1.87\n",
      "7      25     90     1.76\n"
     ]
    }
   ],
   "source": [
    "# Para adicionar uma nova coluna, ao dataframe faça:\n",
    "altura = [1.56, 1.78, 1.89, 1.67, 1.58, 1.90, 1.87, 1.76]\n",
    "df_pd['Alturas'] = altura\n",
    "print (df_pd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Muitas vezes também precisamos remover dimensões de um Dataframe. Para que isso seja possíve, execute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Idades</th>\n",
       "      <th>Pesos</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>23</td>\n",
       "      <td>56</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>25</td>\n",
       "      <td>78</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>24</td>\n",
       "      <td>86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>25</td>\n",
       "      <td>88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>15</td>\n",
       "      <td>66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>25</td>\n",
       "      <td>77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>27</td>\n",
       "      <td>89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>25</td>\n",
       "      <td>90</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Idades  Pesos\n",
       "0      23     56\n",
       "1      25     78\n",
       "2      24     86\n",
       "3      25     88\n",
       "4      15     66\n",
       "5      25     77\n",
       "6      27     89\n",
       "7      25     90"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_pd.drop('Alturas', axis=1, inplace=True)\n",
    "df_pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3 Trabalhando com arquivos CSV\n",
    "\n",
    "Uma das atividades mais realizadas por um cientista de dados é o trabalho com arquivos. Sendo assim, escolhi trabalhar, neste momento, com arquivos do tipo CSV.\n",
    "\n",
    "Utilizaremos o dataset disponibilizado pela World Health Organization (WHO).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Country</th>\n",
       "      <th>Region</th>\n",
       "      <th>Population</th>\n",
       "      <th>Under15</th>\n",
       "      <th>Over60</th>\n",
       "      <th>FertilityRate</th>\n",
       "      <th>LifeExpectancy</th>\n",
       "      <th>ChildMortality</th>\n",
       "      <th>CellularSubscribers</th>\n",
       "      <th>LiteracyRate</th>\n",
       "      <th>GNI</th>\n",
       "      <th>PrimarySchoolEnrollmentMale</th>\n",
       "      <th>PrimarySchoolEnrollmentFemale</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>Eastern Mediterranean</td>\n",
       "      <td>29825</td>\n",
       "      <td>47.42</td>\n",
       "      <td>3.82</td>\n",
       "      <td>5.40</td>\n",
       "      <td>60</td>\n",
       "      <td>98.5</td>\n",
       "      <td>54.26</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1140.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Albania</td>\n",
       "      <td>Europe</td>\n",
       "      <td>3162</td>\n",
       "      <td>21.33</td>\n",
       "      <td>14.93</td>\n",
       "      <td>1.75</td>\n",
       "      <td>74</td>\n",
       "      <td>16.7</td>\n",
       "      <td>96.39</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8820.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Algeria</td>\n",
       "      <td>Africa</td>\n",
       "      <td>38482</td>\n",
       "      <td>27.42</td>\n",
       "      <td>7.17</td>\n",
       "      <td>2.83</td>\n",
       "      <td>73</td>\n",
       "      <td>20.0</td>\n",
       "      <td>98.99</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8310.0</td>\n",
       "      <td>98.2</td>\n",
       "      <td>96.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Andorra</td>\n",
       "      <td>Europe</td>\n",
       "      <td>78</td>\n",
       "      <td>15.20</td>\n",
       "      <td>22.86</td>\n",
       "      <td>NaN</td>\n",
       "      <td>82</td>\n",
       "      <td>3.2</td>\n",
       "      <td>75.49</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>78.4</td>\n",
       "      <td>79.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Angola</td>\n",
       "      <td>Africa</td>\n",
       "      <td>20821</td>\n",
       "      <td>47.58</td>\n",
       "      <td>3.84</td>\n",
       "      <td>6.10</td>\n",
       "      <td>51</td>\n",
       "      <td>163.5</td>\n",
       "      <td>48.38</td>\n",
       "      <td>70.1</td>\n",
       "      <td>5230.0</td>\n",
       "      <td>93.1</td>\n",
       "      <td>78.2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Country                 Region  Population  Under15  Over60  \\\n",
       "0  Afghanistan  Eastern Mediterranean       29825    47.42    3.82   \n",
       "1      Albania                 Europe        3162    21.33   14.93   \n",
       "2      Algeria                 Africa       38482    27.42    7.17   \n",
       "3      Andorra                 Europe          78    15.20   22.86   \n",
       "4       Angola                 Africa       20821    47.58    3.84   \n",
       "\n",
       "   FertilityRate  LifeExpectancy  ChildMortality  CellularSubscribers  \\\n",
       "0           5.40              60            98.5                54.26   \n",
       "1           1.75              74            16.7                96.39   \n",
       "2           2.83              73            20.0                98.99   \n",
       "3            NaN              82             3.2                75.49   \n",
       "4           6.10              51           163.5                48.38   \n",
       "\n",
       "   LiteracyRate     GNI  PrimarySchoolEnrollmentMale  \\\n",
       "0           NaN  1140.0                          NaN   \n",
       "1           NaN  8820.0                          NaN   \n",
       "2           NaN  8310.0                         98.2   \n",
       "3           NaN     NaN                         78.4   \n",
       "4          70.1  5230.0                         93.1   \n",
       "\n",
       "   PrimarySchoolEnrollmentFemale  \n",
       "0                            NaN  \n",
       "1                            NaN  \n",
       "2                           96.4  \n",
       "3                           79.4  \n",
       "4                           78.2  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('dados/WHO.csv', delimiter=',')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Population</th>\n",
       "      <th>Under15</th>\n",
       "      <th>Over60</th>\n",
       "      <th>FertilityRate</th>\n",
       "      <th>LifeExpectancy</th>\n",
       "      <th>ChildMortality</th>\n",
       "      <th>CellularSubscribers</th>\n",
       "      <th>LiteracyRate</th>\n",
       "      <th>GNI</th>\n",
       "      <th>PrimarySchoolEnrollmentMale</th>\n",
       "      <th>PrimarySchoolEnrollmentFemale</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1.940000e+02</td>\n",
       "      <td>194.000000</td>\n",
       "      <td>194.000000</td>\n",
       "      <td>194.000000</td>\n",
       "      <td>194.000000</td>\n",
       "      <td>194.000000</td>\n",
       "      <td>184.000000</td>\n",
       "      <td>194.000000</td>\n",
       "      <td>162.000000</td>\n",
       "      <td>101.000000</td>\n",
       "      <td>193.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>3.635997e+04</td>\n",
       "      <td>28.732423</td>\n",
       "      <td>11.163660</td>\n",
       "      <td>2.773918</td>\n",
       "      <td>70.010309</td>\n",
       "      <td>36.148969</td>\n",
       "      <td>93.641522</td>\n",
       "      <td>83.667526</td>\n",
       "      <td>13320.925926</td>\n",
       "      <td>90.850495</td>\n",
       "      <td>92.181865</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>1.379031e+05</td>\n",
       "      <td>10.534573</td>\n",
       "      <td>7.149331</td>\n",
       "      <td>1.591610</td>\n",
       "      <td>9.259075</td>\n",
       "      <td>37.992935</td>\n",
       "      <td>41.400447</td>\n",
       "      <td>17.102326</td>\n",
       "      <td>15192.988650</td>\n",
       "      <td>11.017147</td>\n",
       "      <td>10.601230</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>13.120000</td>\n",
       "      <td>0.810000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>47.000000</td>\n",
       "      <td>2.200000</td>\n",
       "      <td>2.570000</td>\n",
       "      <td>31.100000</td>\n",
       "      <td>340.000000</td>\n",
       "      <td>37.200000</td>\n",
       "      <td>32.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.695750e+03</td>\n",
       "      <td>18.717500</td>\n",
       "      <td>5.200000</td>\n",
       "      <td>1.702500</td>\n",
       "      <td>64.000000</td>\n",
       "      <td>8.425000</td>\n",
       "      <td>63.567500</td>\n",
       "      <td>72.300000</td>\n",
       "      <td>2335.000000</td>\n",
       "      <td>87.700000</td>\n",
       "      <td>90.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>7.790000e+03</td>\n",
       "      <td>28.650000</td>\n",
       "      <td>8.530000</td>\n",
       "      <td>2.335000</td>\n",
       "      <td>72.500000</td>\n",
       "      <td>18.600000</td>\n",
       "      <td>97.745000</td>\n",
       "      <td>89.600000</td>\n",
       "      <td>7870.000000</td>\n",
       "      <td>94.700000</td>\n",
       "      <td>96.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>2.453525e+04</td>\n",
       "      <td>37.752500</td>\n",
       "      <td>16.687500</td>\n",
       "      <td>3.767500</td>\n",
       "      <td>76.000000</td>\n",
       "      <td>55.975000</td>\n",
       "      <td>120.805000</td>\n",
       "      <td>97.700000</td>\n",
       "      <td>17557.500000</td>\n",
       "      <td>98.100000</td>\n",
       "      <td>98.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.390000e+06</td>\n",
       "      <td>49.990000</td>\n",
       "      <td>31.920000</td>\n",
       "      <td>7.580000</td>\n",
       "      <td>83.000000</td>\n",
       "      <td>181.600000</td>\n",
       "      <td>196.410000</td>\n",
       "      <td>99.800000</td>\n",
       "      <td>86440.000000</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>100.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Population     Under15      Over60  FertilityRate  LifeExpectancy  \\\n",
       "count  1.940000e+02  194.000000  194.000000     194.000000      194.000000   \n",
       "mean   3.635997e+04   28.732423   11.163660       2.773918       70.010309   \n",
       "std    1.379031e+05   10.534573    7.149331       1.591610        9.259075   \n",
       "min    1.000000e+00   13.120000    0.810000       0.000000       47.000000   \n",
       "25%    1.695750e+03   18.717500    5.200000       1.702500       64.000000   \n",
       "50%    7.790000e+03   28.650000    8.530000       2.335000       72.500000   \n",
       "75%    2.453525e+04   37.752500   16.687500       3.767500       76.000000   \n",
       "max    1.390000e+06   49.990000   31.920000       7.580000       83.000000   \n",
       "\n",
       "       ChildMortality  CellularSubscribers  LiteracyRate           GNI  \\\n",
       "count      194.000000           184.000000    194.000000    162.000000   \n",
       "mean        36.148969            93.641522     83.667526  13320.925926   \n",
       "std         37.992935            41.400447     17.102326  15192.988650   \n",
       "min          2.200000             2.570000     31.100000    340.000000   \n",
       "25%          8.425000            63.567500     72.300000   2335.000000   \n",
       "50%         18.600000            97.745000     89.600000   7870.000000   \n",
       "75%         55.975000           120.805000     97.700000  17557.500000   \n",
       "max        181.600000           196.410000     99.800000  86440.000000   \n",
       "\n",
       "       PrimarySchoolEnrollmentMale  PrimarySchoolEnrollmentFemale  \n",
       "count                   101.000000                     193.000000  \n",
       "mean                     90.850495                      92.181865  \n",
       "std                      11.017147                      10.601230  \n",
       "min                      37.200000                      32.500000  \n",
       "25%                      87.700000                      90.400000  \n",
       "50%                      94.700000                      96.400000  \n",
       "75%                      98.100000                      98.500000  \n",
       "max                     100.000000                     100.000000  "
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Realizando análise descritiva do dataset\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.4 Realizando limpeza de dados (Data Cleaning)\n",
    "\n",
    "Na vida real, o cientista de dados irá passar por diversas dificuldades. O mundo real é muito diferente do mundo ideal... e, meio a tudo isso, o processo de Extração, Transformação e Carregamento (Extract, Transform, Load – ETL) se configuram como um dos mais complexos. \n",
    "\n",
    "Iniciaremos as atividades com a detecção de dados faltantes (missing data)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Country                          False\n",
       "Region                           False\n",
       "Population                       False\n",
       "Under15                          False\n",
       "Over60                           False\n",
       "FertilityRate                     True\n",
       "LifeExpectancy                   False\n",
       "ChildMortality                   False\n",
       "CellularSubscribers               True\n",
       "LiteracyRate                      True\n",
       "GNI                               True\n",
       "PrimarySchoolEnrollmentMale       True\n",
       "PrimarySchoolEnrollmentFemale     True\n",
       "dtype: bool"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# O any() possibibilitará saber, coluna a coluna, se existe qualquer um valor inexistente.\n",
    "df.isnull().any()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Country                           0\n",
       "Region                            0\n",
       "Population                        0\n",
       "Under15                           0\n",
       "Over60                            0\n",
       "FertilityRate                    11\n",
       "LifeExpectancy                    0\n",
       "ChildMortality                    0\n",
       "CellularSubscribers              10\n",
       "LiteracyRate                     91\n",
       "GNI                              32\n",
       "PrimarySchoolEnrollmentMale      93\n",
       "PrimarySchoolEnrollmentFemale    93\n",
       "dtype: int64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Contando quantos dados faltantes existem por coluna\n",
    "dados_faltantes = df.isnull().sum()\n",
    "dados_faltantes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total de dados faltantes: 330\n",
      "Em um percentual de: 13.08485329103886\n"
     ]
    }
   ],
   "source": [
    "# Quantos dados faltantes em sua totalidade\n",
    "print ('Total de dados faltantes:', dados_faltantes.sum())\n",
    "print ('Em um percentual de:', (dados_faltantes.sum()/ np.product(df.shape)) * 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Country                          False\n",
      "Region                           False\n",
      "Population                       False\n",
      "Under15                          False\n",
      "Over60                           False\n",
      "FertilityRate                    False\n",
      "LifeExpectancy                   False\n",
      "ChildMortality                   False\n",
      "CellularSubscribers              False\n",
      "LiteracyRate                     False\n",
      "GNI                              False\n",
      "PrimarySchoolEnrollmentMale      False\n",
      "PrimarySchoolEnrollmentFemale    False\n",
      "dtype: bool\n",
      "Número de registros: (194, 13)\n"
     ]
    }
   ],
   "source": [
    "# Verificando se existe alguma coluna em branco.\n",
    "print (df.isnull().all())\n",
    "print ('Número de registros:', df.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**P.S: Um fator importante aqui: São dados não informados ou eles realmente são inexistentes?**\n",
    "\n",
    "\n",
    "## 2.5 Removendo dados faltantes\n",
    "\n",
    "Se você realmente deseja remover as linhas que apresentam dados faltantes, faça:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Número de registros: (50, 13)\n"
     ]
    }
   ],
   "source": [
    "# Removendo todas as linhas que apresentam dados faltantes\n",
    "# O parâmetro thresh=2 deve ser utilizado quando você desejar remover apenas linhas/colunas que apresentam\n",
    "# mais de dois NaN; how='all' se tiver uma linha completa em NaN\n",
    "df.dropna(inplace=True)\n",
    "print ('Número de registros:', df.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['Country', 'Region', 'Population', 'Under15', 'Over60', 'FertilityRate',\n",
      "       'LifeExpectancy', 'ChildMortality', 'LiteracyRate', 'GNI',\n",
      "       'PrimarySchoolEnrollmentMale', 'PrimarySchoolEnrollmentFemale'],\n",
      "      dtype='object')\n"
     ]
    }
   ],
   "source": [
    "# Removendo uma coluna do Dataframe\n",
    "df.drop('CellularSubscribers', axis=1, inplace=True) # axis 1 = coluna; axis 0 = linha.\n",
    "print (df.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.6 Preenchendo dados faltantes\n",
    "\n",
    "Preencher dados faltantes vai precisar do *feeling* do cientista de dados. Creio até que seja uma das atividades mais difíceis de todas, pois um erro aqui pode fazer com que toda a sua análise seja desacreditada.\n",
    "\n",
    "Apresentarei três técnicas: preenchimento com zero, preenchimento com a média/mediana e a *bfill*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>FertilityRate</th>\n",
       "      <th>LiteracyRate</th>\n",
       "      <th>PrimarySchoolEnrollmentFemale</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.40</td>\n",
       "      <td>83.71068</td>\n",
       "      <td>96.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.75</td>\n",
       "      <td>83.71068</td>\n",
       "      <td>96.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.83</td>\n",
       "      <td>83.71068</td>\n",
       "      <td>96.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.00</td>\n",
       "      <td>83.71068</td>\n",
       "      <td>79.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6.10</td>\n",
       "      <td>70.10000</td>\n",
       "      <td>78.2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   FertilityRate  LiteracyRate  PrimarySchoolEnrollmentFemale\n",
       "0           5.40      83.71068                           96.4\n",
       "1           1.75      83.71068                           96.4\n",
       "2           2.83      83.71068                           96.4\n",
       "3           0.00      83.71068                           79.4\n",
       "4           6.10      70.10000                           78.2"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[['FertilityRate', 'LiteracyRate', 'PrimarySchoolEnrollmentFemale']].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Preenchendo os valores faltantes da FertilityRate por 0 (zero)\n",
    "df['FertilityRate'].fillna(0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Preenchendo os valores faltantes da LiteracyRate pela média\n",
    "df['LiteracyRate'].fillna(df['LiteracyRate'].mean(),inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Preenchendo os valores faltantes da LiteracyRate por bfill\n",
    "df['PrimarySchoolEnrollmentFemale'].fillna(method = 'bfill', inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.7 Criando novos Dataframes a partir de um existente\n",
    "\n",
    "A atividade de redução de dimensões é uma atividade muito importante para um cientista de dados. Sendo assim, aprenderemos agora algumas técnicas aplicadas para isso."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>FertilityRate</th>\n",
       "      <th>LiteracyRate</th>\n",
       "      <th>PrimarySchoolEnrollmentFemale</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.40</td>\n",
       "      <td>83.71068</td>\n",
       "      <td>96.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.75</td>\n",
       "      <td>83.71068</td>\n",
       "      <td>96.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.83</td>\n",
       "      <td>83.71068</td>\n",
       "      <td>96.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.00</td>\n",
       "      <td>83.71068</td>\n",
       "      <td>79.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6.10</td>\n",
       "      <td>70.10000</td>\n",
       "      <td>78.2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   FertilityRate  LiteracyRate  PrimarySchoolEnrollmentFemale\n",
       "0           5.40      83.71068                           96.4\n",
       "1           1.75      83.71068                           96.4\n",
       "2           2.83      83.71068                           96.4\n",
       "3           0.00      83.71068                           79.4\n",
       "4           6.10      70.10000                           78.2"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Selecionando dimensões de um Dataframe para criação de um novo Dataframe\n",
    "df_por_dimensoes = df[['FertilityRate', 'LiteracyRate', 'PrimarySchoolEnrollmentFemale']]\n",
    "df_por_dimensoes.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(105, 13)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Criando um novo Dataframe a partir de uma regra\n",
    "df_regra = df[df.GNI < df.GNI.mean()]\n",
    "df_regra.shape"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}