cross-validation.ipynb

{"cells":[{"metadata":{"_cell_guid":"7840f4ef-1a26-4c0a-8a07-c644e0284c9a","_uuid":"04d410833dbf8d44354bdd1be7d5d4bc4e8b21de"},"cell_type":"markdown","source":"*This tutorial is part of the [Learn Machine Learning](https://www.kaggle.com/learn/learn-machine-learning/) series. In this step, you will learn how to use cross-validation for better measures of model performance.* \n\n\n# What is Cross Validation\n\nMachine learning is an iterative process. \n\nYou will face choices about predictive variables to use, what types of models to use,what arguments to supply those models, etc. We make these choices in a data-driven way by measuring model quality of various alternatives.\n\nYou've already learned to use `train_test_split` to split the data, so you can measure model quality on the test data.  Cross-validation extends this approach to model scoring (or \"model validation.\") Compared to `train_test_split`, cross-validation gives you a more reliable measure of your model's quality, though it takes longer to run.\n\n## The Shortcoming of Train-Test Split\n\nImagine you have a dataset with 5000 rows.  The `train_test_split` function has an argument for `test_size` that you can use to decide how many rows go to the training set and how many go to the test set.  The larger the test set, the more reliable your measures of model quality will be.  At an extreme, you could imagine having only 1 row of data in the test set.  If you compare alternative models, which one makes the best predictions on a single data point will be mostly a matter of luck.  \n\nYou will typically keep about 20% as a test dataset.  But even with 1000 rows in the test set, there's some random chance in determining model scores.  A model might do well on one set of 1000 rows, even if it would be inaccurate on a different 1000 rows.  The larger the test set, the less randomness (aka \"noise\") there is in our measure of model quality.\n\nBut we can only get a large test set by removing data from our training data, and smaller training datasets mean worse models.  In fact, the ideal modeling decisions on a small dataset typically aren't the best modeling decisions on large datasets. \n\n---"},{"metadata":{"_cell_guid":"38122f12-d962-4f55-8ca7-1fa73c46b9ec","_uuid":"57c22daaa523b8f33b30b6dc6caabc865f175d6e"},"cell_type":"markdown","source":"## The Cross-Validation Procedure\n\nIn cross-validation, we run our modeling process on different subsets of the data to get multiple measures of model quality. For example, we could have 5 **folds** or experiments.  We divide the data into 5 pieces, each being 20% of the full dataset.  \n\n![cross-validation-graphic](https://i.stack.imgur.com/1fXzJ.png)\n\n\nWe run an experiment called experiment 1 which uses the first fold as a holdout set, and everything else as training data. This gives us a measure of model quality based on a 20% holdout set, much as we got from using the simple train-test split.  \nWe then run a second experiment, where we hold out data from the second fold (using everything except the 2nd fold for training the model.) This gives us a second estimate of model quality.\nWe repeat this process, using every fold once as the holdout.  Putting this together, 100% of the data is used as a holdout at some point.  \n\nReturning to our example above from train-test split, if we have 5000 rows of data, we end up with a measure of model quality based on 5000 rows of holdout (even if we don't use all 5000 rows simultaneously.\n\n## Trade-offs Between Cross-Validation and Train-Test Split\nCross-validation gives a more accurate measure of model quality, which is especially important if you are making a lot of modeling decisions.  However, it can take more time to run, because it estimates models once for each fold.  So it is doing more total work.\n\nGiven these tradeoffs, when should you use each approach?\nOn small datasets, the extra computational burden of running cross-validation isn't a big deal.  These are also the problems where model quality scores would be least reliable with train-test split.  So, if your dataset is smaller, you should run cross-validation.\n\nFor the same reasons, a simple train-test split is sufficient for larger datasets.  It will run faster, and you may have enough data that there's little need to re-use some of it for holdout.\n\nThere's no simple threshold for what constitutes a large vs small dataset.  If your model takes a couple minute or less to run, it's probably worth switching to cross-validation.  If your model takes much longer to run, cross-validation may slow down your workflow more than it's worth.\n\nAlternatively, you can run cross-validation and see if the scores for each experiment seem close. If each experiment gives the same results, train-test split is probably sufficient."},{"metadata":{"_cell_guid":"29408386-fb9f-4c0b-b371-96a077935e7e","_uuid":"00dbe8d5657a671f18ffe3d17c64894c9ad40a6f"},"cell_type":"markdown","source":"# Example"},{"metadata":{"_cell_guid":"334b229d-db29-4cd0-a268-245f0192b828","_uuid":"93d63228dffcde1e1b8305eb71b000257700339d"},"cell_type":"markdown","source":"First we read the data"},{"metadata":{"_cell_guid":"8219a83e-fb0e-4d4f-b10f-6dd4c823c688","_uuid":"d6b611418224f51efad9e90b6aeb4c137ed583fd","trusted":true},"cell_type":"code","source":"import pandas as pd\ndata = pd.read_csv('../input/melb_data.csv')\ncols_to_use = ['Rooms', 'Distance', 'Landsize', 'BuildingArea', 'YearBuilt']\nX = data[cols_to_use]\ny = data.Price","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"f242f6a8-0094-460d-a43e-6c4183d2d009","_uuid":"5f71fcb8838edb64b1c05e801a2982bbae72318c"},"cell_type":"markdown","source":"Then specify a pipeline of our modeling steps (It can be very difficult to do cross-validation properly if you arent't using [pipelines](https://www.kaggle.com/dansbecker/pipelines))"},{"metadata":{"_cell_guid":"76d229a3-9694-4660-be97-3280f77a705b","_uuid":"bab6d2376cf5952a679c2d6d75438fccf9108358","trusted":true},"cell_type":"code","source":"from sklearn.ensemble import RandomForestRegressor\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import Imputer\nmy_pipeline = make_pipeline(Imputer(), RandomForestRegressor())","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"fd20ef5a-8966-40a4-9da9-08dc399b6db8","_uuid":"deffe7bce9a59cbb3ab5966a6532e424ea45755c"},"cell_type":"markdown","source":"Finally get the cross-validation scores:"},{"metadata":{"_cell_guid":"7d7b7adc-8dad-4274-af1c-58460f7a360d","_uuid":"e75a6151e651b4905453868bacaf6a645ddf13bc","trusted":true},"cell_type":"code","source":"\nfrom sklearn.model_selection import cross_val_score\nscores = cross_val_score(my_pipeline, X, y, scoring='neg_mean_absolute_error')\nprint(scores)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"f2fdf1b3-7924-4540-8fe6-422e76bcefe4","_uuid":"4c9b1e96b042daf641c9b7412a7d37bd9999ca6f"},"cell_type":"markdown","source":"You may notice that we specified an argument for *scoring*.  This specifies what measure of model quality to report.  The docs for scikit-learn show a [list of options](http://scikit-learn.org/stable/modules/model_evaluation.html).  \n\nIt is a little surprising that we specify *negative* mean absolute error in this case. Scikit-learn has a convention where all metrics are defined so a high number is better.  Using negatives here allows them to be consistent with that convention, though negative MAE is almost unheard of elsewhere.\n\nYou typically want a single measure of model quality to compare between models.  So we take the average across experiments."},{"metadata":{"_cell_guid":"2a6bf248-acea-4362-b049-de8086444030","_uuid":"25c2ea179d38a44906607707e82ba999fca52a91","trusted":true},"cell_type":"code","source":"print('Mean Absolute Error %2f' %(-1 * scores.mean()))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"edeef0ab-c554-42fe-a707-b6b50dfcc098","_uuid":"9db4ec132f5096b940a8f65f2fd259d1cca3b183"},"cell_type":"markdown","source":"# Conclusion\n\nUsing cross-validation gave us much better measures of model quality, with the added benefit of cleaning up our code (no longer needing to keep track of separate train and test sets.  So, it's a good win.\n\n# Your Turn\n1. Convert the code for your on-going project over from train-test split to cross-validation.  Make sure to remove all code that divides your dataset into training and testing datasets.  Leaving code you don't need any more would be sloppy.\n\n2. Add or remove a predictor from your models.  See the cross-validation score using both sets of predictors, and see how you can compare the scores."},{"metadata":{"trusted":true,"_uuid":"7f5fda3eaff6f03db6a26a2b54745cfebfa52b84"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}