{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Feature Selection\n", "\n", "Featuretools provides users with the ability to remove features that are unlikely to be useful in building an effective machine learning model. Reducing the number of features in the feature matrix can both produce better results in the model as well as reduce the computational cost involved in prediction.\n", "\n", "Featuretools enables users to perform feature selection on the results of Deep Feature Synthesis with three functions:\n", "\n", "- `ft.selection.remove_highly_null_features`\n", "- `ft.selection.remove_single_value_features`\n", "- `ft.selection.remove_highly_correlated_features`\n", "\n", "We will describe each of these functions in depth, but first we must create an entity set with which we can run `ft.dfs`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import featuretools as ft\n", "\n", "from featuretools.selection import (\n", " remove_highly_correlated_features,\n", " remove_highly_null_features,\n", " remove_single_value_features,\n", ")\n", "\n", "from featuretools.demo.flight import load_flight\n", "\n", "es = load_flight(nrows=50)\n", "es" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Remove Highly Null Features\n", "\n", "We might have a dataset with columns that have many null values. Deep Feature Synthesis might build features off of those null columns, creating even more highly null features. In this case, we might want to remove any features whose null values pass a certain threshold. Below is our feature matrix with such a case:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fm, features = ft.dfs(\n", " entityset=es,\n", " target_dataframe_name=\"trip_logs\",\n", " cutoff_time=pd.DataFrame(\n", " {\n", " \"trip_log_id\": [30, 1, 2, 3, 4],\n", " \"time\": pd.to_datetime([\"2016-09-22 00:00:00\"] * 5),\n", " }\n", " ),\n", " trans_primitives=[],\n", " agg_primitives=[],\n", " max_depth=2,\n", ")\n", "fm" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We look at the above feature matrix and decide to remove the highly null features" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ft.selection.remove_highly_null_features(fm)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Notice that calling `remove_highly_null_features` didn't remove every feature that contains a null value. By default, we only remove features where the percentage of null values in the calculated feature matrix is above 95%. If we want to lower that threshold, we can set the `pct_null_threshold` paramter ourselves." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "remove_highly_null_features(fm, pct_null_threshold=0.2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Remove Single Value Features\n", "\n", "Another situation we might run into is one where our calculated features don't have any variance. In those cases, we are likely to want to remove the uninteresting features. For that, we use `remove_single_value_features`.\n", "\n", "Let's see what happens when we remove the single value features of the feature matrix below." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fm" ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ ".. note ::\n", " A list of feature definitions such as those created by `dfs` can be provided to the feature selection functions.\n", " Doing this will change the outputs to include an updated list of feature definitions." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "new_fm, new_features = remove_single_value_features(fm, features=features)\n", "new_fm" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we have the features definitions for the updated feature matrix, we can see that the features that were removed are:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "set(features) - set(new_features)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With the function used as it is above, null values are not considered when counting a feature's unique values. If we'd like to consider `NaN` its own value, we can set `count_nan_as_value` to `True` and we'll see `flights.carrier` and `flights.flight_num` back in the matrix." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "new_fm, new_features = remove_single_value_features(\n", " fm, features=features, count_nan_as_value=True\n", ")\n", "new_fm" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The features that were removed are:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "set(features) - set(new_features)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Remove Highly Correlated Features\n", "\n", "The last feature selection function we have allows us to remove features that would likely be redundant to the model we're attempting to build by considering the correlation between pairs of calculated features.\n", "\n", "When two features are determined to be highly correlated, we remove the more complex of the two. For example, say we have two features: `col` and `-(col)`.\n", "\n", "We can see that `-(col)` is just the negation of `col`, and so we can guess those features are going to be highly correlated. `-(col)` has has the `Negate` primitive applied to it, so it is more complex than the identity feature `col`. Therefore, if we only want one of `col` and `-(col)`, we should keep the identity feature. For features that don't have an obvious difference in complexity, we discard the feature that comes later in the feature matrix. \n", "\n", "Let's try this out on our data:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fm, features = ft.dfs(\n", " entityset=es,\n", " target_dataframe_name=\"trip_logs\",\n", " trans_primitives=[\"negate\"],\n", " agg_primitives=[],\n", " max_depth=3,\n", ")\n", "fm.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that we have some pretty clear correlations here between all the features and their negations.\n", "\n", "Now, using `remove_highly_correlated_features`, our default threshold for correlation is 95% correlated, and we get all of the obviously correlated features removed, leaving just the less complex features." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "new_fm, new_features = remove_highly_correlated_features(fm, features=features)\n", "new_fm.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The features that were removed are:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "set(features) - set(new_features)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Change the correlation threshold\n", "\n", "We can lower the threshold at which to remove correlated features if we'd like to be more restrictive by using the `pct_corr_threshold` parameter." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "new_fm, new_features = remove_highly_correlated_features(\n", " fm, features=features, pct_corr_threshold=0.9\n", ")\n", "new_fm.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The features that were removed are:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "set(features) - set(new_features)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Check a Subset of Features\n", "\n", "If we only want to check a subset of features, we can set `features_to_check` to the list of features whose correlation we'd like to check, and no features outside of that list will be removed." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "new_fm, new_features = remove_highly_correlated_features(\n", " fm,\n", " features=features,\n", " features_to_check=[\"air_time\", \"distance\", \"flights.distance_group\"],\n", ")\n", "new_fm.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The features that were removed are:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "set(features) - set(new_features)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Protect Features from Removal\n", "\n", "To protect specific features from being removed from the feature matrix, we can include a list of `features_to_keep`, and these features will not be removed" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "new_fm, new_features = remove_highly_correlated_features(\n", " fm,\n", " features=features,\n", " features_to_keep=[\"air_time\", \"distance\", \"flights.distance_group\"],\n", ")\n", "new_fm.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The features that were removed are:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "set(features) - set(new_features)" ] } ], "metadata": { "celltoolbar": "Raw Cell Format", "interpreter": { "hash": "eadebc3a8a3dd54e52de25d3077ea0e41c7a462ff73c567da199d6de4c02ed7d" }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.2" } }, "nbformat": 4, "nbformat_minor": 2 }