NOTICE
The upcoming release of Featuretools 1.0.0 contains several breaking changes. Users are encouraged to test this version prior to release by installing from GitHub:
pip install https://github.com/alteryx/featuretools/archive/woodwork-integration.zip
For details on migrating to the new version, refer to Transitioning to Featuretools Version 1.0. Please report any issues in the Featuretools GitHub repo or by messaging in Alteryx Open Source Slack.
Featuretools provides users with the ability to remove features that are unlikely to be useful in building an effective machine learning model. Reducing the number of features in the feature matrix can both produce better results in the model as well as reduce the computational cost involved in prediction.
Featuretools enables users to perform feature selection on the results of Deep Feature Synthesis with three functions:
ft.selection.remove_highly_null_features
ft.selection.remove_single_value_features
ft.selection.remove_highly_correlated_features
We will describe each of these functions in depth, but first we must create an entity set with which we can run ft.dfs.
ft.dfs
[1]:
import pandas as pd import featuretools as ft from featuretools.selection import ( remove_highly_correlated_features, remove_highly_null_features, remove_single_value_features, ) from featuretools.primitives import NaturalLanguage from featuretools.demo.flight import load_flight es = load_flight(nrows=50) es
Downloading data ...
Entityset: Flight Data Entities: trip_logs [Rows: 50, Columns: 21] flights [Rows: 6, Columns: 9] airlines [Rows: 1, Columns: 1] airports [Rows: 4, Columns: 3] Relationships: trip_logs.flight_id -> flights.flight_id flights.carrier -> airlines.carrier flights.dest -> airports.dest
We might have a dataset with columns that have many null values. Deep Feature Synthesis might build features off of those null columns, creating even more highly null features. In this case, we might want to remove any features whose null values pass a certain threshold. Below is our feature matrix with such a case:
[2]:
fm, features = ft.dfs(entityset=es, target_entity="trip_logs", cutoff_time=pd.DataFrame({ 'trip_log_id':[30, 1, 2, 3, 4], 'time':pd.to_datetime(['2016-09-22 00:00:00']*5) }), trans_primitives=[], agg_primitives=[], max_depth=2) fm
We look at the above feature matrix and decide to remove the highly null features
[3]:
ft.selection.remove_highly_null_features(fm)
Notice that calling remove_highly_null_features didn’t remove every feature that contains a null value. By default, we only remove features where the percentage of null values in the calculated feature matrix is above 95%. If we want to lower that threshold, we can set the pct_null_threshold paramter ourselves.
remove_highly_null_features
pct_null_threshold
[4]:
remove_highly_null_features(fm, pct_null_threshold=.2)
Another situation we might run into is one where our calculated features don’t have any variance. In those cases, we are likely to want to remove the uninteresting features. For that, we use remove_single_value_features.
remove_single_value_features
Let’s see what happens when we remove the single value features of the feature matrix below.
[5]:
fm
Note
A list of feature definitions such as those created by dfs can be provided to the feature selection functions. Doing this will change the outputs to include an updated list of feature definitions.
dfs
[6]:
new_fm, new_features = remove_single_value_features(fm, features=features) new_fm
Now that we have the features definitions for the updated feature matrix, we can see that the features that were removed are:
[7]:
set(features) - set(new_features)
{<Feature: air_time>, <Feature: arr_delay>, <Feature: canceled>, <Feature: carrier_delay>, <Feature: dep_delay>, <Feature: diverted>, <Feature: flights.carrier>, <Feature: flights.flight_num>, <Feature: late_aircraft_delay>, <Feature: national_airspace_delay>, <Feature: security_delay>, <Feature: taxi_in>, <Feature: taxi_out>, <Feature: weather_delay>}
With the function used as it is above, null values are not considered when counting a feature’s unique values. If we’d like to consider NaN its own value, we can set count_nan_as_value to True and we’ll see flights.carrier and flights.flight_num back in the matrix.
NaN
count_nan_as_value
True
flights.carrier
flights.flight_num
[8]:
new_fm, new_features = remove_single_value_features(fm, features=features, count_nan_as_value=True) new_fm
The features that were removed are:
[9]:
{<Feature: air_time>, <Feature: arr_delay>, <Feature: canceled>, <Feature: carrier_delay>, <Feature: dep_delay>, <Feature: diverted>, <Feature: late_aircraft_delay>, <Feature: national_airspace_delay>, <Feature: security_delay>, <Feature: taxi_in>, <Feature: taxi_out>, <Feature: weather_delay>}
The last feature selection function we have allows us to remove features that would likely be redundant to the model we’re attempting to build by considering the correlation between pairs of calculated features.
When two features are determined to be highly correlated, we remove the more complex of the two. For example, say we have two features: col and -(col).
col
-(col)
We can see that -(col) is just the negation of col, and so we can guess those features are going to be highly correlated. -(col) has has the Negate primitive applied to it, so it is more complex than the identity feature col. Therefore, if we only want one of col and -(col), we should keep the identity feature. For features that don’t have an obvious difference in complexity, we discard the feature that comes later in the feature matrix.
Negate
Let’s try this out on our data:
[10]:
fm, features = ft.dfs(entityset=es, target_entity="trip_logs", trans_primitives=['negate'], agg_primitives=[], max_depth=3) fm.head()
Note that we have some pretty clear correlations here between all the features and their negations.
Now, using remove_highly_correlated_features, our default threshold for correlation is 95% correlated, and we get all of the obviously correlated features removed, leaving just the less complex features.
remove_highly_correlated_features
[11]:
new_fm, new_features = remove_highly_correlated_features(fm, features=features) new_fm.head()
[12]:
{<Feature: -(air_time)>, <Feature: -(arr_delay)>, <Feature: -(carrier_delay)>, <Feature: -(dep_delay)>, <Feature: -(distance)>, <Feature: -(late_aircraft_delay)>, <Feature: -(national_airspace_delay)>, <Feature: -(scheduled_elapsed_time)>, <Feature: -(taxi_in)>, <Feature: -(taxi_out)>, <Feature: air_time>, <Feature: distance>, <Feature: flights.distance_group>}
We can lower the threshold at which to remove correlated features if we’d like to be more restrictive by using the pct_corr_threshold parameter.
pct_corr_threshold
[13]:
new_fm , new_features = remove_highly_correlated_features(fm, features=features, pct_corr_threshold=.9) new_fm.head()
[14]:
{<Feature: -(air_time)>, <Feature: -(arr_delay)>, <Feature: -(carrier_delay)>, <Feature: -(dep_delay)>, <Feature: -(distance)>, <Feature: -(late_aircraft_delay)>, <Feature: -(national_airspace_delay)>, <Feature: -(scheduled_elapsed_time)>, <Feature: -(taxi_in)>, <Feature: -(taxi_out)>, <Feature: air_time>, <Feature: distance>, <Feature: flights.distance_group>, <Feature: national_airspace_delay>}
If we only want to check a subset of features, we can set features_to_check to the list of features whose correlation we’d like to check, and no features outside of that list will be removed.
features_to_check
[15]:
new_fm, new_features = remove_highly_correlated_features(fm, features=features, features_to_check=['air_time', 'distance', 'flights.distance_group']) new_fm.head()
[16]:
{<Feature: distance>, <Feature: flights.distance_group>}
To protect specific features from being removed from the feature matrix, we can include a list of features_to_keep, and these features will not be removed
features_to_keep
[17]:
new_fm, new_features = remove_highly_correlated_features(fm, features=features, features_to_keep=['air_time', 'distance', 'flights.distance_group']) new_fm.head()
[18]:
{<Feature: -(late_aircraft_delay)>, <Feature: -(arr_delay)>, <Feature: -(scheduled_elapsed_time)>, <Feature: -(carrier_delay)>, <Feature: -(taxi_in)>, <Feature: -(national_airspace_delay)>, <Feature: -(taxi_out)>, <Feature: -(air_time)>, <Feature: -(dep_delay)>, <Feature: -(distance)>}