Transitioning to Featuretools Version 1.0#
Featuretools version 1.0 incorporates many significant changes that impact the way EntitySets are created, how primitives are defined, and in some cases the resulting feature matrix that is created. This document will provide an overview of the significant changes, helping existing Featuretools users transition to version 1.0.
Background and Introduction#
Why make these changes?#
The lack of a unified type system across libraries makes sharing information between libraries more difficult. This problem led to the development of Woodwork. Updating Featuretools to use Woodwork for managing column typing information enables easy sharing of feature matrix column types with other libraries without costly conversions between custom type systems. As an example, EvalML, which has also adopted Woodwork, can now use Woodwork typing information on a feature matrix directly to create machine learning models, without first inferring or redefining column types.
Other benefits of using Woodwork for managing typing in Featuretools include:
Simplified code - custom type management code has been removed
Seamless integration of new types and improvements to type integration as Woodwork improves
Easy and flexible storage of additional information about columns. For example, we can now store whether a feature was engineered by Featuretools or present in the original data.
What has changed?#
The legacy Featuretools custom typing system has been replaced with Woodwork for managing column types
Both the
Entity
andVariable
classes have been removed from FeaturetoolsSeveral key Featuretools methods have been moved or updated
Comparison between legacy typing system and Woodwork typing systems#
Featuretools < 1.0 |
Featuretools 1.0 |
Description |
---|---|---|
Entity |
Woodwork DataFrame |
stores typing information for all columns |
Variable |
ColumnSchema |
stores typing information for a single column |
Variable subclass |
LogicalType and semantic_tags |
elements used to define a column type |
Summary of significant method changes#
The table below outlines the most significant changes that have occurred. In Summary: In some cases, the method arguments have also changed, and those changes are outlined in more detail throughout this document.
Older Versions |
Featuretools 1.0 |
---|---|
EntitySet.entity_from_dataframe |
EntitySet.add_dataframe |
EntitySet.normalize_entity |
EntitySet.normalize_dataframe |
EntitySet.update_data |
EntitySet.replace_dataframe |
Entity.variable_types |
es[‘dataframe_name’].ww |
es[‘entity_id’][‘variable_name’] |
es[‘dataframe_name’].ww.columns[‘column_name’] |
Entity.convert_variable_type |
es[‘dataframe_name’].ww.set_types |
Entity.add_interesting_values |
es.add_interesting_values(dataframe_name=‘df_name’, …) |
Entity.set_secondary_time_index |
es.set_secondary_time_index(dataframe_name=‘df_name’, …) |
Feature(es[‘entity_id’][‘variable_name’]) |
Feature(es[‘dataframe_name’].ww[‘column_name’]) |
dfs(target_entity=‘entity_id’, …) |
dfs(target_dataframe_name=‘dataframe_name’, …) |
For more information on how Woodwork manages typing information, refer to the Woodwork Understanding Types and Tags guide.
What do these changes mean for users?#
Removing these classes required moving several methods from the Entity
to the EntitySet
object. This change also impacts the way relationships, features and primitives are defined, requiring different parameters than were previously required. Also, because the Woodwork typing system is not identical to the old Featuretools typing system, in some cases the feature matrix that is returned can be slightly different as a result of columns being identified as different types.
All of these changes, and more, will be reviewed in detail throughout this document, providing examples of both the old and new API where possible.
Removal of Entity
Class and Updates to EntitySet
#
In previous versions of Featuretools an EntitySet was created by adding multiple entities and then defining relationships between variables (columns) in different entities. Starting in Featuretools version 1.0, EntitySets are now created by adding multiple dataframes and defining relationships between columns in the dataframes. While conceptually similar, there are some minor differences in the process.
Adding dataframes to an EntitySet#
When adding dataframes to an EntitySet, users can pass in a Woodwork dataframe or a regular dataframe without Woodwork typing information. As before, Featuretools supports creating EntitySets from pandas, Dask and Spark dataframes. If users supply a dataframe that has Woodwork typing information initialized, Featuretools will simply use this typing information directly. If users supply a dataframe without Woodwork initialized, Featuretools will initialize Woodwork on the dataframe, performing type inference for any column that does not have typing information specified.
Below are some examples to illustrate this process. First we will create two small dataframes to use for the example.
[1]:
import pandas as pd
import featuretools as ft
[2]:
orders_df = pd.DataFrame(
{"order_id": [0, 1, 2], "order_date": ["2021-01-02", "2021-01-03", "2021-01-04"]}
)
items_df = pd.DataFrame(
{
"id": [0, 1, 2, 3, 4],
"order_id": [0, 1, 1, 2, 2],
"item_price": [29.95, 4.99, 10.25, 20.50, 15.99],
"on_sale": [False, True, False, True, False],
}
)
With older versions of Featuretools, users would first create an EntitySet object, and then add dataframes to the EntitySet, by calling entity_from_dataframe
as shown below.
es = ft.EntitySet('old_es')
es.entity_from_dataframe(dataframe=orders_df,
entity_id='orders',
index='order_id',
time_index='order_date')
es.entity_from_dataframe(dataframe=items_df,
entity_id='items',
index='id')
Entityset: old_es
Entities:
orders [Rows: 3, Columns: 2]
items [Rows: 5, Columns: 3]
Relationships:
No relationships
With Featuretools 1.0, the steps for adding a dataframe to an EntitySet are the same, but some of the details have changed. First, create an EntitySet as before. To add the dataframe call EntitySet.add_dataframe
in place of the previous EntitySet.entity_from_dataframe
call. Note that the name of the dataframe is specified in the dataframe_name
argument, which was previously called entity_id
.
[3]:
es = ft.EntitySet("new_es")
es.add_dataframe(
dataframe=orders_df,
dataframe_name="orders",
index="order_id",
time_index="order_date",
)
[3]:
Entityset: new_es
DataFrames:
orders [Rows: 3, Columns: 2]
Relationships:
No relationships
You can also define the name, index, and time index by first initializing Woodwork on the dataframe and then passing the Woodwork initialized dataframe directly to the add_dataframe
call. For this example we will initialize Woodwork on items_df
, setting the dataframe name as items
and specifying that the index should be the
id
column.
[4]:
items_df.ww.init(name="items", index="id")
items_df.ww
[4]:
Physical Type | Logical Type | Semantic Tag(s) | |
---|---|---|---|
Column | |||
id | int64 | Integer | ['index'] |
order_id | int64 | Integer | ['numeric'] |
item_price | float64 | Double | ['numeric'] |
on_sale | bool | Boolean | [] |
With Woodwork initialized, we no longer need to specify values for the dataframe_name
or index
arguments when calling add_dataframe
as Featuretools will simply use the values that were already specified when Woodwork was initialized.
[5]:
es.add_dataframe(dataframe=items_df)
[5]:
Entityset: new_es
DataFrames:
orders [Rows: 3, Columns: 2]
items [Rows: 5, Columns: 4]
Relationships:
No relationships
Accessing column typing information#
Previously, column variable type information could be accessed for an entire Entity through Entity.variable_types
or for an individual column by selecting the individual column first through es['entity_id']['col_id']
.
es['items'].variable_types
{'id': featuretools.variable_types.variable.Index,
'order_id': featuretools.variable_types.variable.Numeric,
'item_price': featuretools.variable_types.variable.Numeric}
es['items']['item_price']
<Variable: item_price (dtype = numeric)>
With the updated version of Featuretools, the logical types and semantic tags for all of the columns in a single dataframe can be viewed through the .ww
namespace on the dataframe. First, select the dataframe from the EntitySet with es['dataframe_name']
and then access the typing information by chaining a .ww
call on the end as shown below.
[6]:
es["items"].ww
[6]:
Physical Type | Logical Type | Semantic Tag(s) | |
---|---|---|---|
Column | |||
id | int64 | Integer | ['index'] |
order_id | int64 | Integer | ['numeric'] |
item_price | float64 | Double | ['numeric'] |
on_sale | bool | Boolean | [] |
The logical type and semantic tags for a single column can be obtained from the Woodwork columns dictionary stored on the dataframe, returning a Woodwork.ColumnSchema
object that stores the typing information:
[7]:
es["items"].ww.columns["item_price"]
[7]:
<ColumnSchema (Logical Type = Double) (Semantic Tags = ['numeric'])>
Type inference and updating column types#
Featuretools will attempt to infer types for any columns that do not have types defined by the user. Prior to version 1.0, Featuretools implemented custom type inference code to determine what variable type should be assigned to each column. You could see the inferred variable types by viewing the contents of the Entity.variable_types
dictionary.
Starting in Featuretools 1.0, column type inference is being handled by Woodwork. Any columns that do not have a logical type assigned by the user when adding a dataframe to an EntitySet will have their logical types inferred by Woodwork. As before, type inference can be skipped for any columns in a dataframe by passing the appropriate logical types in a dictionary when calling EntitySet.add_dataframe
.
As an example, we can create a new dataframe and add it to an EntitySet, specifying the logical type for the user’s full name as the Woodwork PersonFullName
logical type.
[8]:
users_df = pd.DataFrame(
{"id": [0, 1, 2], "name": ["John Doe", "Rita Book", "Teri Dactyl"]}
)
[9]:
es.add_dataframe(
dataframe=users_df,
dataframe_name="users",
index="id",
logical_types={"name": "PersonFullName"},
)
es["users"].ww
[9]:
Physical Type | Logical Type | Semantic Tag(s) | |
---|---|---|---|
Column | |||
id | int64 | Integer | ['index'] |
name | string | PersonFullName | [] |
Looking at the typing information above, we can see that the logical type for the name
column was set to PersonFullName
as we specified.
Situations will occur where type inference identifies a column as having the incorrect logical type. In these situations, the logical type can be updated using the Woodwork set_types
method. Let’s say we want the order_id
column of the orders
dataframe to have a Categorical
logical type instead of the Integer
type that was inferred. Previously, this would have accomplished through the Entity.convert_variable_type
method.
from featuretools.variable_types import Categorical
es['items'].convert_variable_type(variable_id='order_id', new_type=Categorical)
Now, we can perform this same update using Woodwork:
[10]:
es["items"].ww.set_types(logical_types={"order_id": "Categorical"})
es["items"].ww
[10]:
Physical Type | Logical Type | Semantic Tag(s) | |
---|---|---|---|
Column | |||
id | int64 | Integer | ['index'] |
order_id | category | Categorical | ['category'] |
item_price | float64 | Double | ['numeric'] |
on_sale | bool | Boolean | [] |
For additional information on Woodwork typing and how it is used in Featuretools, refer to Woodwork Typing in Featuretools.
Adding interesting values#
Interesting values can be added to all dataframes in an EntitySet, a single dataframe in an EntitySet, or to a single column of a dataframe in an EntitySet.
To add interesting values for all of the dataframes in an EntitySet, simply call EntitySet.add_interesting_values
, optionally specifying the maximum number of values to add for each column. This remains unchanged from older versions of Featuretools to the 1.0 release.
Adding values for a single dataframe or for a single column has changed. Previously to add interesting values for an Entity, users would call Entity.add_interesting_values()
:
es['items'].add_interesting_values()
Now, in order to specify interesting values for a single dataframe, you call add_interesting_values
on the EntitySet, and pass the name of the dataframe for which you want interesting values added:
[11]:
es.add_interesting_values(dataframe_name="items")
Previously, to manually add interesting values for a column, you would simply assign them to the attribute of the variable:
es['items']['order_id'].interesting_values = [1, 2]
Now, this is done through EntitySet.add_interesting_values
, passing in the name of the dataframe and a dictionary mapping column names to the interesting values to assign for that column. For example, to assign the interesting values of [1, 2]
to the order_id
column of the items
dataframe, use the following approach:
[12]:
es.add_interesting_values(dataframe_name="items", values={"order_id": [1, 2]})
Interesting values for multiple columns in the same dataframe can be assigned by adding more entries to the dictionary passed to the values
parameter.
Accessing interesting values has changed as well. Previously interesting values could be viewed from the variable:
es['items']['order_id'].interesting_values
Interesting values are now stored in the Woodwork metadata for the columns in a dataframe:
[13]:
es["items"].ww.columns["order_id"].metadata["interesting_values"]
[13]:
[1, 2]
Setting a secondary time index#
In earlier versions of Featuretools, a secondary time index could be set on an Entity by calling Entity.set_secondary_time_index
.
es_flight = ft.demo.load_flight(nrows=100)
arr_time_columns = ['arr_delay', 'dep_delay', 'carrier_delay', 'weather_delay',
'national_airspace_delay', 'security_delay',
'late_aircraft_delay', 'canceled', 'diverted',
'taxi_in', 'taxi_out', 'air_time', 'dep_time']
es_flight['trip_logs'].set_secondary_time_index({'arr_time': arr_time_columns})
Since the Entity
class has been removed in Featuretools 1.0, this now needs to be done through the EntitySet
instead:
[14]:
es_flight = ft.demo.load_flight(nrows=100)
arr_time_columns = [
"arr_delay",
"dep_delay",
"carrier_delay",
"weather_delay",
"national_airspace_delay",
"security_delay",
"late_aircraft_delay",
"canceled",
"diverted",
"taxi_in",
"taxi_out",
"air_time",
"dep_time",
]
es_flight.set_secondary_time_index(
dataframe_name="trip_logs", secondary_time_index={"arr_time": arr_time_columns}
)
Downloading data ...
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-featuretools/envs/v1.30.0/lib/python3.9/site-packages/featuretools/demo/flight.py:291: PerformanceWarning: Adding/subtracting object-dtype array to TimedeltaArray not vectorized.
clean_data.loc[:, "dep_time"] = clean_data["scheduled_dep_time"] + pd.to_timedelta(
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-featuretools/envs/v1.30.0/lib/python3.9/site-packages/featuretools/demo/flight.py:296: PerformanceWarning: Adding/subtracting object-dtype array to TimedeltaArray not vectorized.
clean_data.loc[:, "arr_time"] = clean_data["dep_time"] + pd.to_timedelta(
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-featuretools/envs/v1.30.0/lib/python3.9/site-packages/featuretools/demo/flight.py:302: PerformanceWarning: Adding/subtracting object-dtype array to TimedeltaArray not vectorized.
clean_data["scheduled_dep_time"] + clean_data["scheduled_elapsed_time"]
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-featuretools/envs/v1.30.0/lib/python3.9/site-packages/woodwork/type_sys/utils.py:40: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.
pd.to_datetime(
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-featuretools/envs/v1.30.0/lib/python3.9/site-packages/woodwork/type_sys/utils.py:40: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.
pd.to_datetime(
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-featuretools/envs/v1.30.0/lib/python3.9/site-packages/woodwork/logical_types.py:897: FutureWarning: Downcasting behavior in `replace` is deprecated and will be removed in a future version. To retain the old behavior, explicitly call `result.infer_objects(copy=False)`. To opt-in to the future behavior, set `pd.set_option('future.no_silent_downcasting', True)`
series = series.replace(ww.config.get_option("nan_values"), np.nan)
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-featuretools/envs/v1.30.0/lib/python3.9/site-packages/woodwork/logical_types.py:897: FutureWarning: Downcasting behavior in `replace` is deprecated and will be removed in a future version. To retain the old behavior, explicitly call `result.infer_objects(copy=False)`. To opt-in to the future behavior, set `pd.set_option('future.no_silent_downcasting', True)`
series = series.replace(ww.config.get_option("nan_values"), np.nan)
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-featuretools/envs/v1.30.0/lib/python3.9/site-packages/woodwork/type_sys/utils.py:40: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.
pd.to_datetime(
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-featuretools/envs/v1.30.0/lib/python3.9/site-packages/woodwork/type_sys/utils.py:40: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.
pd.to_datetime(
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-featuretools/envs/v1.30.0/lib/python3.9/site-packages/woodwork/type_sys/utils.py:40: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.
pd.to_datetime(
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-featuretools/envs/v1.30.0/lib/python3.9/site-packages/woodwork/type_sys/utils.py:40: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.
pd.to_datetime(
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-featuretools/envs/v1.30.0/lib/python3.9/site-packages/woodwork/type_sys/utils.py:40: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.
pd.to_datetime(
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-featuretools/envs/v1.30.0/lib/python3.9/site-packages/woodwork/type_sys/utils.py:40: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.
pd.to_datetime(
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-featuretools/envs/v1.30.0/lib/python3.9/site-packages/woodwork/type_sys/utils.py:40: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.
pd.to_datetime(
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-featuretools/envs/v1.30.0/lib/python3.9/site-packages/woodwork/type_sys/utils.py:40: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.
pd.to_datetime(
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-featuretools/envs/v1.30.0/lib/python3.9/site-packages/woodwork/type_sys/utils.py:40: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.
pd.to_datetime(
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-featuretools/envs/v1.30.0/lib/python3.9/site-packages/woodwork/type_sys/utils.py:40: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.
pd.to_datetime(
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-featuretools/envs/v1.30.0/lib/python3.9/site-packages/woodwork/type_sys/utils.py:40: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.
pd.to_datetime(
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-featuretools/envs/v1.30.0/lib/python3.9/site-packages/woodwork/type_sys/utils.py:40: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.
pd.to_datetime(
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-featuretools/envs/v1.30.0/lib/python3.9/site-packages/woodwork/logical_types.py:897: FutureWarning: Downcasting behavior in `replace` is deprecated and will be removed in a future version. To retain the old behavior, explicitly call `result.infer_objects(copy=False)`. To opt-in to the future behavior, set `pd.set_option('future.no_silent_downcasting', True)`
series = series.replace(ww.config.get_option("nan_values"), np.nan)
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-featuretools/envs/v1.30.0/lib/python3.9/site-packages/woodwork/logical_types.py:897: FutureWarning: Downcasting behavior in `replace` is deprecated and will be removed in a future version. To retain the old behavior, explicitly call `result.infer_objects(copy=False)`. To opt-in to the future behavior, set `pd.set_option('future.no_silent_downcasting', True)`
series = series.replace(ww.config.get_option("nan_values"), np.nan)
Previously, the secondary time index could be accessed directly from the Entity with es_flight['trip_logs'].secondary_time_index
. Starting in Featuretools 1.0 the secondary time index and the associated columns are stored in the Woodwork dataframe metadata and can be accessed as shown below.
[15]:
es_flight["trip_logs"].ww.metadata["secondary_time_index"]
[15]:
{'arr_time': ['arr_delay',
'dep_delay',
'carrier_delay',
'weather_delay',
'national_airspace_delay',
'security_delay',
'late_aircraft_delay',
'canceled',
'diverted',
'taxi_in',
'taxi_out',
'air_time',
'dep_time',
'arr_time']}
Normalizing Entities/DataFrames#
EntitySet.normalize_entity
has been renamed to EntitySet.normalize_dataframe
in Featuretools 1.0. The new method works in the same way as the old method, but some of the parameters have been renamed. The table below shows the old and new names for reference. When calling this method, the new parameter names need to be used.
Old Parameter Name |
New Parameter Name |
---|---|
base_entity_id |
base_dataframe_name |
new_entity_id |
new_dataframe_name |
additional_variables |
additional_columns |
copy_variables |
copy_columns |
new_entity_time_index |
new_dataframe_time_index |
new_entity_secondary_time_index |
new_dataframe_secondary_time_index |
Defining and adding relationships#
In earlier versions of Featuretools, relationships were defined by creating a Relationship
object, which took two Variables
as inputs. To define a relationship between the orders Entity and the items Entity, we would first create a Relationship
and then add it to the EntitySet:
relationship = ft.Relationship(es['orders']['order_id'], es['items']['order_id'])
es.add_relationship(relationship)
With Featuretools 1.0, the process is similar, but there are two different ways to add the relationship to the EntitySet. One way is to pass the dataframe and column names to EntitySet.add_relationship
, and another is to pass a previously created Relationship
object to the relationship
keyword argument. Both approaches are demonstrated below.
[16]:
# Undo change from above and change child column logical type to match parent and prevent warning
# NOTE: This cell is hidden in the docs build
es["items"].ww.set_types(logical_types={"order_id": "Integer"})
[17]:
es.add_relationship(
parent_dataframe_name="orders",
parent_column_name="order_id",
child_dataframe_name="items",
child_column_name="order_id",
)
[17]:
Entityset: new_es
DataFrames:
orders [Rows: 3, Columns: 2]
items [Rows: 5, Columns: 4]
users [Rows: 3, Columns: 2]
Relationships:
items.order_id -> orders.order_id
[18]:
# Reset the relationship so we can add it again
# NOTE: This cell is hidden in the docs build
es.relationships = []
Alternatively, we can first create a Relationship
and pass that to EntitySet.add_relationship
. When defining a Relationship
we need to pass in the EntitySet to which it belongs along with the names for the parent dataframe and parent column and the name of the child dataframe and child column.
[19]:
relationship = ft.Relationship(
entityset=es,
parent_dataframe_name="orders",
parent_column_name="order_id",
child_dataframe_name="items",
child_column_name="order_id",
)
es.add_relationship(relationship=relationship)
[19]:
Entityset: new_es
DataFrames:
orders [Rows: 3, Columns: 2]
items [Rows: 5, Columns: 4]
users [Rows: 3, Columns: 2]
Relationships:
items.order_id -> orders.order_id
Updating data for a dataframe in an EntitySet#
Previously to update (replace) the data associated with an Entity, users could call Entity.update_data
and pass in the new dataframe. As an example, let’s update the data in our users
Entity:
new_users_df = pd.DataFrame({
'id': [3, 4],
'name': ['Anne Teak', 'Art Decco']
})
es['users'].update_data(df=new_users_df)
To accomplish this task with Featuretools 1.0, we will use the EntitySet.replace_dataframe
method instead:
[20]:
new_users_df = pd.DataFrame({"id": [0, 1], "name": ["Anne Teak", "Art Decco"]})
es.replace_dataframe(dataframe_name="users", df=new_users_df)
es["users"]
[20]:
id | name | |
---|---|---|
0 | 0 | Anne Teak |
1 | 1 | Art Decco |
Defining features#
The syntax for defining features has changed slightly in Featuretools 1.0. Previously, identity features could be defined simply by passing in the variable that should be used to build the feature.
feature = ft.Feature(es['items']['item_price'])
Starting with Featuretools 1.0, a similar syntax can be used, but because es['items']
will now return a Woodwork dataframe instead of an Entity
, we need to update the syntax slightly to access the Woodwork column. To update, simply add .ww
between the dataframe name selector and the column selector as shown below.
[21]:
feature = ft.Feature(es["items"].ww["item_price"])
Defining primitives#
In earlier versions of Featuretools, primitive input and return types were defined by specifying the appropriate Variable
class. Starting in version 1.0, the input and return types are defined by Woodwork ColumnSchema
objects.
To illustrate this change, let’s look closer at the Age
transform primitive. This primitive takes a datetime representing a date of birth and returns a numeric value corresponding to a person’s age. In previous versions of Featuretools, the input type was defined by specifying the DateOfBirth
variable type and the return type was specified by the Numeric
variable type:
input_types = [DateOfBirth]
return_type = Numeric
Woodwork does not have a specific DateOfBirth
logical type, but rather identifies a column as a date of birth column by specifying the logical type as Datetime
with a semantic tag of date_of_birth
. There is also no Numeric
logical type in Woodwork, but rather Woodwork identifies all columns that can be used for numeric operations with the semantic tag of numeric
. Furthermore, we know the Age
primitive will return a floating point number, which would correspond to a
Woodwork logical type of Double
. With these items in mind, we can redefine the Age
input types and return types with ColumnSchema
objects as follows:
input_types = [ColumnSchema(logical_type=Datetime, semantic_tags={'date_of_birth'})]
return_type = ColumnSchema(logical_type=Double, semantic_tags={'numeric'})
Aside from changing the way input and return types are defined, the rest of the process for defining primitives remains unchanged.
Mapping from old Featuretools variable types to Woodwork ColumnSchemas#
Types defined by Woodwork differ from the old variable types that were defined by Featuretools prior to version 1.0. While there is not a direct mapping from the old variable types to the new Woodwork types defined by ColumnSchema
objects, the approximate mapping is shown below.
Featuretools Variable |
Woodwork Column Schema |
---|---|
Boolean |
ColumnSchema(logical_type=Boolean) or ColumnSchema(logical_type=BooleanNullable) |
Categorical |
ColumnSchema(logical_type=Categorical) |
CountryCode |
ColumnSchema(logical_type=CountryCode) |
Datetime |
ColumnSchema(logical_type=Datetime) |
DateOfBirth |
ColumnSchema(logical_type=Datetime, semantic_tags={‘date_of_birth’}) |
DatetimeTimeIndex |
ColumnSchema(logical_type=Datetime, semantic_tags={‘time_index’}) |
Discrete |
ColumnSchema(semantic_tags={‘category’}) |
EmailAddress |
ColumnSchema(logical_type=EmailAddress) |
FilePath |
ColumnSchema(logical_type=Filepath) |
FullName |
ColumnSchema(logical_type=PersonFullName) |
Id |
ColumnSchema(semantic_tags={‘foreign_key’}) |
Index |
ColumnSchema(semantic_tags={‘index’}) |
IPAddress |
ColumnSchema(logical_type=IPAddress) |
LatLong |
ColumnSchema(logical_type=LatLong) |
NaturalLanguage |
ColumnSchema(logical_type=NaturalLanguage) |
Numeric |
ColumnSchema(semantic_tags={‘numeric’}) |
NumericTimeIndex |
ColumnSchema(semantic_tags={‘numeric’, ‘time_index’}) |
Ordinal |
ColumnSchema(logical_type=Ordinal) |
PhoneNumber |
ColumnSchema(logical_type=PhoneNumber) |
SubRegionCode |
ColumnSchema(logical_type=SubRegionCode) |
Timedelta |
ColumnSchema(logical_type=Timedelta) |
TimeIndex |
ColumnSchema(semantic_tags={‘time_index’}) |
URL |
ColumnSchema(logical_type=URL) |
Unknown |
ColumnSchema(logical_type=Unknown) |
ZIPCode |
ColumnSchema(logical_type=PostalCode) |
Changes to Deep Feature Synthesis and Calculate Feature Matrix#
The argument names for both featuretools.dfs
and featuretools.calculate_feature_matrix
have changed slightly in Featuretools 1.0. In prior versions, users could generate a list of features using the default primitives and options like this:
features = ft.dfs(entityset=es,
target_entity='items',
features_only=True)
In Featuretools 1.0, the target_entity
argument has been renamed to target_dataframe_name
, but otherwise this basic call remains the same.
[22]:
features = ft.dfs(entityset=es, target_dataframe_name="items", features_only=True)
features
[22]:
[<Feature: order_id>,
<Feature: item_price>,
<Feature: on_sale>,
<Feature: orders.COUNT(items)>,
<Feature: orders.MAX(items.item_price)>,
<Feature: orders.MEAN(items.item_price)>,
<Feature: orders.MIN(items.item_price)>,
<Feature: orders.PERCENT_TRUE(items.on_sale)>,
<Feature: orders.SKEW(items.item_price)>,
<Feature: orders.STD(items.item_price)>,
<Feature: orders.SUM(items.item_price)>,
<Feature: orders.DAY(order_date)>,
<Feature: orders.MONTH(order_date)>,
<Feature: orders.WEEKDAY(order_date)>,
<Feature: orders.YEAR(order_date)>]
In addition, the dfs
argument ignore_entities
was renamed to ignore_dataframes
and ignore_variables
was renamed to ignore_columns
. Similarly, if specifying primitive options, all references to entities
should be replaced with dataframes
and references to variables
should be replaced with columns. For example, the primitive option of include_groupby_entities
is now include_groupby_dataframes
and include_variables
is now include_columns
.
The basic call to featuretools.calculate_feature_matrix
remains unchanged if passing in an EntitySet along with a list of features to caluculate. However, users calling calculate_feature_matrix
by passing in a list of entities
and relationships
should note that the entities
argument has been renamed to dataframes
and the values in the dictionary values should now include Woodwork logical types instead of Featuretools Variable
classes.
[23]:
feature_matrix = ft.calculate_feature_matrix(features=features, entityset=es)
feature_matrix
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-featuretools/envs/v1.30.0/lib/python3.9/site-packages/featuretools/computational_backends/feature_set_calculator.py:824: FutureWarning: The provided callable <function min at 0x7f2a103265e0> is currently using SeriesGroupBy.min. In a future version of pandas, the provided callable will be used directly. To keep current behavior pass the string "min" instead.
to_merge = base_frame.groupby(
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-featuretools/envs/v1.30.0/lib/python3.9/site-packages/featuretools/computational_backends/feature_set_calculator.py:824: FutureWarning: The provided callable <function max at 0x7f2a103264c0> is currently using SeriesGroupBy.max. In a future version of pandas, the provided callable will be used directly. To keep current behavior pass the string "max" instead.
to_merge = base_frame.groupby(
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-featuretools/envs/v1.30.0/lib/python3.9/site-packages/featuretools/computational_backends/feature_set_calculator.py:824: FutureWarning: The provided callable <function std at 0x7f2a10326ee0> is currently using SeriesGroupBy.std. In a future version of pandas, the provided callable will be used directly. To keep current behavior pass the string "std" instead.
to_merge = base_frame.groupby(
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-featuretools/envs/v1.30.0/lib/python3.9/site-packages/featuretools/computational_backends/feature_set_calculator.py:824: FutureWarning: The provided callable <function sum at 0x7f2a10322e50> is currently using SeriesGroupBy.sum. In a future version of pandas, the provided callable will be used directly. To keep current behavior pass the string "sum" instead.
to_merge = base_frame.groupby(
/home/docs/checkouts/readthedocs.org/user_builds/feature-labs-inc-featuretools/envs/v1.30.0/lib/python3.9/site-packages/featuretools/computational_backends/feature_set_calculator.py:824: FutureWarning: The provided callable <function mean at 0x7f2a10326dc0> is currently using SeriesGroupBy.mean. In a future version of pandas, the provided callable will be used directly. To keep current behavior pass the string "mean" instead.
to_merge = base_frame.groupby(
[23]:
order_id | item_price | on_sale | orders.COUNT(items) | orders.MAX(items.item_price) | orders.MEAN(items.item_price) | orders.MIN(items.item_price) | orders.PERCENT_TRUE(items.on_sale) | orders.SKEW(items.item_price) | orders.STD(items.item_price) | orders.SUM(items.item_price) | orders.DAY(order_date) | orders.MONTH(order_date) | orders.WEEKDAY(order_date) | orders.YEAR(order_date) | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
id | |||||||||||||||
0 | 0 | 29.95 | False | 1 | 29.95 | 29.950 | 29.95 | 0.0 | NaN | NaN | 29.95 | 2 | 1 | 5 | 2021 |
1 | 1 | 4.99 | True | 2 | 10.25 | 7.620 | 4.99 | 0.5 | NaN | 3.719382 | 15.24 | 3 | 1 | 6 | 2021 |
2 | 1 | 10.25 | False | 2 | 10.25 | 7.620 | 4.99 | 0.5 | NaN | 3.719382 | 15.24 | 3 | 1 | 6 | 2021 |
3 | 2 | 20.50 | True | 2 | 20.50 | 18.245 | 15.99 | 0.5 | NaN | 3.189052 | 36.49 | 4 | 1 | 0 | 2021 |
4 | 2 | 15.99 | False | 2 | 20.50 | 18.245 | 15.99 | 0.5 | NaN | 3.189052 | 36.49 | 4 | 1 | 0 | 2021 |
In addition to the changes in argument names, there are a couple other changes to the returned feature matrix that users should be aware of. First, because of slight differences in the way Woodwork defines column types compared to how the prior Featuretools implementation did, there can be some differences in the features that are generated between old and new versions. The most notable impact is in the way foreign key columns are handled. Previously, Featuretools treated all foreign key
(previously Id
) columns as categorical columns, and would generate appropriate features from these columns. Starting in version 1.0, foreign key columns are not constrained to be categorical, and if they are another type such as Integer
, features will not be generated from these columns. Manually converting foreign key columns to Categorical
as shown above will result in features much closer to those achieved with previous versions.
Also, because Woodwork’s type inference process differs from the previous Featuretools type inference process, an EntitySet may have column types identified differently. This difference in column types could impact the features that are generated. If it is important to have the same set of features, check all of the logical types in the EntitySet dataframes and update them to the expected types if there are columns that have been inferred as unexpected types.
Finally, the feature matrix calculated by Featuretools will now have Woodwork initialized. This means that users can view feature matrix column typing information through the Woodwork namespace as follows.
[24]:
feature_matrix.ww
[24]:
Physical Type | Logical Type | Semantic Tag(s) | |
---|---|---|---|
Column | |||
order_id | int64 | Integer | ['numeric', 'foreign_key'] |
item_price | float64 | Double | ['numeric'] |
on_sale | bool | Boolean | [] |
orders.COUNT(items) | Int64 | IntegerNullable | ['numeric'] |
orders.MAX(items.item_price) | float64 | Double | ['numeric'] |
orders.MEAN(items.item_price) | float64 | Double | ['numeric'] |
orders.MIN(items.item_price) | float64 | Double | ['numeric'] |
orders.PERCENT_TRUE(items.on_sale) | float64 | Double | ['numeric'] |
orders.SKEW(items.item_price) | float64 | Double | ['numeric'] |
orders.STD(items.item_price) | float64 | Double | ['numeric'] |
orders.SUM(items.item_price) | float64 | Double | ['numeric'] |
orders.DAY(order_date) | category | Ordinal: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31] | ['category'] |
orders.MONTH(order_date) | category | Ordinal: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12] | ['category'] |
orders.WEEKDAY(order_date) | category | Ordinal: [0, 1, 2, 3, 4, 5, 6] | ['category'] |
orders.YEAR(order_date) | category | Ordinal: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363, 364, 365, 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 389, 390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418, 419, 420, 421, 422, 423, 424, 425, 426, 427, 428, 429, 430, 431, 432, 433, 434, 435, 436, 437, 438, 439, 440, 441, 442, 443, 444, 445, 446, 447, 448, 449, 450, 451, 452, 453, 454, 455, 456, 457, 458, 459, 460, 461, 462, 463, 464, 465, 466, 467, 468, 469, 470, 471, 472, 473, 474, 475, 476, 477, 478, 479, 480, 481, 482, 483, 484, 485, 486, 487, 488, 489, 490, 491, 492, 493, 494, 495, 496, 497, 498, 499, 500, 501, 502, 503, 504, 505, 506, 507, 508, 509, 510, 511, 512, 513, 514, 515, 516, 517, 518, 519, 520, 521, 522, 523, 524, 525, 526, 527, 528, 529, 530, 531, 532, 533, 534, 535, 536, 537, 538, 539, 540, 541, 542, 543, 544, 545, 546, 547, 548, 549, 550, 551, 552, 553, 554, 555, 556, 557, 558, 559, 560, 561, 562, 563, 564, 565, 566, 567, 568, 569, 570, 571, 572, 573, 574, 575, 576, 577, 578, 579, 580, 581, 582, 583, 584, 585, 586, 587, 588, 589, 590, 591, 592, 593, 594, 595, 596, 597, 598, 599, 600, 601, 602, 603, 604, 605, 606, 607, 608, 609, 610, 611, 612, 613, 614, 615, 616, 617, 618, 619, 620, 621, 622, 623, 624, 625, 626, 627, 628, 629, 630, 631, 632, 633, 634, 635, 636, 637, 638, 639, 640, 641, 642, 643, 644, 645, 646, 647, 648, 649, 650, 651, 652, 653, 654, 655, 656, 657, 658, 659, 660, 661, 662, 663, 664, 665, 666, 667, 668, 669, 670, 671, 672, 673, 674, 675, 676, 677, 678, 679, 680, 681, 682, 683, 684, 685, 686, 687, 688, 689, 690, 691, 692, 693, 694, 695, 696, 697, 698, 699, 700, 701, 702, 703, 704, 705, 706, 707, 708, 709, 710, 711, 712, 713, 714, 715, 716, 717, 718, 719, 720, 721, 722, 723, 724, 725, 726, 727, 728, 729, 730, 731, 732, 733, 734, 735, 736, 737, 738, 739, 740, 741, 742, 743, 744, 745, 746, 747, 748, 749, 750, 751, 752, 753, 754, 755, 756, 757, 758, 759, 760, 761, 762, 763, 764, 765, 766, 767, 768, 769, 770, 771, 772, 773, 774, 775, 776, 777, 778, 779, 780, 781, 782, 783, 784, 785, 786, 787, 788, 789, 790, 791, 792, 793, 794, 795, 796, 797, 798, 799, 800, 801, 802, 803, 804, 805, 806, 807, 808, 809, 810, 811, 812, 813, 814, 815, 816, 817, 818, 819, 820, 821, 822, 823, 824, 825, 826, 827, 828, 829, 830, 831, 832, 833, 834, 835, 836, 837, 838, 839, 840, 841, 842, 843, 844, 845, 846, 847, 848, 849, 850, 851, 852, 853, 854, 855, 856, 857, 858, 859, 860, 861, 862, 863, 864, 865, 866, 867, 868, 869, 870, 871, 872, 873, 874, 875, 876, 877, 878, 879, 880, 881, 882, 883, 884, 885, 886, 887, 888, 889, 890, 891, 892, 893, 894, 895, 896, 897, 898, 899, 900, 901, 902, 903, 904, 905, 906, 907, 908, 909, 910, 911, 912, 913, 914, 915, 916, 917, 918, 919, 920, 921, 922, 923, 924, 925, 926, 927, 928, 929, 930, 931, 932, 933, 934, 935, 936, 937, 938, 939, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 952, 953, 954, 955, 956, 957, 958, 959, 960, 961, 962, 963, 964, 965, 966, 967, 968, 969, 970, 971, 972, 973, 974, 975, 976, 977, 978, 979, 980, 981, 982, 983, 984, 985, 986, 987, 988, 989, 990, 991, 992, 993, 994, 995, 996, 997, 998, 999, 1000, 1001, 1002, 1003, 1004, 1005, 1006, 1007, 1008, 1009, 1010, 1011, 1012, 1013, 1014, 1015, 1016, 1017, 1018, 1019, 1020, 1021, 1022, 1023, 1024, 1025, 1026, 1027, 1028, 1029, 1030, 1031, 1032, 1033, 1034, 1035, 1036, 1037, 1038, 1039, 1040, 1041, 1042, 1043, 1044, 1045, 1046, 1047, 1048, 1049, 1050, 1051, 1052, 1053, 1054, 1055, 1056, 1057, 1058, 1059, 1060, 1061, 1062, 1063, 1064, 1065, 1066, 1067, 1068, 1069, 1070, 1071, 1072, 1073, 1074, 1075, 1076, 1077, 1078, 1079, 1080, 1081, 1082, 1083, 1084, 1085, 1086, 1087, 1088, 1089, 1090, 1091, 1092, 1093, 1094, 1095, 1096, 1097, 1098, 1099, 1100, 1101, 1102, 1103, 1104, 1105, 1106, 1107, 1108, 1109, 1110, 1111, 1112, 1113, 1114, 1115, 1116, 1117, 1118, 1119, 1120, 1121, 1122, 1123, 1124, 1125, 1126, 1127, 1128, 1129, 1130, 1131, 1132, 1133, 1134, 1135, 1136, 1137, 1138, 1139, 1140, 1141, 1142, 1143, 1144, 1145, 1146, 1147, 1148, 1149, 1150, 1151, 1152, 1153, 1154, 1155, 1156, 1157, 1158, 1159, 1160, 1161, 1162, 1163, 1164, 1165, 1166, 1167, 1168, 1169, 1170, 1171, 1172, 1173, 1174, 1175, 1176, 1177, 1178, 1179, 1180, 1181, 1182, 1183, 1184, 1185, 1186, 1187, 1188, 1189, 1190, 1191, 1192, 1193, 1194, 1195, 1196, 1197, 1198, 1199, 1200, 1201, 1202, 1203, 1204, 1205, 1206, 1207, 1208, 1209, 1210, 1211, 1212, 1213, 1214, 1215, 1216, 1217, 1218, 1219, 1220, 1221, 1222, 1223, 1224, 1225, 1226, 1227, 1228, 1229, 1230, 1231, 1232, 1233, 1234, 1235, 1236, 1237, 1238, 1239, 1240, 1241, 1242, 1243, 1244, 1245, 1246, 1247, 1248, 1249, 1250, 1251, 1252, 1253, 1254, 1255, 1256, 1257, 1258, 1259, 1260, 1261, 1262, 1263, 1264, 1265, 1266, 1267, 1268, 1269, 1270, 1271, 1272, 1273, 1274, 1275, 1276, 1277, 1278, 1279, 1280, 1281, 1282, 1283, 1284, 1285, 1286, 1287, 1288, 1289, 1290, 1291, 1292, 1293, 1294, 1295, 1296, 1297, 1298, 1299, 1300, 1301, 1302, 1303, 1304, 1305, 1306, 1307, 1308, 1309, 1310, 1311, 1312, 1313, 1314, 1315, 1316, 1317, 1318, 1319, 1320, 1321, 1322, 1323, 1324, 1325, 1326, 1327, 1328, 1329, 1330, 1331, 1332, 1333, 1334, 1335, 1336, 1337, 1338, 1339, 1340, 1341, 1342, 1343, 1344, 1345, 1346, 1347, 1348, 1349, 1350, 1351, 1352, 1353, 1354, 1355, 1356, 1357, 1358, 1359, 1360, 1361, 1362, 1363, 1364, 1365, 1366, 1367, 1368, 1369, 1370, 1371, 1372, 1373, 1374, 1375, 1376, 1377, 1378, 1379, 1380, 1381, 1382, 1383, 1384, 1385, 1386, 1387, 1388, 1389, 1390, 1391, 1392, 1393, 1394, 1395, 1396, 1397, 1398, 1399, 1400, 1401, 1402, 1403, 1404, 1405, 1406, 1407, 1408, 1409, 1410, 1411, 1412, 1413, 1414, 1415, 1416, 1417, 1418, 1419, 1420, 1421, 1422, 1423, 1424, 1425, 1426, 1427, 1428, 1429, 1430, 1431, 1432, 1433, 1434, 1435, 1436, 1437, 1438, 1439, 1440, 1441, 1442, 1443, 1444, 1445, 1446, 1447, 1448, 1449, 1450, 1451, 1452, 1453, 1454, 1455, 1456, 1457, 1458, 1459, 1460, 1461, 1462, 1463, 1464, 1465, 1466, 1467, 1468, 1469, 1470, 1471, 1472, 1473, 1474, 1475, 1476, 1477, 1478, 1479, 1480, 1481, 1482, 1483, 1484, 1485, 1486, 1487, 1488, 1489, 1490, 1491, 1492, 1493, 1494, 1495, 1496, 1497, 1498, 1499, 1500, 1501, 1502, 1503, 1504, 1505, 1506, 1507, 1508, 1509, 1510, 1511, 1512, 1513, 1514, 1515, 1516, 1517, 1518, 1519, 1520, 1521, 1522, 1523, 1524, 1525, 1526, 1527, 1528, 1529, 1530, 1531, 1532, 1533, 1534, 1535, 1536, 1537, 1538, 1539, 1540, 1541, 1542, 1543, 1544, 1545, 1546, 1547, 1548, 1549, 1550, 1551, 1552, 1553, 1554, 1555, 1556, 1557, 1558, 1559, 1560, 1561, 1562, 1563, 1564, 1565, 1566, 1567, 1568, 1569, 1570, 1571, 1572, 1573, 1574, 1575, 1576, 1577, 1578, 1579, 1580, 1581, 1582, 1583, 1584, 1585, 1586, 1587, 1588, 1589, 1590, 1591, 1592, 1593, 1594, 1595, 1596, 1597, 1598, 1599, 1600, 1601, 1602, 1603, 1604, 1605, 1606, 1607, 1608, 1609, 1610, 1611, 1612, 1613, 1614, 1615, 1616, 1617, 1618, 1619, 1620, 1621, 1622, 1623, 1624, 1625, 1626, 1627, 1628, 1629, 1630, 1631, 1632, 1633, 1634, 1635, 1636, 1637, 1638, 1639, 1640, 1641, 1642, 1643, 1644, 1645, 1646, 1647, 1648, 1649, 1650, 1651, 1652, 1653, 1654, 1655, 1656, 1657, 1658, 1659, 1660, 1661, 1662, 1663, 1664, 1665, 1666, 1667, 1668, 1669, 1670, 1671, 1672, 1673, 1674, 1675, 1676, 1677, 1678, 1679, 1680, 1681, 1682, 1683, 1684, 1685, 1686, 1687, 1688, 1689, 1690, 1691, 1692, 1693, 1694, 1695, 1696, 1697, 1698, 1699, 1700, 1701, 1702, 1703, 1704, 1705, 1706, 1707, 1708, 1709, 1710, 1711, 1712, 1713, 1714, 1715, 1716, 1717, 1718, 1719, 1720, 1721, 1722, 1723, 1724, 1725, 1726, 1727, 1728, 1729, 1730, 1731, 1732, 1733, 1734, 1735, 1736, 1737, 1738, 1739, 1740, 1741, 1742, 1743, 1744, 1745, 1746, 1747, 1748, 1749, 1750, 1751, 1752, 1753, 1754, 1755, 1756, 1757, 1758, 1759, 1760, 1761, 1762, 1763, 1764, 1765, 1766, 1767, 1768, 1769, 1770, 1771, 1772, 1773, 1774, 1775, 1776, 1777, 1778, 1779, 1780, 1781, 1782, 1783, 1784, 1785, 1786, 1787, 1788, 1789, 1790, 1791, 1792, 1793, 1794, 1795, 1796, 1797, 1798, 1799, 1800, 1801, 1802, 1803, 1804, 1805, 1806, 1807, 1808, 1809, 1810, 1811, 1812, 1813, 1814, 1815, 1816, 1817, 1818, 1819, 1820, 1821, 1822, 1823, 1824, 1825, 1826, 1827, 1828, 1829, 1830, 1831, 1832, 1833, 1834, 1835, 1836, 1837, 1838, 1839, 1840, 1841, 1842, 1843, 1844, 1845, 1846, 1847, 1848, 1849, 1850, 1851, 1852, 1853, 1854, 1855, 1856, 1857, 1858, 1859, 1860, 1861, 1862, 1863, 1864, 1865, 1866, 1867, 1868, 1869, 1870, 1871, 1872, 1873, 1874, 1875, 1876, 1877, 1878, 1879, 1880, 1881, 1882, 1883, 1884, 1885, 1886, 1887, 1888, 1889, 1890, 1891, 1892, 1893, 1894, 1895, 1896, 1897, 1898, 1899, 1900, 1901, 1902, 1903, 1904, 1905, 1906, 1907, 1908, 1909, 1910, 1911, 1912, 1913, 1914, 1915, 1916, 1917, 1918, 1919, 1920, 1921, 1922, 1923, 1924, 1925, 1926, 1927, 1928, 1929, 1930, 1931, 1932, 1933, 1934, 1935, 1936, 1937, 1938, 1939, 1940, 1941, 1942, 1943, 1944, 1945, 1946, 1947, 1948, 1949, 1950, 1951, 1952, 1953, 1954, 1955, 1956, 1957, 1958, 1959, 1960, 1961, 1962, 1963, 1964, 1965, 1966, 1967, 1968, 1969, 1970, 1971, 1972, 1973, 1974, 1975, 1976, 1977, 1978, 1979, 1980, 1981, 1982, 1983, 1984, 1985, 1986, 1987, 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025, 2026, 2027, 2028, 2029, 2030, 2031, 2032, 2033, 2034, 2035, 2036, 2037, 2038, 2039, 2040, 2041, 2042, 2043, 2044, 2045, 2046, 2047, 2048, 2049, 2050, 2051, 2052, 2053, 2054, 2055, 2056, 2057, 2058, 2059, 2060, 2061, 2062, 2063, 2064, 2065, 2066, 2067, 2068, 2069, 2070, 2071, 2072, 2073, 2074, 2075, 2076, 2077, 2078, 2079, 2080, 2081, 2082, 2083, 2084, 2085, 2086, 2087, 2088, 2089, 2090, 2091, 2092, 2093, 2094, 2095, 2096, 2097, 2098, 2099, 2100, 2101, 2102, 2103, 2104, 2105, 2106, 2107, 2108, 2109, 2110, 2111, 2112, 2113, 2114, 2115, 2116, 2117, 2118, 2119, 2120, 2121, 2122, 2123, 2124, 2125, 2126, 2127, 2128, 2129, 2130, 2131, 2132, 2133, 2134, 2135, 2136, 2137, 2138, 2139, 2140, 2141, 2142, 2143, 2144, 2145, 2146, 2147, 2148, 2149, 2150, 2151, 2152, 2153, 2154, 2155, 2156, 2157, 2158, 2159, 2160, 2161, 2162, 2163, 2164, 2165, 2166, 2167, 2168, 2169, 2170, 2171, 2172, 2173, 2174, 2175, 2176, 2177, 2178, 2179, 2180, 2181, 2182, 2183, 2184, 2185, 2186, 2187, 2188, 2189, 2190, 2191, 2192, 2193, 2194, 2195, 2196, 2197, 2198, 2199, 2200, 2201, 2202, 2203, 2204, 2205, 2206, 2207, 2208, 2209, 2210, 2211, 2212, 2213, 2214, 2215, 2216, 2217, 2218, 2219, 2220, 2221, 2222, 2223, 2224, 2225, 2226, 2227, 2228, 2229, 2230, 2231, 2232, 2233, 2234, 2235, 2236, 2237, 2238, 2239, 2240, 2241, 2242, 2243, 2244, 2245, 2246, 2247, 2248, 2249, 2250, 2251, 2252, 2253, 2254, 2255, 2256, 2257, 2258, 2259, 2260, 2261, 2262, 2263, 2264, 2265, 2266, 2267, 2268, 2269, 2270, 2271, 2272, 2273, 2274, 2275, 2276, 2277, 2278, 2279, 2280, 2281, 2282, 2283, 2284, 2285, 2286, 2287, 2288, 2289, 2290, 2291, 2292, 2293, 2294, 2295, 2296, 2297, 2298, 2299, 2300, 2301, 2302, 2303, 2304, 2305, 2306, 2307, 2308, 2309, 2310, 2311, 2312, 2313, 2314, 2315, 2316, 2317, 2318, 2319, 2320, 2321, 2322, 2323, 2324, 2325, 2326, 2327, 2328, 2329, 2330, 2331, 2332, 2333, 2334, 2335, 2336, 2337, 2338, 2339, 2340, 2341, 2342, 2343, 2344, 2345, 2346, 2347, 2348, 2349, 2350, 2351, 2352, 2353, 2354, 2355, 2356, 2357, 2358, 2359, 2360, 2361, 2362, 2363, 2364, 2365, 2366, 2367, 2368, 2369, 2370, 2371, 2372, 2373, 2374, 2375, 2376, 2377, 2378, 2379, 2380, 2381, 2382, 2383, 2384, 2385, 2386, 2387, 2388, 2389, 2390, 2391, 2392, 2393, 2394, 2395, 2396, 2397, 2398, 2399, 2400, 2401, 2402, 2403, 2404, 2405, 2406, 2407, 2408, 2409, 2410, 2411, 2412, 2413, 2414, 2415, 2416, 2417, 2418, 2419, 2420, 2421, 2422, 2423, 2424, 2425, 2426, 2427, 2428, 2429, 2430, 2431, 2432, 2433, 2434, 2435, 2436, 2437, 2438, 2439, 2440, 2441, 2442, 2443, 2444, 2445, 2446, 2447, 2448, 2449, 2450, 2451, 2452, 2453, 2454, 2455, 2456, 2457, 2458, 2459, 2460, 2461, 2462, 2463, 2464, 2465, 2466, 2467, 2468, 2469, 2470, 2471, 2472, 2473, 2474, 2475, 2476, 2477, 2478, 2479, 2480, 2481, 2482, 2483, 2484, 2485, 2486, 2487, 2488, 2489, 2490, 2491, 2492, 2493, 2494, 2495, 2496, 2497, 2498, 2499, 2500, 2501, 2502, 2503, 2504, 2505, 2506, 2507, 2508, 2509, 2510, 2511, 2512, 2513, 2514, 2515, 2516, 2517, 2518, 2519, 2520, 2521, 2522, 2523, 2524, 2525, 2526, 2527, 2528, 2529, 2530, 2531, 2532, 2533, 2534, 2535, 2536, 2537, 2538, 2539, 2540, 2541, 2542, 2543, 2544, 2545, 2546, 2547, 2548, 2549, 2550, 2551, 2552, 2553, 2554, 2555, 2556, 2557, 2558, 2559, 2560, 2561, 2562, 2563, 2564, 2565, 2566, 2567, 2568, 2569, 2570, 2571, 2572, 2573, 2574, 2575, 2576, 2577, 2578, 2579, 2580, 2581, 2582, 2583, 2584, 2585, 2586, 2587, 2588, 2589, 2590, 2591, 2592, 2593, 2594, 2595, 2596, 2597, 2598, 2599, 2600, 2601, 2602, 2603, 2604, 2605, 2606, 2607, 2608, 2609, 2610, 2611, 2612, 2613, 2614, 2615, 2616, 2617, 2618, 2619, 2620, 2621, 2622, 2623, 2624, 2625, 2626, 2627, 2628, 2629, 2630, 2631, 2632, 2633, 2634, 2635, 2636, 2637, 2638, 2639, 2640, 2641, 2642, 2643, 2644, 2645, 2646, 2647, 2648, 2649, 2650, 2651, 2652, 2653, 2654, 2655, 2656, 2657, 2658, 2659, 2660, 2661, 2662, 2663, 2664, 2665, 2666, 2667, 2668, 2669, 2670, 2671, 2672, 2673, 2674, 2675, 2676, 2677, 2678, 2679, 2680, 2681, 2682, 2683, 2684, 2685, 2686, 2687, 2688, 2689, 2690, 2691, 2692, 2693, 2694, 2695, 2696, 2697, 2698, 2699, 2700, 2701, 2702, 2703, 2704, 2705, 2706, 2707, 2708, 2709, 2710, 2711, 2712, 2713, 2714, 2715, 2716, 2717, 2718, 2719, 2720, 2721, 2722, 2723, 2724, 2725, 2726, 2727, 2728, 2729, 2730, 2731, 2732, 2733, 2734, 2735, 2736, 2737, 2738, 2739, 2740, 2741, 2742, 2743, 2744, 2745, 2746, 2747, 2748, 2749, 2750, 2751, 2752, 2753, 2754, 2755, 2756, 2757, 2758, 2759, 2760, 2761, 2762, 2763, 2764, 2765, 2766, 2767, 2768, 2769, 2770, 2771, 2772, 2773, 2774, 2775, 2776, 2777, 2778, 2779, 2780, 2781, 2782, 2783, 2784, 2785, 2786, 2787, 2788, 2789, 2790, 2791, 2792, 2793, 2794, 2795, 2796, 2797, 2798, 2799, 2800, 2801, 2802, 2803, 2804, 2805, 2806, 2807, 2808, 2809, 2810, 2811, 2812, 2813, 2814, 2815, 2816, 2817, 2818, 2819, 2820, 2821, 2822, 2823, 2824, 2825, 2826, 2827, 2828, 2829, 2830, 2831, 2832, 2833, 2834, 2835, 2836, 2837, 2838, 2839, 2840, 2841, 2842, 2843, 2844, 2845, 2846, 2847, 2848, 2849, 2850, 2851, 2852, 2853, 2854, 2855, 2856, 2857, 2858, 2859, 2860, 2861, 2862, 2863, 2864, 2865, 2866, 2867, 2868, 2869, 2870, 2871, 2872, 2873, 2874, 2875, 2876, 2877, 2878, 2879, 2880, 2881, 2882, 2883, 2884, 2885, 2886, 2887, 2888, 2889, 2890, 2891, 2892, 2893, 2894, 2895, 2896, 2897, 2898, 2899, 2900, 2901, 2902, 2903, 2904, 2905, 2906, 2907, 2908, 2909, 2910, 2911, 2912, 2913, 2914, 2915, 2916, 2917, 2918, 2919, 2920, 2921, 2922, 2923, 2924, 2925, 2926, 2927, 2928, 2929, 2930, 2931, 2932, 2933, 2934, 2935, 2936, 2937, 2938, 2939, 2940, 2941, 2942, 2943, 2944, 2945, 2946, 2947, 2948, 2949, 2950, 2951, 2952, 2953, 2954, 2955, 2956, 2957, 2958, 2959, 2960, 2961, 2962, 2963, 2964, 2965, 2966, 2967, 2968, 2969, 2970, 2971, 2972, 2973, 2974, 2975, 2976, 2977, 2978, 2979, 2980, 2981, 2982, 2983, 2984, 2985, 2986, 2987, 2988, 2989, 2990, 2991, 2992, 2993, 2994, 2995, 2996, 2997, 2998, 2999] | ['category'] |
Featuretools now labels features by whether they were originally in the dataframes, or whether they were created by Featuretools. This information is stored in the Woodwork origin
attribute for the column. Columns that were in the original data will be labeled with base
and features that were created by Featuretools will be labeled with engineered
.
As a demonstration of how to access this information, let’s compare two features in the feature matrix: item_price
and orders.MEAN(items.item_price)
. item_price
was present in the original data, and orders.MEAN(items.item_price)
was created by Featuretools.
[25]:
feature_matrix.ww["item_price"].ww.origin
[25]:
'base'
[26]:
feature_matrix.ww["orders.MEAN(items.item_price)"].ww.origin
[26]:
'engineered'
Other changes#
In addition to the changes outlined above, there are several other smaller changes in Featuretools 1.0 of which existing users should be aware.
Column ordering of an dataframe in an EntitySet might be different than it was before. Previously, Featuretools would reorder the columns such that the index column would always be the first column in the dataframe. This behavior has been removed, and the index column is no longer guaranteed to be the first column in the dataframe. Now the index column will remain in the position it was when the dataframe was added to the EntitySet.
For
LatLong
columns, older versions of Featuretools would replace singlenan
values in the columns with a tuple(nan, nan)
. This is no longer the case, and singlenan
values will now remain in theLatLong
column. Based on the behavior in Woodwork, any values of(nan, nan)
in aLatLong
column will be replaced with a singlenan
value.Since Featuretools no longer defines
Variable
objects with relationships between them, thefeaturetools.variable_types.graph_variable_types
function has been removed.The
featuretools.variable_types.list_variable_types
utility function has been removed and replaced with two corresponding Woodwork functions:woodwork.list_logical_types
andwoodwork.list_semantic_tags
. Starting in Featuretools 1.0, the Woodwork utility functions should be used to obtain information on the logical types and semantic tags that can be applied to dataframe columns.