NOTICE
The upcoming release of Featuretools 1.0.0 contains several breaking changes. Users are encouraged to test this version prior to release by installing from GitHub:
pip install https://github.com/alteryx/featuretools/archive/woodwork-integration.zip
For details on migrating to the new version, refer to Transitioning to Featuretools Version 1.0. Please report any issues in the Featuretools GitHub repo or by messaging in Alteryx Open Source Slack.
An EntitySet is a collection of dataframes and the relationships between them. They are useful for preparing raw, structured datasets for feature engineering. While many functions in Featuretools take dataframes and relationships as separate arguments, it is recommended to create an EntitySet, so you can more easily manipulate your data as needed.
EntitySet
dataframes
relationships
Below we have two tables of data (represented as Pandas DataFrames) related to customer transactions. The first is a merge of transactions, sessions, and customers so that the result looks like something you might see in a log file:
[1]:
import featuretools as ft data = ft.demo.load_mock_customer() transactions_df = data["transactions"].merge(data["sessions"]).merge(data["customers"]) transactions_df.sample(10)
And the second dataframe is a list of products involved in those transactions.
[2]:
products_df = data["products"] products_df
First, we initialize an EntitySet. If you’d like to give it a name, you can optionally provide an id to the constructor.
id
[3]:
es = ft.EntitySet(id="customer_data")
To get started, we add the transactions dataframe to the EntitySet. In the call to add_dataframe, we specify three important parameters:
add_dataframe
The index parameter specifies the column that uniquely identifies rows in the dataframe.
index
The time_index parameter tells Featuretools when the data was created.
time_index
The logical_types parameter indicates that “product_id” should be interpreted as a Categorical column, even though it is just an integer in the underlying data.
logical_types
[4]:
from woodwork.logical_types import Categorical, PostalCode es = es.add_dataframe( dataframe_name="transactions", dataframe=transactions_df, index="transaction_id", time_index="transaction_time", logical_types={ "product_id": Categorical, "zip_code": PostalCode, }, ) es
Entityset: customer_data DataFrames: transactions [Rows: 500, Columns: 11] Relationships: No relationships
Note
You can also display your EntitySet structure graphically by calling EntitySet.plot().
EntitySet.plot()
This method associates each column in the dataframe to a Woodwork logical type. Each logical type can have an associated standard semantic tag that helps define the column data type. If you don’t specify the logical type for a column, it gets inferred based on the underlying data. The logical types and semantic tags are listed in the schema of the dataframe. For more information on working with logical types and semantic tags, take a look at the Woodwork documention.
[5]:
es["transactions"].ww.schema
Now, we can do that same thing with our products dataframe.
[6]:
es = es.add_dataframe( dataframe_name="products", dataframe=products_df, index="product_id") es
Entityset: customer_data DataFrames: transactions [Rows: 500, Columns: 11] products [Rows: 5, Columns: 2] Relationships: No relationships
With two dataframes in our EntitySet, we can add a relationship between them.
We want to relate these two dataframes by the columns called “product_id” in each dataframe. Each product has multiple transactions associated with it, so it is called the parent dataframe, while the transactions dataframe is known as the child dataframe. When specifying relationships, we need four parameters: the parent dataframe name, the parent column name, the child dataframe name, and the child column name. Note that each relationship must denote a one-to-many relationship rather than a relationship which is one-to-one or many-to-many.
[7]:
es = es.add_relationship("products", "product_id", "transactions", "product_id") es
Entityset: customer_data DataFrames: transactions [Rows: 500, Columns: 11] products [Rows: 5, Columns: 2] Relationships: transactions.product_id -> products.product_id
Now, we see the relationship has been added to our EntitySet.
When working with raw data, it is common to have sufficient information to justify the creation of new dataframes. In order to create a new dataframe and relationship for sessions, we “normalize” the transaction dataframe.
[8]:
es = es.normalize_dataframe( base_dataframe_name="transactions", new_dataframe_name="sessions", index="session_id", make_time_index="session_start", additional_columns=[ "device", "customer_id", "zip_code", "session_start", "join_date", ], ) es
Entityset: customer_data DataFrames: transactions [Rows: 500, Columns: 6] products [Rows: 5, Columns: 2] sessions [Rows: 35, Columns: 6] Relationships: transactions.product_id -> products.product_id transactions.session_id -> sessions.session_id
Looking at the output above, we see this method did two operations:
It created a new dataframe called “sessions” based on the “session_id” and “session_start” columns in “transactions”
It added a relationship connecting “transactions” and “sessions”
If we look at the schema from the transactions dataframe and the new sessions dataframe, we see two more operations that were performed automatically:
[9]:
[10]:
es["sessions"].ww.schema
It removed “device”, “customer_id”, “zip_code” and “join_date” from “transactions” and created a new columns in the sessions dataframe. This reduces redundant information as the those properties of a session don’t change between transactions.
It copied and marked “session_start” as a time index column into the new sessions dataframe to indicate the beginning of a session. If the base dataframe has a time index and make_time_index is not set, normalize_dataframe will create a time index for the new dataframe. In this case it would create a new time index called “first_transactions_time” using the time of the first transaction of each session. If we don’t want this time index to be created, we can set make_time_index=False.
make_time_index
normalize_dataframe
make_time_index=False
If we look at the dataframes, we can see what normalize_dataframe did to the actual data.
[11]:
es["sessions"].head(5)
[12]:
es["transactions"].head(5)
To finish preparing this dataset, create a “customers” dataframe using the same method call.
[13]:
es = es.normalize_dataframe( base_dataframe_name="sessions", new_dataframe_name="customers", index="customer_id", make_time_index="join_date", additional_columns=["zip_code", "join_date"], ) es
Entityset: customer_data DataFrames: transactions [Rows: 500, Columns: 6] products [Rows: 5, Columns: 2] sessions [Rows: 35, Columns: 4] customers [Rows: 5, Columns: 3] Relationships: transactions.product_id -> products.product_id transactions.session_id -> sessions.session_id sessions.customer_id -> customers.customer_id
Finally, we are ready to use this EntitySet with any functionality within Featuretools. For example, let’s build a feature matrix for each product in our dataset.
[14]:
feature_matrix, feature_defs = ft.dfs(entityset=es, target_dataframe_name="products") feature_matrix
5 rows × 25 columns
As we can see, the features from DFS use the relational structure of our EntitySet. Therefore it is important to think carefully about the dataframes that we create.
EntitySets can also be created using Dask dataframes or Koalas dataframes. For more information refer to Using Dask EntitySets (BETA) and Using Koalas EntitySets (BETA).