Featuretools groups features into four general types:
In Featuretools, each feature is defined as a combination of other features. At the lowest level are
IdentityFeature features which are equal to the value of a single variable.
Most of the time, identity features will be defined transparently for you, such as in the transform feature example below. They may also be defined explicitly:
In : time_feature = ft.Feature(es["transactions"]["transaction_time"]) In : time_feature Out: <Feature: transaction_time>
Direct features are used to “inherit” feature values from a parent to a child entity. Suppose each event is associated with a single instance of the entity products.
This entity has metadata about different products, such as brand, price, etc. We can pull the brand of the product into a feature of the event entity by including the event entity as an argument to
In this case,
Feature is an alias for
In : brand = ft.Feature(es["products"]["brand"], entity=es["transactions"]) In : brand Out: <Feature: products.brand>
Transform features take one or more features on an
Entity and create a single new feature for that same entity. For example, we may want to take a fine-grained “timestamp” feature and convert it into the hour of the day in which it occurred.
In : from featuretools.primitives import Hour In : ft.Feature(time_feature, primitive=Hour) Out: <Feature: HOUR(transaction_time)>
Using algebraic and boolean operations, transform features can combine other features into arbitrary expressions. For example, to determine if a given event event happened in the afternoon, we can write:
In : hour_feature = ft.Feature(time_feature, primitive=Hour) In : after_twelve = hour_feature > 12 In : after_twelve Out: <Feature: HOUR(transaction_time) > 12> In : at_twelve = hour_feature == 12 In : before_five = hour_feature <= 17 In : is_afternoon = after_twelve & before_five In : is_afternoon Out: <Feature: AND(HOUR(transaction_time) > 12, HOUR(transaction_time) <= 17)>
Aggregation features are used to create features for a parent entity by summarizing data from a child entity. For example, we can create a
Count feature which counts the total number of events for each customer:
In : from featuretools.primitives import Count In : total_events = ft.Feature(es["transactions"]["transaction_id"], parent_entity=es["customers"], primitive=Count) In : fm = ft.calculate_feature_matrix([total_events], es) In : fm.head() Out: COUNT(transactions) customer_id 5 79 4 109 1 126 3 93 2 93
For users who have written aggregations in SQL, this concept will be familiar. One key difference in featuretools is that
GROUP BY and
JOIN are implicit. Since the parent and child entities are specified, featuretools can infer how to group the child entity and then join the resulting aggregation back to the parent entity.
Often times, we only want to aggregate using a certain amount of previous data. For example, we might only want to count events from the past 30 days. In this case, we can provide the
In : total_events_last_30_days = ft.Feature(es["transactions"]["transaction_id"], ....: parent_entity=es["customers"], ....: use_previous="30 days", ....: primitive=Count) ....: In : fm = ft.calculate_feature_matrix([total_events_last_30_days], es) In : fm.head() Out: COUNT(transactions, Last 30 Days) customer_id 5 0.0 4 0.0 1 0.0 3 0.0 2 0.0
Unlike with cumulative transform features, the
use_previous parameter here is evaluated relative to instances of the parent entity, not the child entity. The above feature translates roughly to the following: “For each customer, count the events which occurred in the 30 days preceding the customer’s timestamp.”
Find the list of the supported aggregation features here.
When defining aggregation or cumulative transform features, we can provide a
where parameter to filter the instances we are aggregating over. Using the
is_afternoon feature from earlier, we can count the total number of events which occurred in the afternoon:
In : afternoon_events = ft.Feature(es["transactions"]["transaction_id"], ....: parent_entity=es["customers"], ....: where=is_afternoon, ....: primitive=Count).rename("afternoon_events") ....: In : fm = ft.calculate_feature_matrix([afternoon_events], es) In : fm.head() Out: afternoon_events customer_id 5 0.0 4 0.0 1 0.0 3 0.0 2 0.0
The where argument can be any previously-defined boolean feature. Only instances for which the where feature is True are included in the final calculation.
Aggregations of Direct Feature¶
Composing multiple feature types is an extremely powerful abstraction that Featuretools makes simple. For instance, we can aggregate direct features on a child entity from a different parent entity. For example, to calculate the most common brand a customer interacted with:
In : from featuretools.primitives import Mode In : brand = ft.Feature(es["products"]["brand"], entity=es["transactions"]) In : favorite_brand = ft.Feature(brand, parent_entity=es["customers"], primitive=Mode) In : fm = ft.calculate_feature_matrix([favorite_brand], es) In : fm.head() Out: MODE(transactions.products.brand) customer_id 5 B 4 B 1 B 3 B 2 B
Side note: Feature equality overrides default equality¶
Because we can check if two features are equal (or a feature is equal to a value), we override Python’s equals (==) operator. This means to check if two feature objects are equal (instead of their computed values in the feature matrix), we need to compare their hashes:
In : hour_feature.hash() == hour_feature.hash() Out: True In : hour_feature.hash() != hour_feature.hash() Out: False
dictionaries and sets use equality underneath, so those keys need to be hashes as well
In : myset = set() In : myset.add(hour_feature.hash()) In : hour_feature.hash() in myset Out: True In : mydict = dict() In : mydict[hour_feature.hash()] = hour_feature In : hour_feature.hash() in mydict Out: True