NOTICE

The upcoming release of Featuretools 1.0.0 contains several breaking changes. Users are encouraged to test this version prior to release:

pip install featuretools==1.0.0rc1

For details on migrating to the new version, refer to Transitioning to Featuretools Version 1.0. Please report any issues in the Featuretools GitHub repo or by messaging in Alteryx Open Source Slack.


Advanced Custom Primitives Guide

[1]:
from featuretools.primitives import TransformPrimitive
from featuretools.tests.testing_utils import make_ecommerce_entityset
from featuretools.variable_types import DatetimeTimeIndex, NaturalLanguage, Numeric
import featuretools as ft
import numpy as np
import re

Primitives With Additional Arguments

Some features require more advanced calculations than others. Advanced features usually entail additional arguments to help output the desired value. With custom primitives, you can use primitive arguments to help you create advanced features.

String Count Example

In this example, you will learn how to make custom primitives that take in additional arguments. You will create a primitive to count the number of times a specific string value occurs inside a text.

First, derive a new transform primitive class using TransformPrimitive as a base. The primitive will take in a text column as the input and return a numeric column as the output, so set the input type as NaturalLanguage and the return type as Numeric. The specific string value is the additional argument, so define it as a keyword argument inside __init__(). Then, override get_function() to return a primitive function that will calculate the feature.

[2]:
class StringCount(TransformPrimitive):
    '''Count the number of times the string value occurs.'''
    name = 'string_count'
    input_types = [NaturalLanguage]
    return_type = Numeric

    def __init__(self, string=None):
        self.string = string

    def get_function(self):
        def string_count(column):
            assert self.string is not None, "string to count needs to be defined"
            # this is a naive implementation used for clarity
            counts = [text.lower().count(self.string) for text in column]
            return counts

        return string_count

Now you have a primitive that is reusable for different string values. For example, you can create features based on the number of times the word “the” appears in a text. Create an instance of the primitive where the string value is “the” and pass the primitive into DFS to generate the features. The feature name will automatically reflect the string value of the primitive.

[3]:
es = make_ecommerce_entityset()

feature_matrix, features = ft.dfs(
    entityset=es,
    target_entity="sessions",
    agg_primitives=["sum", "mean", "std"],
    trans_primitives=[StringCount(string="the")],
)

feature_matrix[[
    'STD(log.STRING_COUNT(comments, string=the))',
    'SUM(log.STRING_COUNT(comments, string=the))',
    'MEAN(log.STRING_COUNT(comments, string=the))',
]]
[3]:
STD(log.STRING_COUNT(comments, string=the)) SUM(log.STRING_COUNT(comments, string=the)) MEAN(log.STRING_COUNT(comments, string=the))
id
0 47.124304 209 41.80
1 36.509131 109 27.25
2 NaN 29 29.00
3 49.497475 70 35.00
4 0.000000 0 0.00
5 1.414214 4 2.00

Features with Multiple Outputs

Some calculations output more than a single value. With custom primitives, you can make the most of these calculations by creating a feature for each output value.

Case Count Example

In this example, you will learn how to make custom primitives that output multiple features. You will create a primitive that outputs the count of upper case and lower case letters of a text.

First, derive a new transform primitive class using TransformPrimitive as a base. The primitive will take in a text column as the input and return two numeric columns as the output, so set the input type as NaturalLanguage, the return type as Numeric, and number_output_features to two. Then, override get_function() to return a primitive function that will calculate the feature and return a list of columns.

[4]:
class CaseCount(TransformPrimitive):
    '''Return the count of upper case and lower case letters of a text.'''
    name = 'case_count'
    input_types = [NaturalLanguage]
    return_type = Numeric
    number_output_features = 2

    def get_function(self):
        def case_count(array):
            # this is a naive implementation used for clarity
            upper = np.array([len(re.findall('[A-Z]', i)) for i in array])
            lower = np.array([len(re.findall('[a-z]', i)) for i in array])
            return upper, lower

        return case_count

Now you have a primitive that outputs two columns. One column contains the count for the upper case letters. The other column contains the count for the lower case letters. Pass the primitive into DFS to generate features. By default, the feature name will reflect the index of the output.

[5]:
feature_matrix, features = ft.dfs(
    entityset=es,
    target_entity="sessions",
    agg_primitives=[],
    trans_primitives=[CaseCount],
)

feature_matrix[[
    'customers.CASE_COUNT(favorite_quote)[0]',
    'customers.CASE_COUNT(favorite_quote)[1]',
]]
[5]:
customers.CASE_COUNT(favorite_quote)[0] customers.CASE_COUNT(favorite_quote)[1]
id
0 1 44
1 1 44
2 1 44
3 1 41
4 1 41
5 1 57

Custom Naming for Multiple Outputs

When you create a primitive that outputs multiple features, you can also define custom naming for each of those features.

Hourly Sine and Cosine Example

In this example, you will learn how to apply custom naming for multiple outputs. You will create a primitive that outputs the sine and cosine of the hour.

First, derive a new transform primitive class using TransformPrimitive as a base. The primitive will take in the time index as the input and return two numeric columns as the output, so set the input type as DatetimeTimeIndex, the return type as Numeric, and number_output_features to two. Then, override get_function() to return a primitive function that will calculate the feature and return a list of columns. Also, override generate_names() to return a list of the feature names that you define.

[6]:
class HourlySineAndCosine(TransformPrimitive):
    '''Returns the sine and cosine of the hour.'''
    name = 'hourly_sine_and_cosine'
    input_types = [DatetimeTimeIndex]
    return_type = Numeric
    number_output_features = 2

    def get_function(self):
        def hourly_sine_and_cosine(column):
            sine = np.sin(column.dt.hour)
            cosine = np.cos(column.dt.hour)
            return sine, cosine

        return hourly_sine_and_cosine

    def generate_names(self, base_feature_names):
        name = self.generate_name(base_feature_names)
        return f'{name}[sine]', f'{name}[cosine]'

Now you have a primitive that outputs two columns. One column contains the sine of the hour. The other column contains the cosine of the hour. Pass the primitive into DFS to generate features. The feature name will reflect the custom naming you defined.

[7]:
feature_matrix, features = ft.dfs(
    entityset=es,
    target_entity="log",
    agg_primitives=[],
    trans_primitives=[HourlySineAndCosine],
)

feature_matrix.head()[[
    'HOURLY_SINE_AND_COSINE(datetime)[sine]',
    'HOURLY_SINE_AND_COSINE(datetime)[cosine]',
]]
[7]:
HOURLY_SINE_AND_COSINE(datetime)[sine] HOURLY_SINE_AND_COSINE(datetime)[cosine]
id
0 -0.544021 -0.839072
1 -0.544021 -0.839072
2 -0.544021 -0.839072
3 -0.544021 -0.839072
4 -0.544021 -0.839072