{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Handling Time\n", "\n", "\n", "When performing feature engineering with temporal data, carefully selecting the data that is used for any calculation is paramount. By annotating dataframes with a Woodwork **time index** column and providing a **cutoff time** during feature calculation, Featuretools will automatically filter out any data after the cutoff time before running any calculations.\n", "\n", "## What is the Time Index?\n", "\n", "\n", "The time index is the column in the data that specifies when the data in each row became known. For example, let's examine a table of customer transactions:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "nbsphinx": "hidden" }, "outputs": [], "source": [ "import pandas as pd\n", "pd.options.display.max_columns = 200" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import featuretools as ft\n", "\n", "es = ft.demo.load_mock_customer(return_entityset=True, random_seed=0)\n", "es['transactions'].head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this table, there is one row for every transaction and a ``transaction_time`` column that specifies when the transaction took place. This means that ``transaction_time`` is the time index because it indicates when the information in each row became known and available for feature calculations. For now, ignore the ``_ft_last_time`` column. That is a featuretools-generated column that will be discussed later on.\n", "\n", "However, not every datetime column is a time index. Consider the ``customers`` dataframe:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "es['customers']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here, we have two time columns, ``join_date`` and ``birthday``. While either column might be useful for making features, the ``join_date`` should be used as the time index because it indicates when that customer first became available in the dataset." ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ ".. important::\n", "\n", " The **time index** is defined as the first time that any information from a row can be used. If a cutoff time is specified when calculating features, rows that have a later value for the time index are automatically ignored." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# What is the Cutoff Time?\n", "The **cutoff_time** specifies the last point in time that a row’s data can be used for a feature calculation. Any data after this point in time will be filtered out before calculating features.\n", "\n", "For example, let's consider a dataset of timestamped customer transactions, where we want to predict whether customers ``1``, ``2`` and ``3`` will spend $500 between ``04:00`` on January 1 and the end of the day. When building features for this prediction problem, we need to ensure that no data after ``04:00`` is used in our calculations.\n", "\n", "\"retail" ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ "We pass the cutoff time to :func:`featuretools.dfs` or :func:`featuretools.calculate_feature_matrix` using the ``cutoff_time`` argument like this:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fm, features = ft.dfs(entityset=es,\n", " target_dataframe_name='customers',\n", " cutoff_time=pd.Timestamp(\"2014-1-1 04:00\"),\n", " instance_ids=[1,2,3],\n", " cutoff_time_in_index=True)\n", "fm" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Even though the entityset contains the complete transaction history for each customer, only data with a time index up to and including the cutoff time was used to calculate the features above.\n", "\n", "## Using a Cutoff Time DataFrame\n", "\n", "\n", "Oftentimes, the training examples for machine learning will come from different points in time. To specify a unique cutoff time for each row of the resulting feature matrix, we can pass a dataframe which includes one column for the instance id and another column for the corresponding cutoff time. These columns can be in any order, but they must be named properly. The column with the instance ids must either be named ``instance_id`` or have the same name as the target dataframe ``index``. The column with the cutoff time values must either be named ``time`` or have the same name as the target dataframe ``time_index``.\n", "\n", "The column names for the instance ids and the cutoff time values should be unambiguous. Passing a dataframe that contains both a column with the same name as the target dataframe ``index`` and a column named ``instance_id`` will result in an error. Similarly, if the cutoff time dataframe contains both a column with the same name as the target dataframe ``time_index`` and a column named ``time`` an error will be raised." ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ ".. note::\n", "\n", " Only the columns corresponding to the instance ids and the cutoff times are used to calculate features. Any additional columns passed through are appended to the resulting feature matrix. This is typically used to pass through machine learning labels to ensure that they stay aligned with the feature matrix." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "cutoff_times = pd.DataFrame()\n", "cutoff_times['customer_id'] = [1, 2, 3, 1]\n", "cutoff_times['time'] = pd.to_datetime(['2014-1-1 04:00',\n", " '2014-1-1 05:00',\n", " '2014-1-1 06:00',\n", " '2014-1-1 08:00'])\n", "cutoff_times['label'] = [True, True, False, True]\n", "cutoff_times\n", "fm, features = ft.dfs(entityset=es,\n", " target_dataframe_name='customers',\n", " cutoff_time=cutoff_times,\n", " cutoff_time_in_index=True)\n", "fm" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can now see that every row of the feature matrix is calculated at the corresponding time in the cutoff time dataframe. Because we calculate each row at a different time, it is possible to have a repeat customer. In this case, we calculated the feature vector for customer 1 at both ``04:00`` and ``08:00``.\n", "\n", "Training Window\n", "---------------\n", "\n", "By default, all data up to and including the cutoff time is used. We can restrict the amount of historical data that is selected for calculations using a \"training window.\"\n", "\n", "Here's an example of using a two hour training window:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "window_fm, window_features = ft.dfs(entityset=es,\n", " target_dataframe_name=\"customers\",\n", " cutoff_time=cutoff_times,\n", " cutoff_time_in_index=True,\n", " training_window=\"2 hour\")\n", "\n", "window_fm" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see that that the counts for the same feature are lower after we shorten the training window:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fm[[\"COUNT(transactions)\"]]\n", "\n", "window_fm[[\"COUNT(transactions)\"]]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setting a Last Time Index\n", "\n", "The training window in Featuretools limits the amount of past data that can be used while calculating a particular feature vector. A row in the dataframe is filtered out if the value of its time index is either before or after the training window. This works for dataframes where a row occurs at a single point in time. However, a row can sometimes exist for a duration.\n", "\n", "For example, a customer's session has multiple transactions which can happen at different points in time. If we are trying to count the number of sessions a user has in a given time period, we often want to count all the sessions that had *any* transaction during the training window. To accomplish this, we need to not only know when a session starts, but also when it ends. The last time that an instance appears in the data is stored in the `_ft_last_time` column on the dataframe. We can compare the time index and the last time index of the ``sessions`` dataframe above:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "last_time_index_col = es['sessions'].ww.metadata.get('last_time_index')\n", "es['sessions'][['session_start', last_time_index_col]].head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Featuretools can automatically add last time indexes to every DataFrame in an ``Entityset`` by running ``EntitySet.add_last_time_indexes()``. When using a training window, if a `last_time_index has` been set, Featuretools will check to see if the `last_time_index` is after the start of the training window. That, combined with the cutoff time, allows DFS to discover which data is relevant for a given training window.\n", "\n", "\n", "## Excluding data at cutoff times" ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ "The ``cutoff_time`` is the last point in time where data can be used for feature\n", "calculation. If you don't want to use the data at the cutoff time in feature\n", "calculation, you can exclude that data by setting ``include_cutoff_time`` to\n", "``False`` in :func:`featuretools.dfs` or :func:`featuretools.calculate_feature_matrix`.\n", "If you set it to ``True`` (the default behavior), data from the cutoff time point\n", "will be used." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Setting ``include_cutoff_time`` to ``False`` also impacts how data at the edges\n", "of training windows are included or excluded. Take this slice of data as an example:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df = es['transactions']\n", "df[df[\"session_id\"] == 1].head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Looking at the data, transactions occur every 65 seconds. To check how ``include_cutoff_time``\n", "effects training windows, we can calculate features at the time of a transaction\n", "while using a 65 second training window. This creates a training window with a\n", "transaction at both endpoints of the window. For this example, we'll find the sum\n", "of all transactions for session id 1 that are in the training window." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from featuretools.primitives import Sum\n", "\n", "sum_log = ft.Feature(\n", " es['transactions'].ww['amount'],\n", " parent_dataframe_name='sessions',\n", " primitive=Sum,\n", ")\n", "cutoff_time = pd.DataFrame({\n", " 'session_id': [1],\n", " 'time': ['2014-01-01 00:04:20'],\n", "}).astype({'time': 'datetime64[ns]'})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With ``include_cutoff_time=True``, the oldest point in the training window\n", "(``2014-01-01 00:03:15``) is excluded and the cutoff time point is included. This\n", "means only transaction 371 is in the training window, so the sum of all transaction\n", "amounts is 31.54" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Case1. include_cutoff_time = True\n", "actual = ft.calculate_feature_matrix(\n", " features=[sum_log],\n", " entityset=es,\n", " cutoff_time=cutoff_time,\n", " cutoff_time_in_index=True,\n", " training_window='65 seconds',\n", " include_cutoff_time=True,\n", ")\n", "actual" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Whereas with ``include_cutoff_time=False``, the oldest point in the window is\n", "included and the cutoff time point is excluded. So in this case transaction 116\n", "is included and transaction 371 is exluded, and the sum is 78.92\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Case2. include_cutoff_time = False\n", "actual = ft.calculate_feature_matrix(\n", " features=[sum_log],\n", " entityset=es,\n", " cutoff_time=cutoff_time,\n", " cutoff_time_in_index=True,\n", " training_window='65 seconds',\n", " include_cutoff_time=False,\n", ")\n", "actual" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Approximating Features by Rounding Cutoff Times\n", "-----------------------------------------------\n", "\n", "For each unique cutoff time, Featuretools must perform operations to select the data that’s valid for computations. If there are a large number of unique cutoff times relative to the number of instances for which we are calculating features, the time spent filtering data can add up. By reducing the number of unique cutoff times, we minimize the overhead from searching for and extracting data for feature calculations.\n", "\n", "One way to decrease the number of unique cutoff times is to round cutoff times to an earlier point in time. An earlier cutoff time is always valid for predictive modeling — it just means we’re not using some of the data we could potentially use while calculating that feature. So, we gain computational speed by losing a small amount of information.\n", "\n", "To understand when an approximation is useful, consider calculating features for a model to predict fraudulent credit card transactions. In this case, an important feature might be, \"the average transaction amount for this card in the past\". While this value can change every time there is a new transaction, updating it less frequently might not impact accuracy." ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ ".. note::\n", "\n", " The bank BBVA used approximation when building a predictive model for credit card fraud using Featuretools. For more details, see the \"Real-time deployment considerations\" section of the `white paper `_ describing the work involved.\n" ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ "The frequency of approximation is controlled using the ``approximate`` parameter to :func:`featuretools.dfs` or :func:`featuretools.calculate_feature_matrix`. For example, the following code would approximate aggregation features at 1 day intervals::" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " fm = ft.calculate_feature_matrix(features=features,\n", " entityset=es_transactions,\n", " cutoff_time=ct_transactions,\n", " approximate=\"1 day\")\n", "\n", "In this computation, features that can be approximated will be calculated at 1 day intervals, while features that cannot be approximated (e.g \"where did this transaction occur?\") will be calculated at the exact cutoff time.\n", "\n", "\n", "## Secondary Time Index\n", "\n", "It is sometimes the case that information in a dataset is updated or added after a row has been created. This means that certain columns may actually become known after the time index for a row. Rather than drop those columns to avoid leaking information, we can create a secondary time index to indicate when those columns become known." ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ "The :func:`Flights ` entityset is a good example of a dataset where column values in a row become known at different times. Each trip is recorded in the ``trip_logs`` dataframe, and has many times associated with it." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "nbsphinx": "hidden" }, "outputs": [], "source": [ "import urllib.request as urllib2\n", "opener = urllib2.build_opener()\n", "opener.addheaders = [('Testing', 'True')]\n", "urllib2.install_opener(opener)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "es_flight = ft.demo.load_flight(nrows=100)\n", "es_flight\n", "es_flight['trip_logs'].head(3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For every trip log, the time index is ``date_scheduled``, which is when the airline decided on the scheduled departure and arrival times, as well as what route will be flown. We don't know the rest of the information about the actual departure/arrival times and the details of any delay at this time. However, it is possible to know everything about how a trip went after it has arrived, so we can use that information at any time after the flight lands.\n", "\n", "Using a secondary time index, we can indicate to Featuretools which columns in our flight logs are known at the time the flight is scheduled, plus which are known at the time the flight lands.\n", "\n", "\"flight\n", "\n", "In Featuretools, when adding the dataframe to the ``EntitySet``, we set the secondary time index to be the arrival time like this:\n", "\n", " es = ft.EntitySet('Flight Data')\n", " arr_time_columns = ['arr_delay', 'dep_delay', 'carrier_delay', 'weather_delay',\n", " 'national_airspace_delay', 'security_delay',\n", " 'late_aircraft_delay', 'canceled', 'diverted',\n", " 'taxi_in', 'taxi_out', 'air_time', 'dep_time']\n", "\n", " es.add_dataframe(\n", " dataframe_name='trip_logs',\n", " dataframe=data,\n", " index='trip_log_id',\n", " make_index=True,\n", " time_index='date_scheduled',\n", " secondary_time_index={'arr_time': arr_time_columns})\n", "\n", "By setting a secondary time index, we can still use the delay information from a row, but only when it becomes known." ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ ".. hint::\n", "\n", " It's often a good idea to use a secondary time index if your entityset has inline labels. If you know when the label would be valid for use, it's possible to automatically create very predictive features using historical labels." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Flight Predictions\n", "\n", "Let's make some features at varying times using the flight example described above. Trip ``14`` is a flight from CLT to PHX on January 31, 2017 and trip ``92`` is a flight from PIT to DFW on January 1. We can set any cutoff time before the flight is scheduled to depart, emulating how we would make the prediction at that point in time.\n", "\n", "We set two cutoff times for trip ``14`` at two different times: one which is more than a month before the flight and another which is only 5 days before. For trip ``92``, we'll only set one cutoff time, three days before it is scheduled to leave.\n", "\n", "\"flight\n", "\n", "Our cutoff time dataframe looks like this:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ct_flight = pd.DataFrame()\n", "ct_flight['trip_log_id'] = [14, 14, 92]\n", "ct_flight['time'] = pd.to_datetime(['2016-12-28',\n", " '2017-1-25',\n", " '2016-12-28'])\n", "ct_flight['label'] = [True, True, False]\n", "ct_flight" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, let's calculate the feature matrix:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fm, features = ft.dfs(entityset=es_flight,\n", " target_dataframe_name='trip_logs',\n", " cutoff_time=ct_flight,\n", " cutoff_time_in_index=True,\n", " agg_primitives=[\"max\"],\n", " trans_primitives=[\"month\"],)\n", "fm[['flights.origin', 'flights.dest', 'label', 'flights.MAX(trip_logs.arr_delay)', 'MONTH(scheduled_dep_time)']]\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's understand the output:\n", "\n", "1. A row was made for every id-time pair in ``ct_flight``, which is returned as the index of the feature matrix.\n", "\n", "2. The output was sorted by cutoff time. Because of the sorting, it's often helpful to pass in a label with the cutoff time dataframe so that it will remain sorted in the same fashion as the feature matrix. Any additional columns beyond ``id`` and ``cutoff_time`` will not be used for making features.\n", "\n", "3. The column ``flights.MAX(trip_logs.arr_delay)`` is not always defined. It can only have any real values when there are historical flights to aggregate. Notice that, for trip ``14``, there wasn't any historical data when we made the feature a month in advance, but there **were** flights to aggregate when we shortened it to 5 days. These are powerful features that are often excluded in manual processes because of how hard they are to make.\n", "\n", "\n", "Creating and Flattening a Feature Tensor\n", "----------------------------------------" ] }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/restructuredtext" }, "source": [ "The :func:`featuretools.make_temporal_cutoffs` function generates a series of equally spaced cutoff times from a given set of cutoff times and instance ids." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This function can be paired with DFS to create and flatten a feature tensor rather than making multiple feature matrices at different delays.\n", "\n", "The function\n", "takes in the the following parameters:\n", "\n", " * ``instance_ids (list, pd.Series, or np.ndarray)``: A list of instances.\n", " * ``cutoffs (list, pd.Series, or np.ndarray)``: An associated list of cutoff times.\n", " * ``window_size (str or pandas.DateOffset)``: The amount of time between each cutoff time in the created time series.\n", " * ``start (datetime.datetime or pd.Timestamp)``: The first cutoff time in the created time series.\n", " * ``num_windows (int)``: The number of cutoff times to create in the created time series.\n", "\n", "Only two of the three options ``window_size``, ``start``, and ``num_windows`` need to be specified to uniquely determine an equally-spaced set of cutoff times at which to compute each instance.\n", "\n", "If your cutoff times are the ones used above:\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "cutoff_times" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then passing in ``window_size='1h'`` and ``num_windows=2`` makes one row an hour over the last two hours to produce the following new dataframe. The result can be directly passed into DFS to make features at the different time points." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "temporal_cutoffs = ft.make_temporal_cutoffs(cutoff_times['customer_id'],\n", " cutoff_times['time'],\n", " window_size='1h',\n", " num_windows=2)\n", "temporal_cutoffs\n", "fm, features = ft.dfs(entityset=es,\n", " target_dataframe_name='customers',\n", " cutoff_time=temporal_cutoffs,\n", " cutoff_time_in_index=True)\n", "fm" ] } ], "metadata": { "celltoolbar": "Raw Cell Format", "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.2" } }, "nbformat": 4, "nbformat_minor": 5 }