Deployment

Deployment of machine learning models requires repeating feature engineering steps on new data. In some cases, these steps need to be performed in near real-time. Featuretools has capabilities to ease the deployment of feature engineering.

Saving Features

First, let’s build some generate some training and test data in the same format. We use a random seed to generate different data for the test.

Note

Features saved in one version of Featuretools are not guaranteed to load in another. This means the features might need to be re-created after upgrading Featuretools.

In [1]: import featuretools as ft

In [2]: es_train = ft.demo.load_mock_customer(return_entityset=True)

In [3]: es_test = ft.demo.load_mock_customer(return_entityset=True, random_seed=33)

Now let’s build some features definitions using DFS. Because we have categorical features, we also encode them with one hot encoding based on the values in the training data.

In [4]: feature_matrix, feature_defs = ft.dfs(entityset=es_train,
   ...:                                       target_entity="customers")
   ...: 

In [5]: feature_matrix_enc, features_enc = ft.encode_features(feature_matrix, feature_defs)

In [6]: feature_matrix_enc
Out[6]: 
             zip_code = 60091  zip_code = 13244  zip_code is unknown  COUNT(sessions)  MODE(sessions.device) = mobile  MODE(sessions.device) = desktop  MODE(sessions.device) is unknown  NUM_UNIQUE(sessions.device)  COUNT(transactions)  MAX(transactions.amount)  MEAN(transactions.amount)  MIN(transactions.amount)  MODE(transactions.product_id) = 4  MODE(transactions.product_id) = 5  MODE(transactions.product_id) = 2  MODE(transactions.product_id) = 1  MODE(transactions.product_id) is unknown  NUM_UNIQUE(transactions.product_id)  SKEW(transactions.amount)  STD(transactions.amount)  SUM(transactions.amount)  DAY(date_of_birth) = 18  DAY(date_of_birth) = 28  DAY(date_of_birth) = 21  DAY(date_of_birth) = 15  DAY(date_of_birth) is unknown  DAY(join_date) = 17  DAY(join_date) = 15  DAY(join_date) = 13  DAY(join_date) = 8  DAY(join_date) is unknown  MONTH(date_of_birth) = 8  MONTH(date_of_birth) = 7  MONTH(date_of_birth) = 11  MONTH(date_of_birth) is unknown  MONTH(join_date) = 4  MONTH(join_date) = 8  MONTH(join_date) = 7  MONTH(join_date) is unknown  WEEKDAY(date_of_birth) = 0  WEEKDAY(date_of_birth) = 5  WEEKDAY(date_of_birth) = 4  WEEKDAY(date_of_birth) = 1  WEEKDAY(date_of_birth) is unknown  WEEKDAY(join_date) = 6  WEEKDAY(join_date) = 5  WEEKDAY(join_date) = 4  WEEKDAY(join_date) is unknown  YEAR(date_of_birth) = 2006  YEAR(date_of_birth) = 2003  YEAR(date_of_birth) = 1994  YEAR(date_of_birth) = 1986  YEAR(date_of_birth) = 1984  YEAR(date_of_birth) is unknown  YEAR(join_date) = 2011  YEAR(join_date) = 2012  YEAR(join_date) = 2010  YEAR(join_date) is unknown  MAX(sessions.COUNT(transactions))  MAX(sessions.MEAN(transactions.amount))  MAX(sessions.MIN(transactions.amount))  MAX(sessions.NUM_UNIQUE(transactions.product_id))  MAX(sessions.SKEW(transactions.amount))  MAX(sessions.STD(transactions.amount))  MAX(sessions.SUM(transactions.amount))  MEAN(sessions.COUNT(transactions))  MEAN(sessions.MAX(transactions.amount))  MEAN(sessions.MEAN(transactions.amount))  MEAN(sessions.MIN(transactions.amount))  MEAN(sessions.NUM_UNIQUE(transactions.product_id))  MEAN(sessions.SKEW(transactions.amount))  MEAN(sessions.STD(transactions.amount))  MEAN(sessions.SUM(transactions.amount))  MIN(sessions.COUNT(transactions))  MIN(sessions.MAX(transactions.amount))  MIN(sessions.MEAN(transactions.amount))  MIN(sessions.NUM_UNIQUE(transactions.product_id))  MIN(sessions.SKEW(transactions.amount))  MIN(sessions.STD(transactions.amount))  MIN(sessions.SUM(transactions.amount))  MODE(sessions.DAY(session_start)) = 1  MODE(sessions.DAY(session_start)) is unknown  MODE(sessions.MODE(transactions.product_id)) = 3  MODE(sessions.MODE(transactions.product_id)) = 1  MODE(sessions.MODE(transactions.product_id)) = 4  MODE(sessions.MODE(transactions.product_id)) is unknown  MODE(sessions.MONTH(session_start)) = 1  MODE(sessions.MONTH(session_start)) is unknown  MODE(sessions.WEEKDAY(session_start)) = 2  MODE(sessions.WEEKDAY(session_start)) is unknown  MODE(sessions.YEAR(session_start)) = 2014  MODE(sessions.YEAR(session_start)) is unknown  NUM_UNIQUE(sessions.DAY(session_start))  NUM_UNIQUE(sessions.MODE(transactions.product_id))  NUM_UNIQUE(sessions.MONTH(session_start))  NUM_UNIQUE(sessions.WEEKDAY(session_start))  NUM_UNIQUE(sessions.YEAR(session_start))  SKEW(sessions.COUNT(transactions))  SKEW(sessions.MAX(transactions.amount))  SKEW(sessions.MEAN(transactions.amount))  SKEW(sessions.MIN(transactions.amount))  SKEW(sessions.NUM_UNIQUE(transactions.product_id))  SKEW(sessions.STD(transactions.amount))  SKEW(sessions.SUM(transactions.amount))  STD(sessions.COUNT(transactions))  STD(sessions.MAX(transactions.amount))  STD(sessions.MEAN(transactions.amount))  STD(sessions.MIN(transactions.amount))  STD(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.SKEW(transactions.amount))  STD(sessions.SUM(transactions.amount))  SUM(sessions.MAX(transactions.amount))  SUM(sessions.MEAN(transactions.amount))  SUM(sessions.MIN(transactions.amount))  SUM(sessions.NUM_UNIQUE(transactions.product_id))  SUM(sessions.SKEW(transactions.amount))  SUM(sessions.STD(transactions.amount))  MODE(transactions.sessions.customer_id) = 5  MODE(transactions.sessions.customer_id) = 4  MODE(transactions.sessions.customer_id) = 3  MODE(transactions.sessions.customer_id) = 2  MODE(transactions.sessions.customer_id) = 1  MODE(transactions.sessions.customer_id) is unknown  MODE(transactions.sessions.device) = mobile  MODE(transactions.sessions.device) = desktop  MODE(transactions.sessions.device) is unknown  NUM_UNIQUE(transactions.sessions.customer_id)  NUM_UNIQUE(transactions.sessions.device)
customer_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
5                        True             False                False                6                            True                            False                             False                            3                   79                    149.02                  80.375443                      7.55                              False                               True                              False                              False                                     False                                    5                  -0.025941                 44.095630                   6349.66                    False                     True                    False                    False                          False                 True                False                False               False                      False                     False                      True                      False                            False                 False                 False                  True                        False                       False                        True                       False                       False                              False                   False                    True                   False                          False                       False                       False                       False                       False                        True                           False                   False                   False                    True                       False                                 18                                94.481667                                   20.65                                                  5                                 0.602209                               51.149250                                 1700.67                           13.166667                               139.960000                                 78.705187                                14.415000                                           5.000000                                   0.002397                                43.312326                              1058.276667                                  8                                  128.51                                66.666667                                                  5                                -0.539060                               36.734681                                  543.18                                   True                                         False                                              True                                             False                                             False                                              False                                           True                                           False                                       True                                             False                                       True                                          False                                        1                                                  5                                           1                                            1                                         1                           -0.317685                                -0.333796                                  0.335175                                -0.470410                                           0.000000                                  0.204548                                 0.472342                           3.600926                                7.928001                                11.007471                                4.961414                                           0.000000                                 0.415426                              402.775486                                  839.76                               472.231119                                   86.49                                                 30                                 0.014384                              259.873954                                         True                                        False                                        False                                        False                                        False                                              False                                          True                                         False                                          False                                              1                                         3
4                        True             False                False                8                            True                            False                             False                            3                  109                    149.95                  80.070459                      5.73                              False                              False                               True                              False                                     False                                    5                  -0.036348                 45.068765                   8727.68                    False                    False                    False                     True                          False                False                False                False                True                      False                      True                     False                      False                            False                  True                 False                 False                        False                       False                       False                       False                        True                              False                   False                   False                    True                          False                        True                       False                       False                       False                       False                           False                    True                   False                   False                       False                                 18                               110.450000                                   54.83                                                  5                                 0.382868                               54.293903                                 1351.46                           13.625000                               144.748750                                 81.207189                                16.438750                                           4.625000                                   0.000346                                44.515729                              1090.960000                                 10                                  139.20                                70.638182                                                  4                                -0.711744                               29.026424                                  771.68                                   True                                         False                                             False                                              True                                             False                                              False                                           True                                           False                                       True                                             False                                       True                                          False                                        1                                                  5                                           1                                            1                                         1                            0.282488                                 0.027256                                  1.980948                                 2.103510                                          -0.644061                                 -1.065663                                -0.391805                           3.335416                                3.514421                                13.027258                               16.960575                                           0.517549                                 0.387884                              235.992478                                 1157.99                               649.657515                                  131.51                                                 37                                 0.002764                              356.125829                                        False                                         True                                        False                                        False                                        False                                              False                                          True                                         False                                          False                                              1                                         3
1                        True             False                False                8                            True                            False                             False                            3                  126                    139.43                  71.631905                      5.81                               True                              False                              False                              False                                     False                                    5                   0.019698                 40.442059                   9025.62                     True                    False                    False                    False                          False                 True                False                False               False                      False                     False                      True                      False                            False                  True                 False                 False                        False                        True                       False                       False                       False                              False                    True                   False                   False                          False                       False                       False                        True                       False                       False                           False                    True                   False                   False                       False                                 25                                88.755625                                   26.36                                                  5                                 0.640252                               46.905665                                 1613.93                           15.750000                               132.246250                                 72.774140                                 9.823750                                           5.000000                                  -0.059515                                39.093244                              1128.202500                                 12                                  118.90                                50.623125                                                  5                                -1.038434                               30.450261                                  809.97                                   True                                         False                                             False                                             False                                              True                                              False                                           True                                           False                                       True                                             False                                       True                                          False                                        1                                                  4                                           1                                            1                                         1                            1.946018                                -0.780493                                 -0.424949                                 2.440005                                           0.000000                                 -0.312355                                 0.778170                           4.062019                                7.322191                                13.759314                                6.954507                                           0.000000                                 0.589386                              279.510713                                 1057.97                               582.193117                                   78.59                                                 40                                -0.476122                              312.745952                                        False                                        False                                        False                                        False                                         True                                              False                                          True                                         False                                          False                                              1                                         3
3                       False              True                False                6                           False                             True                             False                            3                   93                    149.15                  67.060430                      5.89                              False                              False                              False                               True                                     False                                    5                   0.418230                 43.683296                   6236.62                    False                    False                     True                    False                          False                False                False                 True               False                      False                     False                     False                       True                            False                 False                  True                 False                        False                       False                       False                        True                       False                              False                   False                    True                   False                          False                       False                        True                       False                       False                       False                           False                    True                   False                   False                       False                                 18                                82.109444                                   20.06                                                  5                                 0.854976                               50.110120                                 1477.97                           15.500000                               141.271667                                 67.539577                                11.035000                                           4.833333                                   0.381014                                42.883316                              1039.436667                                 11                                  126.74                                55.579412                                                  4                                -0.289466                               35.704680                                  889.21                                   True                                         False                                             False                                              True                                             False                                              False                                           True                                           False                                       True                                             False                                       True                                          False                                        1                                                  4                                           1                                            1                                         1                           -1.507217                                -0.941078                                  0.678544                                 1.000771                                          -2.449490                                 -0.245703                                 2.246479                           2.428992                               10.724241                                11.174282                                5.424407                                           0.408248                                 0.429374                              219.021420                                  847.63                               405.237462                                   66.21                                                 29                                 2.286086                              257.299895                                        False                                        False                                         True                                        False                                        False                                              False                                         False                                          True                                          False                                              1                                         3
2                       False              True                False                7                           False                             True                             False                            3                   93                    146.81                  77.422366                      8.73                               True                              False                              False                              False                                     False                                    5                   0.098259                 37.705178                   7200.28                     True                    False                    False                    False                          False                False                 True                False               False                      False                      True                     False                      False                            False                  True                 False                 False                        False                        True                       False                       False                       False                              False                    True                   False                   False                          False                       False                       False                       False                        True                       False                           False                   False                    True                   False                       False                                 18                                96.581000                                   56.46                                                  5                                 0.755711                               47.935920                                 1320.64                           13.285714                               133.090000                                 78.415122                                22.085714                                           5.000000                                  -0.039663                                36.957218                              1028.611429                                  8                                  100.04                                61.910000                                                  5                                -0.763603                               27.839228                                  634.84                                   True                                         False                                              True                                             False                                             False                                              False                                           True                                           False                                       True                                             False                                       True                                          False                                        1                                                  4                                           1                                            1                                         1                           -0.303276                                -1.539467                                  0.235296                                 2.154929                                           0.000000                                  0.013087                                -0.440929                           3.450328                               17.221593                                11.477071                               15.874374                                           0.000000                                 0.509798                              251.609234                                  931.63                               548.905851                                  154.60                                                 35                                -0.277640                              258.700528                                        False                                        False                                        False                                         True                                        False                                              False                                         False                                          True                                          False                                              1                                         3

Now, we can use featuretools.save_features() to save a list features to a json file

In [7]: ft.save_features(features_enc, "feature_definitions.json")

Calculating Feature Matrix for New Data

We can use featuretools.load_features() to read in a list of saved features to calculate for our new entity set.

In [8]: saved_features = ft.load_features('feature_definitions.json')

After we load the features back in, we can calculate the feature matrix.

In [9]: feature_matrix = ft.calculate_feature_matrix(saved_features, es_test)

In [10]: feature_matrix
Out[10]: 
             zip_code = 60091  zip_code = 13244  zip_code is unknown  COUNT(sessions)  MODE(sessions.device) = mobile  MODE(sessions.device) = desktop  MODE(sessions.device) is unknown  NUM_UNIQUE(sessions.device)  COUNT(transactions)  MAX(transactions.amount)  MEAN(transactions.amount)  MIN(transactions.amount)  MODE(transactions.product_id) = 4  MODE(transactions.product_id) = 5  MODE(transactions.product_id) = 2  MODE(transactions.product_id) = 1  MODE(transactions.product_id) is unknown  NUM_UNIQUE(transactions.product_id)  SKEW(transactions.amount)  STD(transactions.amount)  SUM(transactions.amount)  DAY(date_of_birth) = 18  DAY(date_of_birth) = 28  DAY(date_of_birth) = 21  DAY(date_of_birth) = 15  DAY(date_of_birth) is unknown  DAY(join_date) = 17  DAY(join_date) = 15  DAY(join_date) = 13  DAY(join_date) = 8  DAY(join_date) is unknown  MONTH(date_of_birth) = 8  MONTH(date_of_birth) = 7  MONTH(date_of_birth) = 11  MONTH(date_of_birth) is unknown  MONTH(join_date) = 4  MONTH(join_date) = 8  MONTH(join_date) = 7  MONTH(join_date) is unknown  WEEKDAY(date_of_birth) = 0  WEEKDAY(date_of_birth) = 5  WEEKDAY(date_of_birth) = 4  WEEKDAY(date_of_birth) = 1  WEEKDAY(date_of_birth) is unknown  WEEKDAY(join_date) = 6  WEEKDAY(join_date) = 5  WEEKDAY(join_date) = 4  WEEKDAY(join_date) is unknown  YEAR(date_of_birth) = 2006  YEAR(date_of_birth) = 2003  YEAR(date_of_birth) = 1994  YEAR(date_of_birth) = 1986  YEAR(date_of_birth) = 1984  YEAR(date_of_birth) is unknown  YEAR(join_date) = 2011  YEAR(join_date) = 2012  YEAR(join_date) = 2010  YEAR(join_date) is unknown  MAX(sessions.COUNT(transactions))  MAX(sessions.MEAN(transactions.amount))  MAX(sessions.MIN(transactions.amount))  MAX(sessions.NUM_UNIQUE(transactions.product_id))  MAX(sessions.SKEW(transactions.amount))  MAX(sessions.STD(transactions.amount))  MAX(sessions.SUM(transactions.amount))  MEAN(sessions.COUNT(transactions))  MEAN(sessions.MAX(transactions.amount))  MEAN(sessions.MEAN(transactions.amount))  MEAN(sessions.MIN(transactions.amount))  MEAN(sessions.NUM_UNIQUE(transactions.product_id))  MEAN(sessions.SKEW(transactions.amount))  MEAN(sessions.STD(transactions.amount))  MEAN(sessions.SUM(transactions.amount))  MIN(sessions.COUNT(transactions))  MIN(sessions.MAX(transactions.amount))  MIN(sessions.MEAN(transactions.amount))  MIN(sessions.NUM_UNIQUE(transactions.product_id))  MIN(sessions.SKEW(transactions.amount))  MIN(sessions.STD(transactions.amount))  MIN(sessions.SUM(transactions.amount))  MODE(sessions.DAY(session_start)) = 1  MODE(sessions.DAY(session_start)) is unknown  MODE(sessions.MODE(transactions.product_id)) = 3  MODE(sessions.MODE(transactions.product_id)) = 1  MODE(sessions.MODE(transactions.product_id)) = 4  MODE(sessions.MODE(transactions.product_id)) is unknown  MODE(sessions.MONTH(session_start)) = 1  MODE(sessions.MONTH(session_start)) is unknown  MODE(sessions.WEEKDAY(session_start)) = 2  MODE(sessions.WEEKDAY(session_start)) is unknown  MODE(sessions.YEAR(session_start)) = 2014  MODE(sessions.YEAR(session_start)) is unknown  NUM_UNIQUE(sessions.DAY(session_start))  NUM_UNIQUE(sessions.MODE(transactions.product_id))  NUM_UNIQUE(sessions.MONTH(session_start))  NUM_UNIQUE(sessions.WEEKDAY(session_start))  NUM_UNIQUE(sessions.YEAR(session_start))  SKEW(sessions.COUNT(transactions))  SKEW(sessions.MAX(transactions.amount))  SKEW(sessions.MEAN(transactions.amount))  SKEW(sessions.MIN(transactions.amount))  SKEW(sessions.NUM_UNIQUE(transactions.product_id))  SKEW(sessions.STD(transactions.amount))  SKEW(sessions.SUM(transactions.amount))  STD(sessions.COUNT(transactions))  STD(sessions.MAX(transactions.amount))  STD(sessions.MEAN(transactions.amount))  STD(sessions.MIN(transactions.amount))  STD(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.SKEW(transactions.amount))  STD(sessions.SUM(transactions.amount))  SUM(sessions.MAX(transactions.amount))  SUM(sessions.MEAN(transactions.amount))  SUM(sessions.MIN(transactions.amount))  SUM(sessions.NUM_UNIQUE(transactions.product_id))  SUM(sessions.SKEW(transactions.amount))  SUM(sessions.STD(transactions.amount))  MODE(transactions.sessions.customer_id) = 5  MODE(transactions.sessions.customer_id) = 4  MODE(transactions.sessions.customer_id) = 3  MODE(transactions.sessions.customer_id) = 2  MODE(transactions.sessions.customer_id) = 1  MODE(transactions.sessions.customer_id) is unknown  MODE(transactions.sessions.device) = mobile  MODE(transactions.sessions.device) = desktop  MODE(transactions.sessions.device) is unknown  NUM_UNIQUE(transactions.sessions.customer_id)  NUM_UNIQUE(transactions.sessions.device)
customer_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
1                        True             False                False                6                           False                             True                             False                            3                   73                    147.64                  79.128904                      6.47                               True                              False                              False                              False                                     False                                    5                  -0.173042                 41.998795                   5776.41                    False                    False                    False                    False                           True                False                False                False               False                       True                     False                     False                      False                             True                 False                 False                 False                         True                       False                       False                       False                       False                               True                    True                   False                   False                          False                       False                       False                       False                       False                       False                            True                   False                   False                   False                        True                                 19                               136.672500                                  124.58                                                  5                                 0.164928                               51.192518                                 1418.99                           12.166667                               139.013333                                 87.486612                                33.153333                                           4.166667                                  -0.257693                                36.177337                               962.735000                                  4                                  119.86                                64.238824                                                  2                                -0.614442                                9.176046                                  546.69                                   True                                         False                                             False                                             False                                              True                                              False                                           True                                           False                                       True                                             False                                       True                                          False                                        1                                                  4                                           1                                            1                                         1                           -0.243198                                -1.165955                                  1.796000                                 2.290693                                          -1.206607                                 -1.336611                                 0.130163                           5.741661                               11.742343                                25.801702                               45.471977                                           1.329160                                 0.326628                              327.331994                                  834.08                               524.919674                                  198.92                                                 25                                -1.546156                              217.064024                                        False                                        False                                        False                                        False                                         True                                              False                                          True                                         False                                          False                                              1                                         3
4                       False              True                False                9                           False                             True                             False                            3                  126                    147.55                  80.781190                      6.19                              False                              False                              False                              False                                      True                                    5                  -0.179621                 36.523849                  10178.43                    False                    False                    False                    False                           True                False                False                False               False                       True                     False                     False                       True                            False                 False                 False                  True                        False                       False                       False                        True                       False                              False                   False                   False                    True                          False                       False                       False                       False                       False                       False                            True                   False                   False                   False                        True                                 21                               104.565000                                   60.29                                                  5                                 0.417250                               41.627134                                 1650.65                           14.000000                               131.211111                                 81.540322                                21.453333                                           4.777778                                  -0.199690                                35.499735                              1130.936667                                  8                                  118.59                                69.665000                                                  4                                -0.624344                               22.026552                                  557.32                                   True                                         False                                             False                                              True                                             False                                              False                                           True                                           False                                       True                                             False                                       True                                          False                                        1                                                  5                                           1                                            1                                         1                           -0.086578                                 0.230847                                  1.078619                                 1.490781                                          -1.619848                                 -1.743267                                -0.385392                           4.272002                               11.457114                                11.875823                               17.851716                                           0.440959                                 0.324894                              333.923377                                 1180.90                               733.862898                                  193.08                                                 43                                -1.797214                              319.497611                                        False                                         True                                        False                                        False                                        False                                              False                                         False                                          True                                          False                                              1                                         3
3                        True             False                False                5                            True                            False                             False                            2                   64                    148.09                  82.171094                     10.66                              False                               True                              False                              False                                     False                                    5                  -0.081427                 42.416322                   5258.95                    False                    False                    False                    False                           True                False                False                False               False                       True                     False                     False                      False                             True                 False                 False                 False                         True                       False                       False                       False                       False                               True                   False                    True                   False                          False                        True                       False                       False                       False                       False                           False                   False                   False                    True                       False                                 19                                92.004000                                   38.69                                                  5                                 0.785376                               46.402508                                 1691.27                           12.800000                               143.160000                                 81.478110                                21.738000                                           4.600000                                   0.070612                                43.083442                              1051.790000                                  8                                  133.75                                66.563000                                                  4                                -0.608419                               37.945636                                  653.11                                   True                                         False                                             False                                              True                                             False                                              False                                           True                                           False                                       True                                             False                                       True                                          False                                        1                                                  2                                           1                                            1                                         1                            0.570300                                -1.598919                                 -0.730554                                 1.010658                                          -0.608581                                 -0.627238                                 0.727969                           4.868265                                5.596950                                10.012657                               10.829338                                           0.547723                                 0.530094                              450.318768                                  715.80                               407.390549                                  108.69                                                 23                                 0.353061                              215.417211                                        False                                        False                                         True                                        False                                        False                                              False                                          True                                         False                                          False                                              1                                         2
2                       False              True                False                8                           False                             True                             False                            3                  129                    148.34                  76.571085                      8.00                              False                              False                              False                               True                                     False                                    5                   0.040395                 39.913352                   9877.67                    False                    False                    False                    False                           True                False                False                False               False                       True                     False                     False                      False                             True                 False                 False                 False                         True                       False                       False                       False                       False                               True                    True                   False                   False                          False                       False                       False                       False                       False                       False                            True                   False                   False                    True                       False                                 21                                87.669412                                   40.88                                                  5                                 0.454842                               43.950396                                 1690.97                           16.125000                               137.602500                                 76.964367                                17.001250                                           4.875000                                  -0.010253                                39.477166                              1234.708750                                  8                                  120.06                                52.288421                                                  4                                -0.522578                               33.618728                                  619.93                                   True                                         False                                             False                                              True                                             False                                              False                                           True                                           False                                       True                                             False                                       True                                          False                                        1                                                  4                                           1                                            1                                         1                           -1.158217                                -0.964539                                 -2.048650                                 1.981423                                          -2.828427                                 -0.403715                                -0.576079                           4.086126                                9.491736                                10.757169                               10.517928                                           0.353553                                 0.305345                              346.152626                                 1100.82                               615.714934                                  136.01                                                 39                                -0.082021                              315.817331                                        False                                        False                                        False                                         True                                        False                                              False                                         False                                          True                                          False                                              1                                         3
5                        True             False                False                7                           False                             True                             False                            3                  108                    149.53                  83.506852                      6.35                              False                              False                              False                              False                                      True                                    5                  -0.107234                 37.514054                   9018.74                    False                     True                    False                    False                          False                False                False                False               False                       True                     False                     False                      False                             True                 False                 False                 False                         True                       False                       False                       False                        True                              False                   False                   False                    True                          False                       False                       False                       False                       False                       False                            True                   False                   False                   False                        True                                 19                                95.488000                                   34.93                                                  5                                 0.538086                               44.978308                                 1705.25                           15.428571                               142.497143                                 83.471845                                19.642857                                           4.714286                                  -0.085018                                37.362181                              1288.391429                                 11                                  133.41                                73.576471                                                  4                                -0.618136                               26.724118                                  921.73                                   True                                         False                                             False                                              True                                             False                                              False                                           True                                           False                                       True                                             False                                       True                                          False                                        1                                                  4                                           1                                            1                                         1                           -0.566615                                -0.308063                                  0.246613                                 0.013052                                          -1.229634                                 -0.405840                                 0.215180                           2.699206                                5.679896                                 7.689676                               10.431664                                           0.487950                                 0.440504                              263.424967                                  997.48                               584.302915                                  137.50                                                 33                                -0.595128                              261.535265                                         True                                        False                                        False                                        False                                        False                                              False                                         False                                          True                                          False                                              1                                         3

As you can see above, we have the exact same features as before, but calculated using the test data.

Exporting Feature Matrix

Save as csv

The feature matrix is a pandas dataframe that we can save to disk

In [11]: feature_matrix.to_csv("feature_matrix.csv")

We can also read it back in as follows:

In [12]: saved_fm = pd.read_csv("feature_matrix.csv", index_col="customer_id")

In [13]: saved_fm
Out[13]: 
             zip_code = 60091  zip_code = 13244  zip_code is unknown  COUNT(sessions)  MODE(sessions.device) = mobile  MODE(sessions.device) = desktop  MODE(sessions.device) is unknown  NUM_UNIQUE(sessions.device)  COUNT(transactions)  MAX(transactions.amount)  MEAN(transactions.amount)  MIN(transactions.amount)  MODE(transactions.product_id) = 4  MODE(transactions.product_id) = 5  MODE(transactions.product_id) = 2  MODE(transactions.product_id) = 1  MODE(transactions.product_id) is unknown  NUM_UNIQUE(transactions.product_id)  SKEW(transactions.amount)  STD(transactions.amount)  SUM(transactions.amount)  DAY(date_of_birth) = 18  DAY(date_of_birth) = 28  DAY(date_of_birth) = 21  DAY(date_of_birth) = 15  DAY(date_of_birth) is unknown  DAY(join_date) = 17  DAY(join_date) = 15  DAY(join_date) = 13  DAY(join_date) = 8  DAY(join_date) is unknown  MONTH(date_of_birth) = 8  MONTH(date_of_birth) = 7  MONTH(date_of_birth) = 11  MONTH(date_of_birth) is unknown  MONTH(join_date) = 4  MONTH(join_date) = 8  MONTH(join_date) = 7  MONTH(join_date) is unknown  WEEKDAY(date_of_birth) = 0  WEEKDAY(date_of_birth) = 5  WEEKDAY(date_of_birth) = 4  WEEKDAY(date_of_birth) = 1  WEEKDAY(date_of_birth) is unknown  WEEKDAY(join_date) = 6  WEEKDAY(join_date) = 5  WEEKDAY(join_date) = 4  WEEKDAY(join_date) is unknown  YEAR(date_of_birth) = 2006  YEAR(date_of_birth) = 2003  YEAR(date_of_birth) = 1994  YEAR(date_of_birth) = 1986  YEAR(date_of_birth) = 1984  YEAR(date_of_birth) is unknown  YEAR(join_date) = 2011  YEAR(join_date) = 2012  YEAR(join_date) = 2010  YEAR(join_date) is unknown  MAX(sessions.COUNT(transactions))  MAX(sessions.MEAN(transactions.amount))  MAX(sessions.MIN(transactions.amount))  MAX(sessions.NUM_UNIQUE(transactions.product_id))  MAX(sessions.SKEW(transactions.amount))  MAX(sessions.STD(transactions.amount))  MAX(sessions.SUM(transactions.amount))  MEAN(sessions.COUNT(transactions))  MEAN(sessions.MAX(transactions.amount))  MEAN(sessions.MEAN(transactions.amount))  MEAN(sessions.MIN(transactions.amount))  MEAN(sessions.NUM_UNIQUE(transactions.product_id))  MEAN(sessions.SKEW(transactions.amount))  MEAN(sessions.STD(transactions.amount))  MEAN(sessions.SUM(transactions.amount))  MIN(sessions.COUNT(transactions))  MIN(sessions.MAX(transactions.amount))  MIN(sessions.MEAN(transactions.amount))  MIN(sessions.NUM_UNIQUE(transactions.product_id))  MIN(sessions.SKEW(transactions.amount))  MIN(sessions.STD(transactions.amount))  MIN(sessions.SUM(transactions.amount))  MODE(sessions.DAY(session_start)) = 1  MODE(sessions.DAY(session_start)) is unknown  MODE(sessions.MODE(transactions.product_id)) = 3  MODE(sessions.MODE(transactions.product_id)) = 1  MODE(sessions.MODE(transactions.product_id)) = 4  MODE(sessions.MODE(transactions.product_id)) is unknown  MODE(sessions.MONTH(session_start)) = 1  MODE(sessions.MONTH(session_start)) is unknown  MODE(sessions.WEEKDAY(session_start)) = 2  MODE(sessions.WEEKDAY(session_start)) is unknown  MODE(sessions.YEAR(session_start)) = 2014  MODE(sessions.YEAR(session_start)) is unknown  NUM_UNIQUE(sessions.DAY(session_start))  NUM_UNIQUE(sessions.MODE(transactions.product_id))  NUM_UNIQUE(sessions.MONTH(session_start))  NUM_UNIQUE(sessions.WEEKDAY(session_start))  NUM_UNIQUE(sessions.YEAR(session_start))  SKEW(sessions.COUNT(transactions))  SKEW(sessions.MAX(transactions.amount))  SKEW(sessions.MEAN(transactions.amount))  SKEW(sessions.MIN(transactions.amount))  SKEW(sessions.NUM_UNIQUE(transactions.product_id))  SKEW(sessions.STD(transactions.amount))  SKEW(sessions.SUM(transactions.amount))  STD(sessions.COUNT(transactions))  STD(sessions.MAX(transactions.amount))  STD(sessions.MEAN(transactions.amount))  STD(sessions.MIN(transactions.amount))  STD(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.SKEW(transactions.amount))  STD(sessions.SUM(transactions.amount))  SUM(sessions.MAX(transactions.amount))  SUM(sessions.MEAN(transactions.amount))  SUM(sessions.MIN(transactions.amount))  SUM(sessions.NUM_UNIQUE(transactions.product_id))  SUM(sessions.SKEW(transactions.amount))  SUM(sessions.STD(transactions.amount))  MODE(transactions.sessions.customer_id) = 5  MODE(transactions.sessions.customer_id) = 4  MODE(transactions.sessions.customer_id) = 3  MODE(transactions.sessions.customer_id) = 2  MODE(transactions.sessions.customer_id) = 1  MODE(transactions.sessions.customer_id) is unknown  MODE(transactions.sessions.device) = mobile  MODE(transactions.sessions.device) = desktop  MODE(transactions.sessions.device) is unknown  NUM_UNIQUE(transactions.sessions.customer_id)  NUM_UNIQUE(transactions.sessions.device)
customer_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
1                        True             False                False                6                           False                             True                             False                            3                   73                    147.64                  79.128904                      6.47                               True                              False                              False                              False                                     False                                    5                  -0.173042                 41.998795                   5776.41                    False                    False                    False                    False                           True                False                False                False               False                       True                     False                     False                      False                             True                 False                 False                 False                         True                       False                       False                       False                       False                               True                    True                   False                   False                          False                       False                       False                       False                       False                       False                            True                   False                   False                   False                        True                                 19                               136.672500                                  124.58                                                  5                                 0.164928                               51.192518                                 1418.99                           12.166667                               139.013333                                 87.486612                                33.153333                                           4.166667                                  -0.257693                                36.177337                               962.735000                                  4                                  119.86                                64.238824                                                  2                                -0.614442                                9.176046                                  546.69                                   True                                         False                                             False                                             False                                              True                                              False                                           True                                           False                                       True                                             False                                       True                                          False                                        1                                                  4                                           1                                            1                                         1                           -0.243198                                -1.165955                                  1.796000                                 2.290693                                          -1.206607                                 -1.336611                                 0.130163                           5.741661                               11.742343                                25.801702                               45.471977                                           1.329160                                 0.326628                              327.331994                                  834.08                               524.919674                                  198.92                                                 25                                -1.546156                              217.064024                                        False                                        False                                        False                                        False                                         True                                              False                                          True                                         False                                          False                                              1                                         3
4                       False              True                False                9                           False                             True                             False                            3                  126                    147.55                  80.781190                      6.19                              False                              False                              False                              False                                      True                                    5                  -0.179621                 36.523849                  10178.43                    False                    False                    False                    False                           True                False                False                False               False                       True                     False                     False                       True                            False                 False                 False                  True                        False                       False                       False                        True                       False                              False                   False                   False                    True                          False                       False                       False                       False                       False                       False                            True                   False                   False                   False                        True                                 21                               104.565000                                   60.29                                                  5                                 0.417250                               41.627134                                 1650.65                           14.000000                               131.211111                                 81.540322                                21.453333                                           4.777778                                  -0.199690                                35.499735                              1130.936667                                  8                                  118.59                                69.665000                                                  4                                -0.624344                               22.026552                                  557.32                                   True                                         False                                             False                                              True                                             False                                              False                                           True                                           False                                       True                                             False                                       True                                          False                                        1                                                  5                                           1                                            1                                         1                           -0.086578                                 0.230847                                  1.078619                                 1.490781                                          -1.619848                                 -1.743267                                -0.385392                           4.272002                               11.457114                                11.875823                               17.851716                                           0.440959                                 0.324894                              333.923377                                 1180.90                               733.862898                                  193.08                                                 43                                -1.797214                              319.497611                                        False                                         True                                        False                                        False                                        False                                              False                                         False                                          True                                          False                                              1                                         3
3                        True             False                False                5                            True                            False                             False                            2                   64                    148.09                  82.171094                     10.66                              False                               True                              False                              False                                     False                                    5                  -0.081427                 42.416322                   5258.95                    False                    False                    False                    False                           True                False                False                False               False                       True                     False                     False                      False                             True                 False                 False                 False                         True                       False                       False                       False                       False                               True                   False                    True                   False                          False                        True                       False                       False                       False                       False                           False                   False                   False                    True                       False                                 19                                92.004000                                   38.69                                                  5                                 0.785376                               46.402508                                 1691.27                           12.800000                               143.160000                                 81.478110                                21.738000                                           4.600000                                   0.070612                                43.083442                              1051.790000                                  8                                  133.75                                66.563000                                                  4                                -0.608419                               37.945636                                  653.11                                   True                                         False                                             False                                              True                                             False                                              False                                           True                                           False                                       True                                             False                                       True                                          False                                        1                                                  2                                           1                                            1                                         1                            0.570300                                -1.598919                                 -0.730554                                 1.010658                                          -0.608581                                 -0.627238                                 0.727969                           4.868265                                5.596950                                10.012657                               10.829338                                           0.547723                                 0.530094                              450.318768                                  715.80                               407.390549                                  108.69                                                 23                                 0.353061                              215.417211                                        False                                        False                                         True                                        False                                        False                                              False                                          True                                         False                                          False                                              1                                         2
2                       False              True                False                8                           False                             True                             False                            3                  129                    148.34                  76.571085                      8.00                              False                              False                              False                               True                                     False                                    5                   0.040395                 39.913352                   9877.67                    False                    False                    False                    False                           True                False                False                False               False                       True                     False                     False                      False                             True                 False                 False                 False                         True                       False                       False                       False                       False                               True                    True                   False                   False                          False                       False                       False                       False                       False                       False                            True                   False                   False                    True                       False                                 21                                87.669412                                   40.88                                                  5                                 0.454842                               43.950396                                 1690.97                           16.125000                               137.602500                                 76.964367                                17.001250                                           4.875000                                  -0.010253                                39.477166                              1234.708750                                  8                                  120.06                                52.288421                                                  4                                -0.522578                               33.618728                                  619.93                                   True                                         False                                             False                                              True                                             False                                              False                                           True                                           False                                       True                                             False                                       True                                          False                                        1                                                  4                                           1                                            1                                         1                           -1.158217                                -0.964539                                 -2.048650                                 1.981423                                          -2.828427                                 -0.403715                                -0.576079                           4.086126                                9.491736                                10.757169                               10.517928                                           0.353553                                 0.305345                              346.152626                                 1100.82                               615.714934                                  136.01                                                 39                                -0.082021                              315.817331                                        False                                        False                                        False                                         True                                        False                                              False                                         False                                          True                                          False                                              1                                         3
5                        True             False                False                7                           False                             True                             False                            3                  108                    149.53                  83.506852                      6.35                              False                              False                              False                              False                                      True                                    5                  -0.107234                 37.514054                   9018.74                    False                     True                    False                    False                          False                False                False                False               False                       True                     False                     False                      False                             True                 False                 False                 False                         True                       False                       False                       False                        True                              False                   False                   False                    True                          False                       False                       False                       False                       False                       False                            True                   False                   False                   False                        True                                 19                                95.488000                                   34.93                                                  5                                 0.538086                               44.978308                                 1705.25                           15.428571                               142.497143                                 83.471845                                19.642857                                           4.714286                                  -0.085018                                37.362181                              1288.391429                                 11                                  133.41                                73.576471                                                  4                                -0.618136                               26.724118                                  921.73                                   True                                         False                                             False                                              True                                             False                                              False                                           True                                           False                                       True                                             False                                       True                                          False                                        1                                                  4                                           1                                            1                                         1                           -0.566615                                -0.308063                                  0.246613                                 0.013052                                          -1.229634                                 -0.405840                                 0.215180                           2.699206                                5.679896                                 7.689676                               10.431664                                           0.487950                                 0.440504                              263.424967                                  997.48                               584.302915                                  137.50                                                 33                                -0.595128                              261.535265                                         True                                        False                                        False                                        False                                        False                                              False                                         False                                          True                                          False                                              1                                         3