Deployment

Deployment of machine learning models requires repeating feature engineering steps on new data. In some cases, these steps need to be performed in near real-time. Featuretools has capabilities to ease the deployment of feature engineering.

Saving Features

First, let’s build some generate some training and test data in the same format. We use a random seed to generate different data for the test.

Note

Features saved in one version of Featuretools are not guaranteed to load in another. This means the features might need to be re-created after upgrading Featuretools.

In [1]: import featuretools as ft

In [2]: es_train = ft.demo.load_mock_customer(return_entityset=True)

In [3]: es_test = ft.demo.load_mock_customer(return_entityset=True, random_seed=33)

Now let’s build some features definitions using DFS. Because we have categorical features, we also encode them with one hot encoding based on the values in the training data.

In [4]: feature_matrix, feature_defs = ft.dfs(entityset=es_train,
   ...:                                       target_entity="customers")
   ...: 

In [5]: feature_matrix_enc, features_enc = ft.encode_features(feature_matrix, feature_defs)

In [6]: feature_matrix_enc
Out[6]: 
             zip_code = 60091  zip_code = 13244  zip_code is unknown  COUNT(sessions)  NUM_UNIQUE(sessions.device)  MODE(sessions.device) = mobile  MODE(sessions.device) = desktop  MODE(sessions.device) is unknown  SUM(transactions.amount)  STD(transactions.amount)  MAX(transactions.amount)  SKEW(transactions.amount)  MIN(transactions.amount)  MEAN(transactions.amount)  COUNT(transactions)  NUM_UNIQUE(transactions.product_id)  MODE(transactions.product_id) = 4  MODE(transactions.product_id) = 5  MODE(transactions.product_id) = 2  MODE(transactions.product_id) = 1  MODE(transactions.product_id) is unknown  DAY(date_of_birth) = 18  DAY(date_of_birth) = 28  DAY(date_of_birth) = 21  DAY(date_of_birth) = 15  DAY(date_of_birth) is unknown  DAY(join_date) = 17  DAY(join_date) = 15  DAY(join_date) = 13  DAY(join_date) = 8  DAY(join_date) is unknown  YEAR(date_of_birth) = 2006  YEAR(date_of_birth) = 2003  YEAR(date_of_birth) = 1994  YEAR(date_of_birth) = 1986  YEAR(date_of_birth) = 1984  YEAR(date_of_birth) is unknown  YEAR(join_date) = 2011  YEAR(join_date) = 2012  YEAR(join_date) = 2010  YEAR(join_date) is unknown  MONTH(date_of_birth) = 8  MONTH(date_of_birth) = 7  MONTH(date_of_birth) = 11  MONTH(date_of_birth) is unknown  MONTH(join_date) = 4  MONTH(join_date) = 8  MONTH(join_date) = 7  MONTH(join_date) is unknown  WEEKDAY(date_of_birth) = 0  WEEKDAY(date_of_birth) = 5  WEEKDAY(date_of_birth) = 4  WEEKDAY(date_of_birth) = 1  WEEKDAY(date_of_birth) is unknown  WEEKDAY(join_date) = 6  WEEKDAY(join_date) = 5  WEEKDAY(join_date) = 4  WEEKDAY(join_date) is unknown  SUM(sessions.MAX(transactions.amount))  SUM(sessions.SKEW(transactions.amount))  SUM(sessions.MIN(transactions.amount))  SUM(sessions.NUM_UNIQUE(transactions.product_id))  SUM(sessions.MEAN(transactions.amount))  SUM(sessions.STD(transactions.amount))  STD(sessions.COUNT(transactions))  STD(sessions.MAX(transactions.amount))  STD(sessions.SKEW(transactions.amount))  STD(sessions.MIN(transactions.amount))  STD(sessions.SUM(transactions.amount))  STD(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.MEAN(transactions.amount))  MAX(sessions.COUNT(transactions))  MAX(sessions.SKEW(transactions.amount))  MAX(sessions.MIN(transactions.amount))  MAX(sessions.SUM(transactions.amount))  MAX(sessions.NUM_UNIQUE(transactions.product_id))  MAX(sessions.MEAN(transactions.amount))  MAX(sessions.STD(transactions.amount))  SKEW(sessions.COUNT(transactions))  SKEW(sessions.MAX(transactions.amount))  SKEW(sessions.MIN(transactions.amount))  SKEW(sessions.SUM(transactions.amount))  SKEW(sessions.NUM_UNIQUE(transactions.product_id))  SKEW(sessions.MEAN(transactions.amount))  SKEW(sessions.STD(transactions.amount))  MIN(sessions.COUNT(transactions))  MIN(sessions.MAX(transactions.amount))  MIN(sessions.SKEW(transactions.amount))  MIN(sessions.SUM(transactions.amount))  MIN(sessions.NUM_UNIQUE(transactions.product_id))  MIN(sessions.MEAN(transactions.amount))  MIN(sessions.STD(transactions.amount))  MEAN(sessions.COUNT(transactions))  MEAN(sessions.MAX(transactions.amount))  MEAN(sessions.SKEW(transactions.amount))  MEAN(sessions.MIN(transactions.amount))  MEAN(sessions.SUM(transactions.amount))  MEAN(sessions.NUM_UNIQUE(transactions.product_id))  MEAN(sessions.MEAN(transactions.amount))  MEAN(sessions.STD(transactions.amount))  NUM_UNIQUE(sessions.MODE(transactions.product_id))  NUM_UNIQUE(sessions.MONTH(session_start))  NUM_UNIQUE(sessions.YEAR(session_start))  NUM_UNIQUE(sessions.WEEKDAY(session_start))  NUM_UNIQUE(sessions.DAY(session_start))  MODE(sessions.MODE(transactions.product_id)) = 3  MODE(sessions.MODE(transactions.product_id)) = 1  MODE(sessions.MODE(transactions.product_id)) = 4  MODE(sessions.MODE(transactions.product_id)) is unknown  MODE(sessions.MONTH(session_start)) = 1  MODE(sessions.MONTH(session_start)) is unknown  MODE(sessions.YEAR(session_start)) = 2014  MODE(sessions.YEAR(session_start)) is unknown  MODE(sessions.WEEKDAY(session_start)) = 2  MODE(sessions.WEEKDAY(session_start)) is unknown  MODE(sessions.DAY(session_start)) = 1  MODE(sessions.DAY(session_start)) is unknown  NUM_UNIQUE(transactions.sessions.customer_id)  NUM_UNIQUE(transactions.sessions.device)  MODE(transactions.sessions.customer_id) = 5  MODE(transactions.sessions.customer_id) = 4  MODE(transactions.sessions.customer_id) = 3  MODE(transactions.sessions.customer_id) = 2  MODE(transactions.sessions.customer_id) = 1  MODE(transactions.sessions.customer_id) is unknown  MODE(transactions.sessions.device) = mobile  MODE(transactions.sessions.device) = desktop  MODE(transactions.sessions.device) is unknown
customer_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
5                           1                 0                    0                6                            3                               1                                0                                 0                   6349.66                 44.095630                    149.02                  -0.025941                      7.55                  80.375443                   79                                    5                                  0                                  1                                  0                                  0                                         0                        0                        1                        0                        0                              0                    1                    0                    0                   0                          0                           0                           0                           0                           0                           1                               0                       0                       0                       1                           0                         0                         1                          0                                0                     0                     0                     1                            0                           0                           1                           0                           0                                  0                       0                       1                       0                              0                                  839.76                                 0.014384                                   86.49                                                 30                               472.231119                              259.873954                           3.600926                                7.928001                                 0.415426                                4.961414                              402.775486                                           0.000000                                11.007471                                 18                                 0.602209                                   20.65                                 1700.67                                                  5                                94.481667                               51.149250                           -0.317685                                -0.333796                                -0.470410                                 0.472342                                           0.000000                                   0.335175                                 0.204548                                  8                                  128.51                                -0.539060                                  543.18                                                  5                                66.666667                               36.734681                           13.166667                               139.960000                                  0.002397                                14.415000                              1058.276667                                           5.000000                                  78.705187                                43.312326                                                  5                                           1                                         1                                            1                                        1                                                 1                                                 0                                                 0                                                  0                                              1                                               0                                          1                                              0                                          1                                                 0                                      1                                             0                                              1                                         3                                            1                                            0                                            0                                            0                                            0                                                  0                                             1                                             0                                              0
4                           1                 0                    0                8                            3                               1                                0                                 0                   8727.68                 45.068765                    149.95                  -0.036348                      5.73                  80.070459                  109                                    5                                  0                                  0                                  1                                  0                                         0                        0                        0                        0                        1                              0                    0                    0                    0                   1                          0                           1                           0                           0                           0                           0                               0                       1                       0                       0                           0                         1                         0                          0                                0                     1                     0                     0                            0                           0                           0                           0                           1                                  0                       0                       0                       1                              0                                 1157.99                                 0.002764                                  131.51                                                 37                               649.657515                              356.125829                           3.335416                                3.514421                                 0.387884                               16.960575                              235.992478                                           0.517549                                13.027258                                 18                                 0.382868                                   54.83                                 1351.46                                                  5                               110.450000                               54.293903                            0.282488                                 0.027256                                 2.103510                                -0.391805                                          -0.644061                                   1.980948                                -1.065663                                 10                                  139.20                                -0.711744                                  771.68                                                  4                                70.638182                               29.026424                           13.625000                               144.748750                                  0.000346                                16.438750                              1090.960000                                           4.625000                                  81.207189                                44.515729                                                  5                                           1                                         1                                            1                                        1                                                 0                                                 1                                                 0                                                  0                                              1                                               0                                          1                                              0                                          1                                                 0                                      1                                             0                                              1                                         3                                            0                                            1                                            0                                            0                                            0                                                  0                                             1                                             0                                              0
1                           1                 0                    0                8                            3                               1                                0                                 0                   9025.62                 40.442059                    139.43                   0.019698                      5.81                  71.631905                  126                                    5                                  1                                  0                                  0                                  0                                         0                        1                        0                        0                        0                              0                    1                    0                    0                   0                          0                           0                           0                           1                           0                           0                               0                       1                       0                       0                           0                         0                         1                          0                                0                     1                     0                     0                            0                           1                           0                           0                           0                                  0                       1                       0                       0                              0                                 1057.97                                -0.476122                                   78.59                                                 40                               582.193117                              312.745952                           4.062019                                7.322191                                 0.589386                                6.954507                              279.510713                                           0.000000                                13.759314                                 25                                 0.640252                                   26.36                                 1613.93                                                  5                                88.755625                               46.905665                            1.946018                                -0.780493                                 2.440005                                 0.778170                                           0.000000                                  -0.424949                                -0.312355                                 12                                  118.90                                -1.038434                                  809.97                                                  5                                50.623125                               30.450261                           15.750000                               132.246250                                 -0.059515                                 9.823750                              1128.202500                                           5.000000                                  72.774140                                39.093244                                                  4                                           1                                         1                                            1                                        1                                                 0                                                 0                                                 1                                                  0                                              1                                               0                                          1                                              0                                          1                                                 0                                      1                                             0                                              1                                         3                                            0                                            0                                            0                                            0                                            1                                                  0                                             1                                             0                                              0
3                           0                 1                    0                6                            3                               0                                1                                 0                   6236.62                 43.683296                    149.15                   0.418230                      5.89                  67.060430                   93                                    5                                  0                                  0                                  0                                  1                                         0                        0                        0                        1                        0                              0                    0                    0                    1                   0                          0                           0                           1                           0                           0                           0                               0                       1                       0                       0                           0                         0                         0                          1                                0                     0                     1                     0                            0                           0                           0                           1                           0                                  0                       0                       1                       0                              0                                  847.63                                 2.286086                                   66.21                                                 29                               405.237462                              257.299895                           2.428992                               10.724241                                 0.429374                                5.424407                              219.021420                                           0.408248                                11.174282                                 18                                 0.854976                                   20.06                                 1477.97                                                  5                                82.109444                               50.110120                           -1.507217                                -0.941078                                 1.000771                                 2.246479                                          -2.449490                                   0.678544                                -0.245703                                 11                                  126.74                                -0.289466                                  889.21                                                  4                                55.579412                               35.704680                           15.500000                               141.271667                                  0.381014                                11.035000                              1039.436667                                           4.833333                                  67.539577                                42.883316                                                  4                                           1                                         1                                            1                                        1                                                 0                                                 1                                                 0                                                  0                                              1                                               0                                          1                                              0                                          1                                                 0                                      1                                             0                                              1                                         3                                            0                                            0                                            1                                            0                                            0                                                  0                                             0                                             1                                              0
2                           0                 1                    0                7                            3                               0                                1                                 0                   7200.28                 37.705178                    146.81                   0.098259                      8.73                  77.422366                   93                                    5                                  1                                  0                                  0                                  0                                         0                        1                        0                        0                        0                              0                    0                    1                    0                   0                          0                           0                           0                           0                           1                           0                               0                       0                       1                       0                           0                         1                         0                          0                                0                     1                     0                     0                            0                           1                           0                           0                           0                                  0                       1                       0                       0                              0                                  931.63                                -0.277640                                  154.60                                                 35                               548.905851                              258.700528                           3.450328                               17.221593                                 0.509798                               15.874374                              251.609234                                           0.000000                                11.477071                                 18                                 0.755711                                   56.46                                 1320.64                                                  5                                96.581000                               47.935920                           -0.303276                                -1.539467                                 2.154929                                -0.440929                                           0.000000                                   0.235296                                 0.013087                                  8                                  100.04                                -0.763603                                  634.84                                                  5                                61.910000                               27.839228                           13.285714                               133.090000                                 -0.039663                                22.085714                              1028.611429                                           5.000000                                  78.415122                                36.957218                                                  4                                           1                                         1                                            1                                        1                                                 1                                                 0                                                 0                                                  0                                              1                                               0                                          1                                              0                                          1                                                 0                                      1                                             0                                              1                                         3                                            0                                            0                                            0                                            1                                            0                                                  0                                             0                                             1                                              0

Now, we can use featuretools.save_features() to save a list features to a json file

In [7]: ft.save_features(features_enc, "feature_definitions.json")

Calculating Feature Matrix for New Data

We can use featuretools.load_features() to read in a list of saved features to calculate for our new entity set.

In [8]: saved_features = ft.load_features('feature_definitions.json')

After we load the features back in, we can calculate the feature matrix.

In [9]: feature_matrix = ft.calculate_feature_matrix(saved_features, es_test)

In [10]: feature_matrix
Out[10]: 
             zip_code = 60091  zip_code = 13244  zip_code is unknown  COUNT(sessions)  NUM_UNIQUE(sessions.device)  MODE(sessions.device) = mobile  MODE(sessions.device) = desktop  MODE(sessions.device) is unknown  SUM(transactions.amount)  STD(transactions.amount)  MAX(transactions.amount)  SKEW(transactions.amount)  MIN(transactions.amount)  MEAN(transactions.amount)  COUNT(transactions)  NUM_UNIQUE(transactions.product_id)  MODE(transactions.product_id) = 4  MODE(transactions.product_id) = 5  MODE(transactions.product_id) = 2  MODE(transactions.product_id) = 1  MODE(transactions.product_id) is unknown  DAY(date_of_birth) = 18  DAY(date_of_birth) = 28  DAY(date_of_birth) = 21  DAY(date_of_birth) = 15  DAY(date_of_birth) is unknown  DAY(join_date) = 17  DAY(join_date) = 15  DAY(join_date) = 13  DAY(join_date) = 8  DAY(join_date) is unknown  YEAR(date_of_birth) = 2006  YEAR(date_of_birth) = 2003  YEAR(date_of_birth) = 1994  YEAR(date_of_birth) = 1986  YEAR(date_of_birth) = 1984  YEAR(date_of_birth) is unknown  YEAR(join_date) = 2011  YEAR(join_date) = 2012  YEAR(join_date) = 2010  YEAR(join_date) is unknown  MONTH(date_of_birth) = 8  MONTH(date_of_birth) = 7  MONTH(date_of_birth) = 11  MONTH(date_of_birth) is unknown  MONTH(join_date) = 4  MONTH(join_date) = 8  MONTH(join_date) = 7  MONTH(join_date) is unknown  WEEKDAY(date_of_birth) = 0  WEEKDAY(date_of_birth) = 5  WEEKDAY(date_of_birth) = 4  WEEKDAY(date_of_birth) = 1  WEEKDAY(date_of_birth) is unknown  WEEKDAY(join_date) = 6  WEEKDAY(join_date) = 5  WEEKDAY(join_date) = 4  WEEKDAY(join_date) is unknown  SUM(sessions.MAX(transactions.amount))  SUM(sessions.SKEW(transactions.amount))  SUM(sessions.MIN(transactions.amount))  SUM(sessions.NUM_UNIQUE(transactions.product_id))  SUM(sessions.MEAN(transactions.amount))  SUM(sessions.STD(transactions.amount))  STD(sessions.COUNT(transactions))  STD(sessions.MAX(transactions.amount))  STD(sessions.SKEW(transactions.amount))  STD(sessions.MIN(transactions.amount))  STD(sessions.SUM(transactions.amount))  STD(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.MEAN(transactions.amount))  MAX(sessions.COUNT(transactions))  MAX(sessions.SKEW(transactions.amount))  MAX(sessions.MIN(transactions.amount))  MAX(sessions.SUM(transactions.amount))  MAX(sessions.NUM_UNIQUE(transactions.product_id))  MAX(sessions.MEAN(transactions.amount))  MAX(sessions.STD(transactions.amount))  SKEW(sessions.COUNT(transactions))  SKEW(sessions.MAX(transactions.amount))  SKEW(sessions.MIN(transactions.amount))  SKEW(sessions.SUM(transactions.amount))  SKEW(sessions.NUM_UNIQUE(transactions.product_id))  SKEW(sessions.MEAN(transactions.amount))  SKEW(sessions.STD(transactions.amount))  MIN(sessions.COUNT(transactions))  MIN(sessions.MAX(transactions.amount))  MIN(sessions.SKEW(transactions.amount))  MIN(sessions.SUM(transactions.amount))  MIN(sessions.NUM_UNIQUE(transactions.product_id))  MIN(sessions.MEAN(transactions.amount))  MIN(sessions.STD(transactions.amount))  MEAN(sessions.COUNT(transactions))  MEAN(sessions.MAX(transactions.amount))  MEAN(sessions.SKEW(transactions.amount))  MEAN(sessions.MIN(transactions.amount))  MEAN(sessions.SUM(transactions.amount))  MEAN(sessions.NUM_UNIQUE(transactions.product_id))  MEAN(sessions.MEAN(transactions.amount))  MEAN(sessions.STD(transactions.amount))  NUM_UNIQUE(sessions.MODE(transactions.product_id))  NUM_UNIQUE(sessions.MONTH(session_start))  NUM_UNIQUE(sessions.YEAR(session_start))  NUM_UNIQUE(sessions.WEEKDAY(session_start))  NUM_UNIQUE(sessions.DAY(session_start))  MODE(sessions.MODE(transactions.product_id)) = 3  MODE(sessions.MODE(transactions.product_id)) = 1  MODE(sessions.MODE(transactions.product_id)) = 4  MODE(sessions.MODE(transactions.product_id)) is unknown  MODE(sessions.MONTH(session_start)) = 1  MODE(sessions.MONTH(session_start)) is unknown  MODE(sessions.YEAR(session_start)) = 2014  MODE(sessions.YEAR(session_start)) is unknown  MODE(sessions.WEEKDAY(session_start)) = 2  MODE(sessions.WEEKDAY(session_start)) is unknown  MODE(sessions.DAY(session_start)) = 1  MODE(sessions.DAY(session_start)) is unknown  NUM_UNIQUE(transactions.sessions.customer_id)  NUM_UNIQUE(transactions.sessions.device)  MODE(transactions.sessions.customer_id) = 5  MODE(transactions.sessions.customer_id) = 4  MODE(transactions.sessions.customer_id) = 3  MODE(transactions.sessions.customer_id) = 2  MODE(transactions.sessions.customer_id) = 1  MODE(transactions.sessions.customer_id) is unknown  MODE(transactions.sessions.device) = mobile  MODE(transactions.sessions.device) = desktop  MODE(transactions.sessions.device) is unknown
customer_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
1                        True             False                False                6                            3                           False                             True                             False                   5776.41                 41.998795                    147.64                  -0.173042                      6.47                  79.128904                   73                                    5                               True                              False                              False                              False                                     False                    False                    False                    False                    False                           True                False                False                False               False                       True                       False                       False                       False                       False                       False                            True                   False                   False                   False                        True                     False                     False                      False                             True                 False                 False                 False                         True                       False                       False                       False                       False                               True                    True                   False                   False                          False                                  834.08                                -1.546156                                  198.92                                                 25                               524.919674                              217.064024                           5.741661                               11.742343                                 0.326628                               45.471977                              327.331994                                           1.329160                                25.801702                                 19                                 0.164928                                  124.58                                 1418.99                                                  5                               136.672500                               51.192518                           -0.243198                                -1.165955                                 2.290693                                 0.130163                                          -1.206607                                   1.796000                                -1.336611                                  4                                  119.86                                -0.614442                                  546.69                                                  2                                64.238824                                9.176046                           12.166667                               139.013333                                 -0.257693                                33.153333                               962.735000                                           4.166667                                  87.486612                                36.177337                                                  4                                           1                                         1                                            1                                        1                                             False                                             False                                              True                                              False                                           True                                           False                                       True                                          False                                       True                                             False                                   True                                         False                                              1                                         3                                        False                                        False                                        False                                        False                                         True                                              False                                          True                                         False                                          False
4                       False              True                False                9                            3                           False                             True                             False                  10178.43                 36.523849                    147.55                  -0.179621                      6.19                  80.781190                  126                                    5                              False                              False                              False                              False                                      True                    False                    False                    False                    False                           True                False                False                False               False                       True                       False                       False                       False                       False                       False                            True                   False                   False                   False                        True                     False                     False                       True                            False                 False                 False                  True                        False                       False                       False                        True                       False                              False                   False                   False                    True                          False                                 1180.90                                -1.797214                                  193.08                                                 43                               733.862898                              319.497611                           4.272002                               11.457114                                 0.324894                               17.851716                              333.923377                                           0.440959                                11.875823                                 21                                 0.417250                                   60.29                                 1650.65                                                  5                               104.565000                               41.627134                           -0.086578                                 0.230847                                 1.490781                                -0.385392                                          -1.619848                                   1.078619                                -1.743267                                  8                                  118.59                                -0.624344                                  557.32                                                  4                                69.665000                               22.026552                           14.000000                               131.211111                                 -0.199690                                21.453333                              1130.936667                                           4.777778                                  81.540322                                35.499735                                                  5                                           1                                         1                                            1                                        1                                             False                                              True                                             False                                              False                                           True                                           False                                       True                                          False                                       True                                             False                                   True                                         False                                              1                                         3                                        False                                         True                                        False                                        False                                        False                                              False                                         False                                          True                                          False
3                        True             False                False                5                            2                            True                            False                             False                   5258.95                 42.416322                    148.09                  -0.081427                     10.66                  82.171094                   64                                    5                              False                               True                              False                              False                                     False                    False                    False                    False                    False                           True                False                False                False               False                       True                        True                       False                       False                       False                       False                           False                   False                   False                    True                       False                     False                     False                      False                             True                 False                 False                 False                         True                       False                       False                       False                       False                               True                   False                    True                   False                          False                                  715.80                                 0.353061                                  108.69                                                 23                               407.390549                              215.417211                           4.868265                                5.596950                                 0.530094                               10.829338                              450.318768                                           0.547723                                10.012657                                 19                                 0.785376                                   38.69                                 1691.27                                                  5                                92.004000                               46.402508                            0.570300                                -1.598919                                 1.010658                                 0.727969                                          -0.608581                                  -0.730554                                -0.627238                                  8                                  133.75                                -0.608419                                  653.11                                                  4                                66.563000                               37.945636                           12.800000                               143.160000                                  0.070612                                21.738000                              1051.790000                                           4.600000                                  81.478110                                43.083442                                                  2                                           1                                         1                                            1                                        1                                             False                                              True                                             False                                              False                                           True                                           False                                       True                                          False                                       True                                             False                                   True                                         False                                              1                                         2                                        False                                        False                                         True                                        False                                        False                                              False                                          True                                         False                                          False
2                       False              True                False                8                            3                           False                             True                             False                   9877.67                 39.913352                    148.34                   0.040395                      8.00                  76.571085                  129                                    5                              False                              False                              False                               True                                     False                    False                    False                    False                    False                           True                False                False                False               False                       True                       False                       False                       False                       False                       False                            True                   False                   False                    True                       False                     False                     False                      False                             True                 False                 False                 False                         True                       False                       False                       False                       False                               True                    True                   False                   False                          False                                 1100.82                                -0.082021                                  136.01                                                 39                               615.714934                              315.817331                           4.086126                                9.491736                                 0.305345                               10.517928                              346.152626                                           0.353553                                10.757169                                 21                                 0.454842                                   40.88                                 1690.97                                                  5                                87.669412                               43.950396                           -1.158217                                -0.964539                                 1.981423                                -0.576079                                          -2.828427                                  -2.048650                                -0.403715                                  8                                  120.06                                -0.522578                                  619.93                                                  4                                52.288421                               33.618728                           16.125000                               137.602500                                 -0.010253                                17.001250                              1234.708750                                           4.875000                                  76.964367                                39.477166                                                  4                                           1                                         1                                            1                                        1                                             False                                              True                                             False                                              False                                           True                                           False                                       True                                          False                                       True                                             False                                   True                                         False                                              1                                         3                                        False                                        False                                        False                                         True                                        False                                              False                                         False                                          True                                          False
5                        True             False                False                7                            3                           False                             True                             False                   9018.74                 37.514054                    149.53                  -0.107234                      6.35                  83.506852                  108                                    5                              False                              False                              False                              False                                      True                    False                     True                    False                    False                          False                False                False                False               False                       True                       False                       False                       False                       False                       False                            True                   False                   False                   False                        True                     False                     False                      False                             True                 False                 False                 False                         True                       False                       False                       False                        True                              False                   False                   False                    True                          False                                  997.48                                -0.595128                                  137.50                                                 33                               584.302915                              261.535265                           2.699206                                5.679896                                 0.440504                               10.431664                              263.424967                                           0.487950                                 7.689676                                 19                                 0.538086                                   34.93                                 1705.25                                                  5                                95.488000                               44.978308                           -0.566615                                -0.308063                                 0.013052                                 0.215180                                          -1.229634                                   0.246613                                -0.405840                                 11                                  133.41                                -0.618136                                  921.73                                                  4                                73.576471                               26.724118                           15.428571                               142.497143                                 -0.085018                                19.642857                              1288.391429                                           4.714286                                  83.471845                                37.362181                                                  4                                           1                                         1                                            1                                        1                                             False                                              True                                             False                                              False                                           True                                           False                                       True                                          False                                       True                                             False                                   True                                         False                                              1                                         3                                         True                                        False                                        False                                        False                                        False                                              False                                         False                                          True                                          False

As you can see above, we have the exact same features as before, but calculated using the test data.

Exporting Feature Matrix

Save as csv

The feature matrix is a pandas dataframe that we can save to disk

In [11]: feature_matrix.to_csv("feature_matrix.csv")

We can also read it back in as follows:

In [12]: saved_fm = pd.read_csv("feature_matrix.csv", index_col="customer_id")

In [13]: saved_fm
Out[13]: 
             zip_code = 60091  zip_code = 13244  zip_code is unknown  COUNT(sessions)  NUM_UNIQUE(sessions.device)  MODE(sessions.device) = mobile  MODE(sessions.device) = desktop  MODE(sessions.device) is unknown  SUM(transactions.amount)  STD(transactions.amount)  MAX(transactions.amount)  SKEW(transactions.amount)  MIN(transactions.amount)  MEAN(transactions.amount)  COUNT(transactions)  NUM_UNIQUE(transactions.product_id)  MODE(transactions.product_id) = 4  MODE(transactions.product_id) = 5  MODE(transactions.product_id) = 2  MODE(transactions.product_id) = 1  MODE(transactions.product_id) is unknown  DAY(date_of_birth) = 18  DAY(date_of_birth) = 28  DAY(date_of_birth) = 21  DAY(date_of_birth) = 15  DAY(date_of_birth) is unknown  DAY(join_date) = 17  DAY(join_date) = 15  DAY(join_date) = 13  DAY(join_date) = 8  DAY(join_date) is unknown  YEAR(date_of_birth) = 2006  YEAR(date_of_birth) = 2003  YEAR(date_of_birth) = 1994  YEAR(date_of_birth) = 1986  YEAR(date_of_birth) = 1984  YEAR(date_of_birth) is unknown  YEAR(join_date) = 2011  YEAR(join_date) = 2012  YEAR(join_date) = 2010  YEAR(join_date) is unknown  MONTH(date_of_birth) = 8  MONTH(date_of_birth) = 7  MONTH(date_of_birth) = 11  MONTH(date_of_birth) is unknown  MONTH(join_date) = 4  MONTH(join_date) = 8  MONTH(join_date) = 7  MONTH(join_date) is unknown  WEEKDAY(date_of_birth) = 0  WEEKDAY(date_of_birth) = 5  WEEKDAY(date_of_birth) = 4  WEEKDAY(date_of_birth) = 1  WEEKDAY(date_of_birth) is unknown  WEEKDAY(join_date) = 6  WEEKDAY(join_date) = 5  WEEKDAY(join_date) = 4  WEEKDAY(join_date) is unknown  SUM(sessions.MAX(transactions.amount))  SUM(sessions.SKEW(transactions.amount))  SUM(sessions.MIN(transactions.amount))  SUM(sessions.NUM_UNIQUE(transactions.product_id))  SUM(sessions.MEAN(transactions.amount))  SUM(sessions.STD(transactions.amount))  STD(sessions.COUNT(transactions))  STD(sessions.MAX(transactions.amount))  STD(sessions.SKEW(transactions.amount))  STD(sessions.MIN(transactions.amount))  STD(sessions.SUM(transactions.amount))  STD(sessions.NUM_UNIQUE(transactions.product_id))  STD(sessions.MEAN(transactions.amount))  MAX(sessions.COUNT(transactions))  MAX(sessions.SKEW(transactions.amount))  MAX(sessions.MIN(transactions.amount))  MAX(sessions.SUM(transactions.amount))  MAX(sessions.NUM_UNIQUE(transactions.product_id))  MAX(sessions.MEAN(transactions.amount))  MAX(sessions.STD(transactions.amount))  SKEW(sessions.COUNT(transactions))  SKEW(sessions.MAX(transactions.amount))  SKEW(sessions.MIN(transactions.amount))  SKEW(sessions.SUM(transactions.amount))  SKEW(sessions.NUM_UNIQUE(transactions.product_id))  SKEW(sessions.MEAN(transactions.amount))  SKEW(sessions.STD(transactions.amount))  MIN(sessions.COUNT(transactions))  MIN(sessions.MAX(transactions.amount))  MIN(sessions.SKEW(transactions.amount))  MIN(sessions.SUM(transactions.amount))  MIN(sessions.NUM_UNIQUE(transactions.product_id))  MIN(sessions.MEAN(transactions.amount))  MIN(sessions.STD(transactions.amount))  MEAN(sessions.COUNT(transactions))  MEAN(sessions.MAX(transactions.amount))  MEAN(sessions.SKEW(transactions.amount))  MEAN(sessions.MIN(transactions.amount))  MEAN(sessions.SUM(transactions.amount))  MEAN(sessions.NUM_UNIQUE(transactions.product_id))  MEAN(sessions.MEAN(transactions.amount))  MEAN(sessions.STD(transactions.amount))  NUM_UNIQUE(sessions.MODE(transactions.product_id))  NUM_UNIQUE(sessions.MONTH(session_start))  NUM_UNIQUE(sessions.YEAR(session_start))  NUM_UNIQUE(sessions.WEEKDAY(session_start))  NUM_UNIQUE(sessions.DAY(session_start))  MODE(sessions.MODE(transactions.product_id)) = 3  MODE(sessions.MODE(transactions.product_id)) = 1  MODE(sessions.MODE(transactions.product_id)) = 4  MODE(sessions.MODE(transactions.product_id)) is unknown  MODE(sessions.MONTH(session_start)) = 1  MODE(sessions.MONTH(session_start)) is unknown  MODE(sessions.YEAR(session_start)) = 2014  MODE(sessions.YEAR(session_start)) is unknown  MODE(sessions.WEEKDAY(session_start)) = 2  MODE(sessions.WEEKDAY(session_start)) is unknown  MODE(sessions.DAY(session_start)) = 1  MODE(sessions.DAY(session_start)) is unknown  NUM_UNIQUE(transactions.sessions.customer_id)  NUM_UNIQUE(transactions.sessions.device)  MODE(transactions.sessions.customer_id) = 5  MODE(transactions.sessions.customer_id) = 4  MODE(transactions.sessions.customer_id) = 3  MODE(transactions.sessions.customer_id) = 2  MODE(transactions.sessions.customer_id) = 1  MODE(transactions.sessions.customer_id) is unknown  MODE(transactions.sessions.device) = mobile  MODE(transactions.sessions.device) = desktop  MODE(transactions.sessions.device) is unknown
customer_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
1                        True             False                False                6                            3                           False                             True                             False                   5776.41                 41.998795                    147.64                  -0.173042                      6.47                  79.128904                   73                                    5                               True                              False                              False                              False                                     False                    False                    False                    False                    False                           True                False                False                False               False                       True                       False                       False                       False                       False                       False                            True                   False                   False                   False                        True                     False                     False                      False                             True                 False                 False                 False                         True                       False                       False                       False                       False                               True                    True                   False                   False                          False                                  834.08                                -1.546156                                  198.92                                                 25                               524.919674                              217.064024                           5.741661                               11.742343                                 0.326628                               45.471977                              327.331994                                           1.329160                                25.801702                                 19                                 0.164928                                  124.58                                 1418.99                                                  5                               136.672500                               51.192518                           -0.243198                                -1.165955                                 2.290693                                 0.130163                                          -1.206607                                   1.796000                                -1.336611                                  4                                  119.86                                -0.614442                                  546.69                                                  2                                64.238824                                9.176046                           12.166667                               139.013333                                 -0.257693                                33.153333                               962.735000                                           4.166667                                  87.486612                                36.177337                                                  4                                           1                                         1                                            1                                        1                                             False                                             False                                              True                                              False                                           True                                           False                                       True                                          False                                       True                                             False                                   True                                         False                                              1                                         3                                        False                                        False                                        False                                        False                                         True                                              False                                          True                                         False                                          False
4                       False              True                False                9                            3                           False                             True                             False                  10178.43                 36.523849                    147.55                  -0.179621                      6.19                  80.781190                  126                                    5                              False                              False                              False                              False                                      True                    False                    False                    False                    False                           True                False                False                False               False                       True                       False                       False                       False                       False                       False                            True                   False                   False                   False                        True                     False                     False                       True                            False                 False                 False                  True                        False                       False                       False                        True                       False                              False                   False                   False                    True                          False                                 1180.90                                -1.797214                                  193.08                                                 43                               733.862898                              319.497611                           4.272002                               11.457114                                 0.324894                               17.851716                              333.923377                                           0.440959                                11.875823                                 21                                 0.417250                                   60.29                                 1650.65                                                  5                               104.565000                               41.627134                           -0.086578                                 0.230847                                 1.490781                                -0.385392                                          -1.619848                                   1.078619                                -1.743267                                  8                                  118.59                                -0.624344                                  557.32                                                  4                                69.665000                               22.026552                           14.000000                               131.211111                                 -0.199690                                21.453333                              1130.936667                                           4.777778                                  81.540322                                35.499735                                                  5                                           1                                         1                                            1                                        1                                             False                                              True                                             False                                              False                                           True                                           False                                       True                                          False                                       True                                             False                                   True                                         False                                              1                                         3                                        False                                         True                                        False                                        False                                        False                                              False                                         False                                          True                                          False
3                        True             False                False                5                            2                            True                            False                             False                   5258.95                 42.416322                    148.09                  -0.081427                     10.66                  82.171094                   64                                    5                              False                               True                              False                              False                                     False                    False                    False                    False                    False                           True                False                False                False               False                       True                        True                       False                       False                       False                       False                           False                   False                   False                    True                       False                     False                     False                      False                             True                 False                 False                 False                         True                       False                       False                       False                       False                               True                   False                    True                   False                          False                                  715.80                                 0.353061                                  108.69                                                 23                               407.390549                              215.417211                           4.868265                                5.596950                                 0.530094                               10.829338                              450.318768                                           0.547723                                10.012657                                 19                                 0.785376                                   38.69                                 1691.27                                                  5                                92.004000                               46.402508                            0.570300                                -1.598919                                 1.010658                                 0.727969                                          -0.608581                                  -0.730554                                -0.627238                                  8                                  133.75                                -0.608419                                  653.11                                                  4                                66.563000                               37.945636                           12.800000                               143.160000                                  0.070612                                21.738000                              1051.790000                                           4.600000                                  81.478110                                43.083442                                                  2                                           1                                         1                                            1                                        1                                             False                                              True                                             False                                              False                                           True                                           False                                       True                                          False                                       True                                             False                                   True                                         False                                              1                                         2                                        False                                        False                                         True                                        False                                        False                                              False                                          True                                         False                                          False
2                       False              True                False                8                            3                           False                             True                             False                   9877.67                 39.913352                    148.34                   0.040395                      8.00                  76.571085                  129                                    5                              False                              False                              False                               True                                     False                    False                    False                    False                    False                           True                False                False                False               False                       True                       False                       False                       False                       False                       False                            True                   False                   False                    True                       False                     False                     False                      False                             True                 False                 False                 False                         True                       False                       False                       False                       False                               True                    True                   False                   False                          False                                 1100.82                                -0.082021                                  136.01                                                 39                               615.714934                              315.817331                           4.086126                                9.491736                                 0.305345                               10.517928                              346.152626                                           0.353553                                10.757169                                 21                                 0.454842                                   40.88                                 1690.97                                                  5                                87.669412                               43.950396                           -1.158217                                -0.964539                                 1.981423                                -0.576079                                          -2.828427                                  -2.048650                                -0.403715                                  8                                  120.06                                -0.522578                                  619.93                                                  4                                52.288421                               33.618728                           16.125000                               137.602500                                 -0.010253                                17.001250                              1234.708750                                           4.875000                                  76.964367                                39.477166                                                  4                                           1                                         1                                            1                                        1                                             False                                              True                                             False                                              False                                           True                                           False                                       True                                          False                                       True                                             False                                   True                                         False                                              1                                         3                                        False                                        False                                        False                                         True                                        False                                              False                                         False                                          True                                          False
5                        True             False                False                7                            3                           False                             True                             False                   9018.74                 37.514054                    149.53                  -0.107234                      6.35                  83.506852                  108                                    5                              False                              False                              False                              False                                      True                    False                     True                    False                    False                          False                False                False                False               False                       True                       False                       False                       False                       False                       False                            True                   False                   False                   False                        True                     False                     False                      False                             True                 False                 False                 False                         True                       False                       False                       False                        True                              False                   False                   False                    True                          False                                  997.48                                -0.595128                                  137.50                                                 33                               584.302915                              261.535265                           2.699206                                5.679896                                 0.440504                               10.431664                              263.424967                                           0.487950                                 7.689676                                 19                                 0.538086                                   34.93                                 1705.25                                                  5                                95.488000                               44.978308                           -0.566615                                -0.308063                                 0.013052                                 0.215180                                          -1.229634                                   0.246613                                -0.405840                                 11                                  133.41                                -0.618136                                  921.73                                                  4                                73.576471                               26.724118                           15.428571                               142.497143                                 -0.085018                                19.642857                              1288.391429                                           4.714286                                  83.471845                                37.362181                                                  4                                           1                                         1                                            1                                        1                                             False                                              True                                             False                                              False                                           True                                           False                                       True                                          False                                       True                                             False                                   True                                         False                                              1                                         3                                         True                                        False                                        False                                        False                                        False                                              False                                         False                                          True                                          False