{ "cells": [ { "cell_type": "markdown", "id": "b95b28c1", "metadata": {}, "source": [ "# Woodwork Typing in Featuretools\n", "\n", "Featuretools relies on having consistent typing across the creation of EntitySets, Primitives, Features, and feature matrices. Previously, Featuretools used its own type system that contained objects called Variables. Now and moving forward, Featuretools will use an external data typing library for its typing: [Woodwork](https://woodwork.alteryx.com/en/stable/index.html).\n", "\n", "Understanding the Woodwork types that exist and how Featuretools uses Woodwork's type system will allow users to:\n", " - build EntitySets that best represent their data\n", " - understand the possible input and return types for Featuretools' Primitives\n", " - understand what features will get generated from a given set of data and primitives.\n", "\n", "Read the [Understanding Woodwork Logical Types and Semantic Tags](https://woodwork.alteryx.com/en/stable/guides/logical_types_and_semantic_tags.html) guide for an in-depth walkthrough of the available Woodwork types that are outlined below.\n", "\n", "For users that are familiar with the old `Variable` objects, the [Transitioning to Featuretools Version 1.0](../resources/transition_to_ft_v1.0.ipynb) guide will be useful for converting Variable types to Woodwork types.\n", "\n", "## Physical Types \n", "Physical types define how the data in a Woodwork DataFrame is stored on disk or in memory. You might also see the physical type for a column referred to as the column’s `dtype`.\n", "\n", "Knowing a Woodwork DataFrame's physical types is important because Pandas, Dask, and Spark rely on these types when performing DataFrame operations. Each Woodwork `LogicalType` class has a single physical type associated with it.\n", "\n", "## Logical Types\n", "Logical types add additional information about how data should be interpreted or parsed beyond what can be contained in a physical type. In fact, multiple logical types have the same physical type, each imparting a different meaning that's not contained in the physical type alone.\n", "\n", "In Featuretools, a column's logical type informs how data is read into an EntitySet and how it gets used down the line in Deep Feature Synthesis.\n", "\n", "Woodwork provides many different logical types, which can be seen with the `list_logical_types` function." ] }, { "cell_type": "code", "execution_count": null, "id": "497712b0", "metadata": {}, "outputs": [], "source": [ "import featuretools as ft\n", "\n", "ft.list_logical_types()" ] }, { "cell_type": "markdown", "id": "cfe99d0f", "metadata": {}, "source": [ "Featuretools will perform type inference to assign logical types to the data in EntitySets if none are provided, but it is also possible to specify which logical types should be set for any column (provided that the data in that column is compatible with the logical type).\n", "\n", "To learn more about how logical types are used in EntitySets, see the [Creating EntitySets](using_entitysets.ipynb) guide.\n", "\n", "To learn more about setting logical types directly on a DataFrame, see the Woodwork guide on [working with Logical Types](https://woodwork.alteryx.com/en/stable/guides/working_with_types_and_tags.html#Working-with-Logical-Types). \n", "\n", "## Semantic Tags\n", "Semantic tags provide additional information to columns about the meaning or potential uses of data. Columns can have many or no semantic tags. Some tags are added by Woodwork, some are added by Featuretools, and users can add additional tags as they see fit.\n", "\n", "To learn more about setting semantic tags directly on a DataFrame, see the Woodwork guide on [working with Semantic Tags](https://woodwork.alteryx.com/en/stable/guides/working_with_types_and_tags.html#Working-with-Semantic-Tags). \n", "\n", "### Woodwork-defined Semantic Tags\n", "\n", "Woodwork will add certain semantic tags to columns at initialization. These can be standard tags that may be associated with different sets of logical types or index tags. There are also tags that users can add to confer a suggested meaning to columns in Woodwork.\n", "\n", "To get a list of these tags, you can use the `list_semantic_tags` function." ] }, { "cell_type": "code", "execution_count": null, "id": "11f25bd9", "metadata": {}, "outputs": [], "source": [ "ft.list_semantic_tags()" ] }, { "cell_type": "markdown", "id": "29222810", "metadata": {}, "source": [ "Above we see the semantic tags that are defined within Woodwork. These tags inform how Featuretools is able to interpret data, an example of which can be seen in the `Age` primitive, which requires that the `date_of_birth` semantic tag be present on a column.\n", "\n", "The `date_of_birth` tag will not get automatically added by Woodwork, so in order for Featuretools to be able to use the `Age` primitive, the `date_of_birth` tag must be manually added to any columns to which it applies.\n", "\n", "### Featuretools-defined Semantic Tags\n", "\n", "Just like Woodwork specifies semantic tags internally, Featuretools also defines a few tags of its own that allow the full set of Features to be generated. These tags have specific meanings when they are present on a column.\n", "\n", "- `'last_time_index'` - added by Featuretools to the last time index column of a DataFrame. Indicates that this column has been created by Featuretools.\n", "- `'foreign_key'` - used to indicate that this column is the child column of a relationship, meaning that this column is related to a corresponding index column of another dataframe in the EntitySet.\n", "\n", "\n", "## Woodwork Throughout Featuretools\n", "\n", "Now that we've described the elements that make up Woodwork's type system, lets see them in action in Featuretools.\n", "\n", "### Woodwork in EntitySets\n", "For more information on building EntitySets using Woodwork, see the [EntitySet guide](using_entitysets.ipynb).\n", "\n", "Let's look at the Woodwork typing information as it's stored in a demo EntitySet of retail data:" ] }, { "cell_type": "code", "execution_count": null, "id": "bd9c1ec9", "metadata": {}, "outputs": [], "source": [ "es = ft.demo.load_retail()\n", "es" ] }, { "cell_type": "markdown", "id": "267880c4", "metadata": {}, "source": [ "Woodwork typing information is not stored in the EntitySet object, but rather is stored in the individual DataFrames that make up the EntitySet. To look at the Woodwork typing information, we first select a single DataFrame from the EntitySet, and then access the Woodwork information via the `ww` namespace:" ] }, { "cell_type": "code", "execution_count": null, "id": "aa1966fd", "metadata": {}, "outputs": [], "source": [ "df = es[\"products\"]\n", "df.head()" ] }, { "cell_type": "code", "execution_count": null, "id": "164b1138", "metadata": {}, "outputs": [], "source": [ "df.ww" ] }, { "cell_type": "markdown", "id": "4bffac54", "metadata": {}, "source": [ "Notice how the three columns showing this DataFrame's typing information are the three elements of typing information outlined at the beginning of this guide. To reiterate: By defining physical types, logical types, and semantic tags for each column in a DataFrame, we've defined a DataFrame's Woodwork schema, and with it, we can gain an understanding of the contents of each column.\n", "\n", "This column-specific typing information that exists for every column in every DataFrame in an EntitySet is an integral part of Deep Feature Synthesis' ability to generate features for an EntitySet.\n", "\n", "### Woodwork in DFS\n", "As the units of computation in Featuretools, Primitives need to be able to specify the input types that they allow as well as have a predictable return type. For an in-depth explanation of Primitives in Featuretools, see the [Feature Primitives](primitives.ipynb) guide. Here, we'll look at how the Woodwork types come together into a `ColumnSchema` object to describe Primitive input and return types.\n", "\n", "Below is a Woodwork `ColumnSchema` that we've obtained from the `'product_id'` column in the `products` DataFrame in the retail EntitySet." ] }, { "cell_type": "code", "execution_count": null, "id": "349e5274", "metadata": {}, "outputs": [], "source": [ "products_df = es[\"products\"]\n", "product_ids_series = products_df.ww[\"product_id\"]\n", "column_schema = product_ids_series.ww.schema\n", "column_schema" ] }, { "cell_type": "markdown", "id": "8e8c0ccf", "metadata": {}, "source": [ "This combination of logical type and semantic tag typing information is a `ColumnSchema`. In the case above, the `ColumnSchema` describes the **type definition** for a single column of data. \n", "\n", "Notice that there is no physical type in a `ColumnSchema`. This is because a `ColumnSchema` is a collection of Woodwork types that doesn't have any data tied to it and therefore has no physical representation. Because a `ColumnSchema` object is not tied to any data, it can also be used to describe a **type space** into which other columns may or may not fall.\n", "\n", "This flexibility of the `ColumnSchema` class allows `ColumnSchema` objects to be used both as type definitions for every column in an EntitySet as well as input and return type spaces for every Primitive in Featuretools.\n", "\n", "Let's look at a different column in a different DataFrame to see how this works:" ] }, { "cell_type": "code", "execution_count": null, "id": "f3bb3ffe", "metadata": {}, "outputs": [], "source": [ "order_products_df = es[\"order_products\"]\n", "order_products_df.head()" ] }, { "cell_type": "code", "execution_count": null, "id": "1aae3378", "metadata": {}, "outputs": [], "source": [ "quantity_series = order_products_df.ww[\"quantity\"]\n", "column_schema = quantity_series.ww.schema\n", "column_schema" ] }, { "cell_type": "markdown", "id": "f067db9a", "metadata": {}, "source": [ "The `ColumnSchema` above has been pulled from the `'quantity'` column in the `order_products` DataFrame in the retail EntitySet. This is a **type definition**. \n", "\n", "If we look at the Woodwork typing information for the `order_products` DataFrame, we can see that there are several columns that will have similar `ColumnSchema` type definitions. If we wanted to describe subsets of those columns, we could define several `ColumnSchema` **type spaces**" ] }, { "cell_type": "code", "execution_count": null, "id": "bc2bfae6", "metadata": {}, "outputs": [], "source": [ "es[\"order_products\"].ww" ] }, { "cell_type": "markdown", "id": "73257dcf", "metadata": {}, "source": [ "Below are several `ColumnSchema`s that all would include our `quantity` column, but each of them describes a different type space. These `ColumnSchema`s get more restrictive as we go down:\n", "\n", "##### Entire DataFrame\n", "No restrictions have been placed; any column falls into this definition. This would include the whole DataFrame." ] }, { "cell_type": "code", "execution_count": null, "id": "f6614c98", "metadata": {}, "outputs": [], "source": [ "from woodwork.column_schema import ColumnSchema\n", "\n", "ColumnSchema()" ] }, { "cell_type": "markdown", "id": "299fc7d2", "metadata": {}, "source": [ "An example of a Primitive with this `ColumnSchema` as its input type is the `IsNull` transform primitive.\n", "\n", "##### By Semantic Tag\n", "Only columns with the `numeric` tag apply. This can include Double, Integer, and Age logical type columns as well. It will not include the `index` column which, despite containing integers, has had its standard tags replaced by the `'index'` tag." ] }, { "cell_type": "code", "execution_count": null, "id": "16c1a5a9", "metadata": {}, "outputs": [], "source": [ "ColumnSchema(semantic_tags={\"numeric\"})" ] }, { "cell_type": "code", "execution_count": null, "id": "0932d05d", "metadata": {}, "outputs": [], "source": [ "df = es[\"order_products\"].ww.select(include=\"numeric\")\n", "df.ww" ] }, { "cell_type": "markdown", "id": "a5ec95c8", "metadata": {}, "source": [ "And example of a Primitive with this `ColumnSchema` as its input type is the `Mean` aggregation primitive.\n", "\n", "##### By Logical Type\n", "Only columns with logical type of `Integer` are included in this definition. Does not require the `numeric` tag, so an index column (which has its standard tags removed) would still apply." ] }, { "cell_type": "code", "execution_count": null, "id": "79bd3d4f", "metadata": {}, "outputs": [], "source": [ "from woodwork.logical_types import Integer\n", "\n", "ColumnSchema(logical_type=Integer)" ] }, { "cell_type": "code", "execution_count": null, "id": "e905229e", "metadata": {}, "outputs": [], "source": [ "df = es[\"order_products\"].ww.select(include=\"Integer\")\n", "df.ww" ] }, { "cell_type": "markdown", "id": "2f752200", "metadata": {}, "source": [ "##### By Logical Type and Semantic Tag\n", "The column must have logical type `Integer` and have the `numeric` semantic tag, excluding index columns." ] }, { "cell_type": "code", "execution_count": null, "id": "6da51b75", "metadata": {}, "outputs": [], "source": [ "ColumnSchema(logical_type=Integer, semantic_tags={\"numeric\"})" ] }, { "cell_type": "code", "execution_count": null, "id": "a96d92f6", "metadata": {}, "outputs": [], "source": [ "df = es[\"order_products\"].ww.select(include=\"numeric\")\n", "df = df.ww.select(include=\"Integer\")\n", "df.ww" ] }, { "cell_type": "markdown", "id": "71e0359b", "metadata": {}, "source": [ "In this way, a `ColumnSchema` can define a type space under which columns in a Woodwork DataFrame can fall. This is how Featuretools determines which columns in a DataFrame are valid for a Primitive in building Features during DFS.\n", "\n", "Each Primitive has `input_types` and a `return_type` that are described by a Woodwork `ColumnSchema`. Every DataFrame in an EntitySet has Woodwork initialized on it. This means that when an EntitySet is passed into DFS, Featuretools can select the relevant columns in the DataFrame that are valid for the Primitive's `input_types`. We then get a Feature that has a `column_schema` property that indicates what that Feature's typing definition is in a way that lets DFS stack features on top of one another.\n", "\n", "In this way, Featuretools is able to leverage the base unit of Woodwork typing information, the `ColumnSchema`, and use it in concert with an EntitySet of Woodwork DataFrames in order to build Features with Deep Feature Synthesis." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.2" } }, "nbformat": 4, "nbformat_minor": 5 }