Our Feature Store Journey at Launchpad
by: Pankaj Sahai
The modern data science stack has significantly evolved over the last few years. The Machine Learning Community has consolidated a set of frameworks that make the process of training and productionalizing machine learning models more efficient. Furthermore, these frameworks significantly help your ML model’s continued performance over time. Ultimately, for your ML models to be successful in production, it requires an ecosystem of tools and frameworks that come together to help unlock their business potential. One such framework that can have an outsize impact on your ML business endeavors is the Feature Store. In this blog, we will be performing a deep dive on Features Stores -- what they are, what problems they help to solve, and why your business needs one.
Feature Stores are part of a family of ‘ML stores’ -- ML frameworks whose ultimate goal is to create patterns that make productionalizing machine models easy -- or at least easier. They do this by 1. Bringing standardization across your ML team; 2. Increasing discoverability and reusability of your team’s features and artifacts. 3. Reducing silos within teams and across roles. 4. And ensuring that your ML models continue achieving peak performance over time. Ultimately, this translates into increased business revenue by significantly reducing the time it takes from having a business idea to obtaining a productionalized machine learning model that can take maximum advantage.
So what exactly is a feature store? Before we answer that question, let’s first talk about the challenges ML teams face during the lifecycle of productionalizing a model.
Figure 1: The typical lifecycle of productionalizing a machine learning. Feature management in development and production can take as much as 70% of the time.
After conceiving a good business problem that machine learning can solve, the next step in your ML journey usually starts with looking for data to train your model. The process of obtaining your data, preprocessing it, and transforming it into a feature that your model can use, is known to take the longest in the ML lifecycle -- as much as 70% of the Data Scientists’ time. Interestingly enough, there is a good chance that some or many of these features are already getting used in some shape or form within your team. However, a lack of standardization and discoverability ultimately makes the data scientists decide to build their own – or at least glue together a set of data pipelines from which to train their models. Once you have a set of features, you start to train your model and iterate between different feature sets. Finally, your team has a trained model, but in reality, your journey is just starting. The product is not the model but the end-to-end pipeline, so the next step is usually a handoff between the data scientists and engineering. The engineers in charge of productionalizing the model, more likely than not, end up creating their data pipelines to meet the constraints of serving features in real-time, as well as not having to depend on a data warehouse because of their limited governance. Once the production pipeline is live and connected to the machine learning model, the final step is to ensure that the features your model was trained on continue to be healthy over time. If there are any shifts in the data distribution or corruption of it, your team should be able to pick it up as soon as possible. The key here is to identify the underlying data issue in real time instead of identifying it much later when your business loses revenue. To summarize, there is a lot of complexity in preprocessing and transforming features, managing those features during development and production, and ensuring that those features continue to stay healthy over time.
With that backdrop, we can now dive into what a feature store is. Feature store is a framework that allows teams to manage an enterprise set of features by bringing standards and making them discoverable across your team. With this framework in place, your team can easily have a productionalized model within weeks or even days by taking advantage of the ready-to-consume features that others in your team have already created.
Figure 2: A diagram of the Feature Store Modules (large light-blue rectangle). Open-sourced and enterprise feature stores vary in the type of solutions they offer in managing features for an ML organization.
Let’s dive a bit more into the details. What a feature store is, is in fact a moving target. Today, there are many important differences between open-source and enterprise feature stores. To better understand these differences, let’s dive into the different modules that make up a feature store.
- Serving: Delivering a consistent view of your features during training and when serving your model. For training your model, bring point-in-time correct features that are both fast and correct (pretty hard in practice). On the serving side of things, serving fresh features in real-time and at scale.
- Feature Registry: This is an interface that can allow users to interact with a single source of truth. For users, it contains the definition of what a feature is, how the feature is preprocessed and transformed, what its refreshed cadence is, and any additional metadata that can be used to discover new features and share and reuse them across users.
- Storage: This module is where things start to diverge -- Some feature stores, such as enterprise-scale Tecton, offer both online and offline storage layers to store your features. Others, such as open-source Feast, simply act as an abstraction layer that connects to any data warehouse and online data store your team is already using.
- Monitoring: More and more feature stores are offering this critical capability, from just having simple summary statistics on your features and creating alerts if there are drifts to more complex drift detection that can even consider seasonality. Feature stores that currently allow you to do some form of feature health management include Hopsworks and Tecton.
- Transformation: Finally, transformation -- Some feature stores are drawing a hard line between not being involved in most of the engineering aspects of transforming your raw data into features but instead connecting to a 3rd party tool that can do so. However, other feature stores, such as Kaskada and Tecton, are bringing the transformation module into the purview of what a feature store should include. In this way, they manage not only your features but also your data pipelines. In return, ML organization reaps additional efficiencies by reducing the computation cost of transforming features only once and ending up with a single source of truth along the way.
To summarize, Feature Store core modules, as shown in Figure 2, including online and offline serving and feature registry. These modules ensure consistency between training and prediction by having a unified API and making features reusable and scalable. Storage can be found in many enterprise feature stores. By contrast, open-source feature stores, such as feast, can simply connect to any online and offline data store your team is already using. Feature monitoring is critically important, and many feature stores are starting to offer some form of feature health management. Lastly, the transformation module could be considered a slight shift in paradigm, where your feature store is not only managing your features but also your data pipelines.
Figure 3: Managing the complexity of ML features in an enterprise can be daunting. In this simple example on the left, we can observe how a few features with a few users can easily create a data pipeline jungle. A feature store framework (diagram on the left) helps manage this complexity at scale.
Finding your Goldilocks Feature Store:
What solution your team ends up with depends on a combination of factors. These include the number of features and the number of users that your business currently has. If you have, for example, billions of users, you will need a solution that scales between online and offline serving. How big is your team, and is it willing to do some infrastructure work? Going with open-sourced solution will require some engineering effort on your part, something you wouldn’t have to deal with a managed service. Is your team mostly making batch predictions? Is this something that will continue to be accurate in the near future? Thinking about your current and future needs will help you land on a solution that not only scales but will meet the future demands of your business. Below find a comparison between managed and open-sourced feature store solutions.
