What is GrowthBook?

GrowthBook is a modular platform. You can use it for either Feature Flags, Experiment Analysis, or both.

Feature Flags you create in the GrowthBook UI are published to our API as a JSON file. Pass this JSON into our SDKs and use the feature flags throughout your code. If a feature is running as part of an experiment, we'll fire a tracking callback in the SDK so you can record that event in your data warehouse or analytics platform for later analysis.

Experiment Analysis queries your data warehouse for raw experiment data and runs it through our stats engine to produce a report. No raw user-level events or PII are ever sent to GrowthBook. We only get back aggregate info instead (sums, sums of squares, etc.).

Feature Flags​

Feature flags are a very powerful developer tool. They give you deep control over how and when new functionality is released to your users.

GrowthBook supports 4 types of features:

• Boolean (on/off)
• Number
• String
• JSON (with JSON Schema validation)

Using features in your code is easy. Here's an example from our Javascript SDK:

// For boolean features
if (growthbook.isOn("my-feature")) {
  // ... Do something
}

// For number, string, and JSON features
const value = growthbook.getFeatureValue("other-feature", "fallback");

GrowthBook also supports multiple environments, so you can, for example, have a feature enabled in dev but not production.

You can also change the value of a feature with Override Rules. The following types of rules are supported:

• Forced Value - Choose a subset of users based on targeting attributes and assign them all the same value
• Percentage Rollout - Use random sampling to roll out a new feature value to a percent of users
• Safe Rollout - Gradually ramp up traffic while monitoring guardrail metrics and automatically roll back if things go wrong
• Experiment - Run a controlled hypothesis test between 2 or more feature values
• Multi-Arm Bandit - Dynamically allocate traffic to different feature values based on performance

All features for an environment are packaged together into a single JSON file. All you need to do is pass this into our SDKs along with user targeting attributes.

Read more about features here.

Experiment Analysis​

GrowthBook needs to connect to your SQL Data Warehouse in order to query experiment results. GrowthBook is extremely flexible and can support almost any schema structure with a little bit of configuration.

Once connected to a data source, you need to define Fact Tables and Metrics. Fact Tables are SQL queries that select raw data from your warehouse. Metrics are built on top of these Fact Tables and define what you want to measure in your experiments. For example, you might have an Orders fact table that selects one row per order, and then metrics for Average Order Value, Total Revenue, and Number of Orders.

GrowthBook supports many types of metrics:

• Proportion - A simple conversion rate (e.g. signed up, purchased).
• Mean - The average value across experiment units (e.g. revenue, page views). Supports several different aggregation types: sum, count, max, and count distinct.
• Ratio - A ratio of two values (e.g. pages per session, revenue per order).
• Quantile - The value at a specific percentile (e.g. P99 Latency).
• Retention - The percentage of experiment units who return after a delay (e.g. day 7 retention).
• Funnel - The percentage of experiment units who complete a series of steps (e.g. signup funnel conversion). coming soon

Once you create a library of metrics, you can reuse them across multiple experiments.

GrowthBook has an extremely powerful and highly configurable statistics engine for analyzing experiment results. Below are some of the key features and options:

• Frequentist or Bayesian - Choose the statistical framework that best fits your needs
• Sequential Testing - Always-valid p-values that allow you to safely peek at results as often as you like
• CUPED - Use pre-experiment data to reduce variance and improve sensitivity
• Sample Ratio Mismatch (SRM) - Automatically detect when experiment traffic is not being properly split between variations
• Multiple Comparison Corrections - Control for either false positive rate (Holm-Bonferroni) or false discovery rate (Benjamini-Hochberg)
• Different Baselines - Choose any variation as the baseline for comparison, not just control
• Custom Priors - For Bayesian stats, set custom prior distributions for each metric
• Power Calculator - Estimate the required runtime based on historical data before launching your experiment
• Difference Types - View metric differences as either relative lift, absolute lift, or scaled impact

Your data team can drill down into results by custom dimensions, view the raw SQL that GrowthBook is running on your data warehouse, and export results to a Jupyter notebook for even deeper analysis.

Read more about experiments here.

Use Cases​

There are typically three reasons that teams use GrowthBook.

1. Full Experimentation Platform​

In this use case, companies use Feature Flags and our SDKs to run experiments in their applications. Then they use our Experiment Analysis to look at the results and decide on a winner.

This is best for companies that want an all-in-one solution and seamless developer experience.

2. Feature Flags Only​

In this use case, companies don't run experiments at all and just use GrowthBook feature flags within their engineering team.

This is best for companies that don't have enough traffic to run full experiments, but still want all of the benefits that feature flags provide. It's also good for companies that know they will want to run experiments in the future and want to start instrumenting their applications today to get ready.

3. Experiment Analysis Only​

In this use case, companies are already running experiments and analyzing results usually with either a home-built reporting system or by manually creating Jupyter notebooks. They use GrowthBook to automate and improve the analysis process to save time and make better decisions.

This is best for companies that already have a robust process for running experiments and just need a little help analyzing results at scale.

Next Steps​