icon

Feature flags in a Continuous Delivery pipeline

Icon

Andrii Gikalo

Andrii is a passionate hands-on engineering leader that finds it equally joyful to help team members grow and find new brilliant ways of solving problems using technology 🙌

In this story I will cover how we automated management of feature flags using one of the feature flag management vendors. And created safe and reliable system to sync feature flags with codebase of the large app. With a goal to reduce risk of misconfiguring or misusing flags and remove manual steps as much as possible.

Feature flags were integrated into Node.js application built with Apollo GraphQL and Typescript. This is important to mention both of these tech details for context of the solution (hint: we used type checks from TS and GraphQL schema to communicate availability of certain flags).

Requirements

Design and implement an approach that will allow us to separate deployment from release. That means to enable or disable the feature we won’t need a code change and deployment, but a developer, QA, PO, or others can do that directly from the service's control panel.

Solution must be expandable and portable. It must be straightforward and easy to swap a vendor if we want to do so in the future.

What are feature flags?

Feature flags are a combination of patterns and approaches that allows teams to separate deployment and release by providing an ability to enable or disable a functionality.

Current solution

The current solution is based on the static files that live with the repo. If we want to enable or disable a functionality we need to update the corresponding feature flag in a codebase and then deploy it. This solution has several downsides.

Downsides of the current solution

  1. Feature flags and segmentation for all environments are declared in a single deployment template.
  2. Updating the deployment version is a code change and a new deployment of the app. That means to enable or disable feature code change and new deployment are required.
  3. The current solution only supports boolean values - no variations.

New solution

The decision was made to use the LaunchDarkly service for managing feature flags and to configure roll out of new features. LaunchDarkly was selected because of the experience with this service, however there's plenty more to select from, like ConfigCat, FlagShip and many others.

We implemented the developer-first approach for working with feature flags. It means that feature flags are introduced by developers while coding, created via the LaunchDarkly API during the CI pipeline, and used via the LaunchDarkly control panel by developers, QAs, PMs, Ops, and others.

New way for creating a feature flag

  1. The developer starts implementing a new feature, and he should hide it under a feature flag.
  2. The developer introduces a new feature flag and defines it inside our zeta_graphql application in the feature-flags.yml file. Below are an example file, a description of its format, and a JSON schema.
  3. The developer runs the command to generate TypeScript typings for the newly created feature flag: yarn generate-flags.
  4. The developer uses the flag inside the codebase to hide feature implementation.
  5. The developer creates a merge request that gets merged into the master branch.
  6. New feature flags are created in the LaunchDarkly during the execution of a CI pipeline on a master branch.
  7. Other developers, QA, PO, and others can set up segments and configure targeting of the feature in the LaunchDarkly control panel.

Feature flags manifest file

A developer needs to define new feature flags in the feature-flags.yml file in the root of the zeta_graphql repo. An example of the file’s content:

isNewProductPageEnabled:
  name: Enable new product page layout
  variations:
    - value: false
      name: Disabled
    - value: true
      name: Enabled
  default: 0
isSingleSignOnEnabled:
  variations:
    - value: false
    - value: true
  default: 0
productRecommendationsCount:
  name: Number of recommended products to show on a page
  variations:
    - value: 10
    - value: 20
    - value: 30
  default: 10

Feature flags are described as objects in YAML. Where the top-level key is a feature flag key that will be used in the codebase (isNewProductPageEnabled, isSingleSignOnEnabled, and productRecommendationsCount). A feature flag key must match the pattern ^([a-z][a-z0-9])(-.[a-z0-9]+)$.

The variations[].value and default properties are required. And the name and variations[].name are optional.

The variations[].value can be one of type boolean number string.

The default property is of type number and indicates an index of a default variation in the variations array.

Below is a JSON schema to validate feature flags manifest object:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "http://example.com/example.json",
  "type": "object",
  "patternProperties": {
    "^([a-z][a-z0-9]*)(-[a-z0-9]+)*$": {
      "type": "object",
      "properties": {
        "variations": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "value": {
                "oneOf": [
                  { "type": "boolean" },
                  { "type": "number" },
                  { "type": "string" }
                ]
              },
              "name": { "type": "string" },
              "description": { "type": "string" }
            },
            "required": ["value"]
          }
        },
        "default": {
          "type": "number"
        },
        "name": {
          "type": "string"
        }
      },
      "required": ["variations", "default"]
    }
  }
}

Using feature flags from client perspective

There is a single source of truth and the only place where various clients (Web client, iOS and Android Apps) can query feature flags from GraphQL schema of zeta_graphql app.

GraphQL schema will expose the featureFlags field that contains all available feature flags with computed values.

All requests that come to zeta_graphql have the context of the current user using the app. This context is then used to evaluate feature flags via the LaunchDarkly SDK to create personalized experience.

This allows us to segment customers by such info:

  • User ID
  • User Segment
  • Locale
  • Currency
  • any other info we know about a customer can be added here over time

There are several GraphQL clients that already use our schema, and there is also a mechanism to distinguish one from another and to segment customers by the application they use.

The GraphQL server expects clients to send the x-zeta-client header in each incoming HTTP request. And a value of that header is used to differentiate clients.

Consider following GraphQL clients:

  • Zeta Web (web responsive product search and checkout).
  • Android app.
  • iOS app.

The GraphQL server then uses the value of the x-zeta-client header in addition to other customer info when evaluating feature flag value.

picture

Example of GraphQL results:

{
  "data": {
    "featureFlags": {
      "isNewProductPageEnabled": true,
      "isSingleSignOnEnabled": false,
      "productRecommendationsCount": 30
    }
  }
}

Lifecycle of a feature flag

  1. Feature flag definition.
  2. TypeScript typings.
  3. Feature implementation.
  4. Feature flag creation.
  5. Targeting configuration.
  6. Retrieving feature flags.
  7. Archiving feature flags.

Let’s consider the case of creating, evaluating, receiving, and archiving a feature flag that enables or disables product search via Omega Product Search microservice.

Feature flag definition

A developer who works on integrating Omega Product Search into item search flow knows that the functionality should be hidden at the beginning and then it will be rolled out to customers progressively. At this point, the developer defines a unique flag key, possible values, and a default value.

The developer describes the flag in the feature-flags.yml file:

isNewProductPageEnabled:
  name: Product Search via Omega Product Search
  variations:
    - value: false
      name: Disabled
    - value: true
      name: Enabled
  default: 0

The listing above tells us that there is the isNewProductPageEnabled feature flag that can be either true or false, and if the flag is not enabled on an environment it’ll be false (default value).

TypeScript typings

The zeta_graphql is written in TypeScript and developers should have a mechanism of type checking for existing feature flags, including flag keys and possible variations to make sure there's no room for human mistake!

We need to provide TypeScript build engine with typings for our feature flag. It can parse the feature-flags.yml so we need to generate typings ourselves by executing yarn generate-flags command.

This command parses the feature-flags.yml file, generates typings, and emits them to the src/feature-flags.d.ts file which gets picked up by TypeScript automatically.

Below are few scenarios to demonstrate power of this combination

Here developer tries to pass 'yes' instead of boolean value, and Typescript will reject it at build time.

const isNewProductSearchEnabled = evalFeatureFlag(
  'isNewProductSearchEnabled',
  'yes', // 👈 this is incorrect value
  //Arguments of type 'string' is not assignable to parameter of type 'boolean'.ts(2345)
)

In this case developer tries to use a flag that is not present in a Feature Flags manifest file.

const isSsoEnabled = evalFeatureFlag(
  'isSsoEnabled', // 👈 this flag doesn't exist in manifest file
  // Argument of type '"isSingleSignOnEnabled"' is not assignable to parameter of type '"isNewProductSearchEnabled" | "productResultCount" | "isSingleSignOnEnabled"'.ts(2345)
  false,
)

A developer tried to store evaluation results in a variable of an invalid type.

const productsCount: string = evalFeatureFlag('productResultCount', 10) // 👈 this flag is number and not a string
// Type 'number' is not assignable to type 'string'.ts(2322)

Feature implementation

Let’s assume that there is already a working item search feature which calls the eCommerce API. The aim of the developer's work is to add functionality to query items from the Omega Product Search.

A developer knows that the functionality of querying products from the Omega Product Search should be hidden from customers at the very beginning, but it shouldn’t block application deployments and releases of other functionalities for teams that adopted Continuous Delivery principles. Moreover, a developer understands that it will be released to customers progressively.

A developer evaluates the feature flag to choose a data source:

products() {
  const dataSource = evalFeatureFlag( 'isNewProductSearchEnabled', false)
    ? new NewProductSearchService()
    : new ProductApi()
  return dataSource.getProducts()
}

The code above demonstrates that the feature flag evaluation in the code is as simple as passing a feature flag key and default value.

Creating feature flag

Once a developer finished his work and it gets merged into the master branch, the CI pipeline starts checking and building the application. The pipeline includes a job for calling the LaunchDarkly API in order to create feature flags introduced during feature implementation.

Back to our example, the CI pipeline job will call the LaunchDarkly API with the request to create a new feature flag with the isNewProductPageEnabled key, false and true variations, and the default value to be false.

After the successful CI pipeline, feature flags introduced by the developer are ready to be used in the LaunchDarkly control panel.

Targeting configuration

At this stage, a developer, QA, PO, Ops, and others who have access to the dashboard can configure targeting for the feature in a LaunchDarkly dashboard.

LaunchDarkly provides powerful capabilities for rolling out features by configuring its targeting for individual users or segments of users.

Retrieving feature flags

All clients should call the GraphQL server in order to retrieve all available feature flags. The GraphQL server is a single source of truth and it returns already evaluated feature flags for a specific client, partner, site, user, etc.

Let’s consider the following preconditions:

  • There are three clients:
    • Zeta Web (web responsive product search and checkout).
    • Android app.
    • iOS app.
  • A variety of user IDs.
  • Handful of user segments.
  • There are a lot of other user attributes (currency, locale, etc).

The GraphQL server will expose the new featureFlags field:

type FeatureFlags {
  isNewProductPageEnabled: Boolean!
  isSingleSignOnEnabled: Boolean!
  productRecommendationsCount: Int!
}

And clients can query only feature flags they are interested in:

query getFeatureFlags {
  featureFlags {
    isNewProductPageEnabled
    productRecommendationsCount
  }
}

Archiving feature flags

When functionality is rolled out to 100% of the audience and a feature flag is no longer needed, it can be archived in the LaunchDarkly control panel. Then, when a developer will execute yarn generate-flag next time, archived feature flags will be removed from the manifest file and TypeScript typings won’t be generated for them.

TypeScript will detect all references of removed feature flags and emit compilation errors about the usage of a non-existing feature flag.

FAQ

Q: What if the flag is added to feature-flags.yml but yarn generate-flags wasn’t executed?

A: The yarn generate-flags command generates TypeScript typings or feature flags described in the feature-flags.yml, so developers can safely use them in the codebase by having strict type checking and autocomplete for available variations. In case a new feature flag is added to the feature-flags.yml but yarn generate-flags wasn’t executed, a developer won’t have access to the feature flag in the code - TypeScript will emit an error that the feature flag doesn’t exist.

Important to note that in the case above a developer won’t be able to use the feature flags in the code, but that flag will be successfully created in the LaunchDarkly.

Q: What if the flag is removed from feature-flags.yml but yarn generate-flags wasn’t executed?

A: In such a case TypeScript will not emit an error that a feature flag doesn’t exist anymore, and a developer will be able to use it in the code. However, the removed feature flag won’t be removed from the LaunchDarkly. This is needed to protect feature flags to be accidentally removed by developers.

Q: What happens if the flag is presented in feature-flags.yml and yarn generate-flags was executed, but during CI stage we failed to create a flag in LaunchDarkly via API?

A: Since a feature flag has been introduced in the feature-flags.yml and the yarn generate-flags command was executed that means the flag is ready to be used in code. A developer now can evaluate it and fully rely on TypeScript type-checking.

Once the CI stage failed and the flag hasn’t been created in the LaunchDarkly we need to warn developers about the issue, so they will make a decision about how to handle it. There are several possible approaches:

  • retry failed job.
  • leave it as is - it will be sent to the LaunchDarkly again during the next CI pipelines and the code will use default value defined in manifest.

Q: What is the strategy to support multiple clients of GraphQL API?

A: The strategy is described in the Retrieving feature flags section. But long story short, the zeta-graphql app expects clients to send x-zeta-client header in each incoming HTTP request. The value of that header should be unique for each type of the client.

The GraphQL server via the LaunchDarkly SDK will use that info when evaluating the feature flags, so customers can be split into segments by a client type.