Experiments

About Experimentation

A/B/n testing is the practice of comparing multiple variations of a web page or feature to assess which version performs best against a given goal, such as conversion rate, user engagement, or time on page.

By systematically rolling out these variations to segments of users and measuring real-world performance, you gain data-driven insights that help you fine-tune product design, user experience, and messaging.

Why Experiments Matter

• Data-Driven Decisions: Instead of making changes blindly, Experiments use real user data to validate new features or design variations.

• Objective Measurement: Metrics track the numeric values or conversion rates of user actions, eliminating guesswork.

• Continuous Improvement: Experiments allow iterative testing of multiple variants, so you can constantly refine your product or feature.

What is a FlagSync Experiment?

Experiments in FlagSync unify feature flags, events, and metrics into a powerful testing framework.

An experiment in FlagSync is a structured way to test different flag variants (e.g., “baseline” vs. “new changes”) against one or more Metrics.

By correlating Flag Impressions (which variant the user saw) and Events (what the user did next), the experiment evaluates whether a new variant outperforms, underperforms, or performs comparably to the baseline.

Best Practices

1. Limit Variables: Only test one hypothesis per experiment. If you change multiple aspects (e.g., button text and layout and discount rate), it’s harder to pinpoint which change caused the result.

2. Sufficient Sample Size: Ensure that enough users see each variant so that the data isn’t skewed by small sample sizes.

3. Clear Goals: Tie your experiments to metrics that genuinely reflect your business objectives (e.g., revenue, sign-up flow, performance benchmarks).

Prerequisites

• SDK Installation

  • Install the FlagSync SDK (client-side or server-side) in your application.

• User Context

  • Provide a unique user identifier (e.g., user ID, user email, or session ID) for the user context.

  • Ensure even anonymous users are given their own unique context keys.

• Flag Evaluation (Impressions)

  • Call the flag function (e.g., client.flag('register-cta')) to serve the correct variant.

  • The SDK automatically sends a flag impression event to FlagSync each time a variant is returned.

• Event Tracking (Calls to track)

  • Integrate the track function wherever you want to measure specific user actions.

  • Decide whether you need a numeric eventValue (e.g., page load time, purchase amount) or key-value properties (e.g., product details).

  • Ensure each event has a meaningful eventKey (e.g., register-cta-click) that will map cleanly to your metric definitions.

• Create Metrics

  • From the FlagSync dashboard, create metrics based on the events you’re tracking.

  • Choose the appropriate measurement type:

    • Conversion Rate if you want to measure how many users triggered an event versus how many saw the feature flag.

    • Numeric (Count, Sum, Average) if you’re tracking values like purchase amounts or performance metrics.

• A Feature Flag With Variants

  • Set up your flag in FlagSync with the variants you plan to test (e.g., “Register,” “Join Now,” “Sign Up”).

  • Determine which variant will be the baseline for your experiment.