A simple experiment library to safely test new code paths. LabCoat is designed to be highly customizable and play nice with your existing tools/services.

This library is heavily inspired by Scientist, with some key differences:

• Experiments are classes, not modules which means they are stateful by default.
• There is no app wide default experiment that gets magically set.
• The Result only supports one comparison at a time, i.e. only 1 candidate is allowed per run.
• The duration is measured using Ruby's Benchmark.
• The final return value of the Experiment run can be selected dynamically.

Install the gem and add to the application's Gemfile by executing:

bundle add lab_coat

If bundler is not being used to manage dependencies, install the gem by executing:

gem install lab_coat

To do some science, i.e. test out a new code path, start by defining an Experiment. An experiment is any class that inherits from LabCoat::Experiment and implements the required methods.

# your_experiment.rb
class YourExperiment < LabCoat::Experiment
  def initialize
    super("expensive_query_experiment")
  end

  def control
    expensive_query.first
  end

  def candidate
    refactored_version_of_the_query.first
  end

  def enabled?
    true
  end
end

The base initializer for an Experiment requires a name argument; it's a good idea to name your experiments.

See the Experiment class for more details.

You might want to give your experiment some context, or state. You can do this via an initializer or writer methods just like any other Ruby class.

# application_experiment.rb
class ApplicationExperiment < LabCoat::Experiment
  def initialize(user)
    @user = user
    @is_admin = user.admin?
  end
end

You might want to publish! all experiments in a consistent way so that you can analyze the data and make decisions. New Experiment authors should not have to redo the "plumbing" between your experimentation framework (e.g. LabCoat) and your observability (o11y) process.

# application_experiment.rb
class ApplicationExperiment < LabCoat::Experiment
  def publish!(result)
    payload = result.to_h.merge(
      user_id: @user.id, # e.g. something from the `Experiment` state
      build_number: context[:version] # e.g. something from the runtime context
    )
    YourO11yService.track_experiment_result(payload)
  end
end

You might have a common way to enable experiments such as a feature flag system and/or common guards you want to enforce application wide. These might come from a mix of services, the Experiment's state, or the runtime context.

# application_experiment.rb
class ApplicationExperiment < LabCoat::Experiment
  def enabled?
    !@is_admin && YourFeatureFlagService.flag_enabled?(@user.id, name)
  end
end

You might want to track any errors thrown from all your experiments and route them to some service, or log them.

# application_experiment.rb
class ApplicationExperiment < LabCoat::Experiment
  def raised(observation)
    YourErrorService.report_error(
      observation.error,
      tags: observation.to_h
    )
  end
end

You might want to rollout the new code path in certain cases.

# application_experiment.rb
class ApplicationExperiment < LabCoat::Experiment
   def select_observation(result)
    if result.matched? || YourFeatureFlagService.flag_enabled?(@user.id, @context[:rollout_flag_name])
      candidate
    else
      super
    end
  end
end

You don't have to create an Observation yourself; that happens automatically when you call Experiment#run!. The control and candidate Observations are packaged into a Result and passed to Experiment#publish!.

The run! method accepts arbitrary keyword arguments, to allow you to set runtime context for the specific run of the experiment. You can access this Hash via the context reader method, or directly via the @context instance variable.

Observation instances are passed to many of the Experiment methods that you may override.

# your_experiment.rb
def compare(control, candidate)
  return false if control.raised? || candidate.raised?

  control.value.some_method == candidate.value.some_method
end

def ignore?(control, candidate)
  # You might ignore runs that throw errors and handle them separately via `raised`.
  return true if control.raised? || candidate.raised?
  # You might ignore runs where the candidate meets some condition.
  return true if candidate.value.some_condition?

  false
end

def publishable_value(observation)
  return nil if observation.raised?

  # Let's say your control and candidate blocks return objects that don't serialize nicely.
  {
    some_attribute: observation.value.some_attribute,
    some_other_attribute: observation.value.some_other_attribute,
    some_count: observation.value.some_array.count
  }
end

# Elsewhere...
YourExperiment.new(...).run!

A Result represents a single run of an Experiment.

The Result is passed to your implementation of #publish! when an Experiment is finished running. The to_h method on a Result is a good place to start and might be sufficient for most experiments. You might want to include additional data such as the runtime context or other state if you find that relevant for analysis.

# your_experiment.rb
def publish!(result)
  return if result.ignored?

  puts result.to_h.merge(context:)
end

You can always access all of the attributes of the Result and its Observations directly to fully customize what your experiment publishing looks like.

# your_experiment.rb
def publish!(result)
  if result.ignored?
    puts "🙈"
    return
  end

  if result.matched?
    puts "😎"
  else
    control = result.control
    candidate = result.candidate
    puts <<~MSG
      😮

      #{control.slug}
      Value: #{control.publishable_value}
      Duration Real: #{control.duration.real}
      Duration System: #{control.duration.stime}
      Duration User: #{control.duration.utime}
      Error: #{control.error&.message}

      #{candidate.slug}
      Value: #{candidate.publishable_value}
      Duration: #{candidate.duration.real}
      Duration System: #{candidate.duration.stime}
      Duration User: #{candidate.duration.utime}
      Error: #{candidate.error&.message}
    MSG
  end
end

Running a mismatched experiment with this implementation of publish! would produce:

😮

my_experiment.control
Value: 420
Duration Real: 12.934
Duration System: 2.134
Duration User: 10.800
Error:

my_experiment.candidate
Value: 69
Duration Real: 9.702
Duration System: 1.002
Duration User: 8.700
Error:

The Observation class can be used as a standalone wrapper for any code that you want to experiment with. Instantiating an Observation automatically:

• measures the duration of the code block
• captures the return value of the code block
• rescues and stores any errors raised by the code block

10.times do |i|
  observation = Observation.new("test-#{i}", nil) do
    some_code_path
  end

  puts "#{observation.name} results:"
  if observation.raised?
    puts "error: #{observation.error.message}"
  else
    puts <<~MSG
      duration: #{observation.duration.real}
      succeeded: #{!observation.raised?}
    MSG
  end
end

After checking out the repo, run bin/setup to install dependencies. Then, run rake test to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and the created tag, and push the .gem file to rubygems.org.

Bug reports and pull requests are welcome on GitHub at https://github.com/omkarmoghe/lab_coat.

The gem is available as open source under the terms of the MIT License.