Ensemble model using an image captioning and ranking example

Run in Google Colab View source on GitHub

When performing complex tasks like image captioning, using a single ML model may not be the best solution.

This notebook shows how to implement a cascade model in Apache Beam using the RunInference API. The RunInference API enables you to run your Beam transforms as part of your pipeline for optimal machine learning inference.

For more information about the RunInference API, review the RunInference notebook or the Beam ML documentation.

Image captioning with cascade models

Image captioning has various applications, such as image indexing for information retrieval, virtual assistant training, and natural language processing.

This example shows how to generate captions on a a large set of images. Apache Beam is the ideal tool to handle this workflow. We use two models for this task:

  • BLIP: Generates a set of candidate captions for a given image.
  • CLIP: Ranks the generated captions based on accuracy.