IDAH Data Annotation

Learn how to add media processing capabilities to your IDAH plugin using the Media Service backend.

Overview

The Media Service backend handles media file processing (images, videos, audio, etc.). It allows you to process original media files and generate thumbnails, previews, or transformations when media is imported into IDAH.

Add Media Backend to Your Plugin

Option 1: Create New Plugin with Media Backend

When creating a new plugin, select "Media Service" during the setup:

npx idah-plugin create my-plugin ./plugins

When prompted, select Media Service from the backend services options.

Option 2: Add Media Backend to Existing Plugin

If you have an existing plugin without a media backend, you can add it:

npx idah-plugin backend add my-plugin ./plugins

When prompted, select Media Service to add media processing capabilities.

Generated File Structure

The media backend generator creates the following files:

<plugin_name>/
   └─ backends/
      └─ media/
         └─ <plugin_name_underscore>/
            ├─ media.rb                                # Media service module (registers processor)
            ├─ media_spec.rb                           # Media service tests
            ├─ processor.rb                            # Core processing logic
            ├─ processor_spec.rb                       # Processor tests
            ├─ options.rb                              # Options schema and validation
            └─ options_spec.rb                         # Options tests

Implementation Steps

Step 1: Register the Processor

The media module registers your processor with IDAH:

backends/media/<plugin_name_underscore>/media.rb

module YourPlugin
  class Media
    def self.init(context)
      context.register_processor(
        "your-plugin",                           # Processor identifier
        class_name: "YourPlugin::Processor",     # Processor class name
        options_class_name: "YourPlugin::Options" # Options class (optional)
      )
    end
  end
end

Step 2: Implement the Processor

Implement the core processing logic in your processor class:

backends/media/<plugin_name_underscore>/processor.rb

module YourPlugin
  class Processor
    def initialize(options)
      @options = options
    end

    def process(context)
      # Download original media file
      input_path = context.download_original

      # Process the media
      output_io = transform_media(input_path)

      # Upload processed result
      context.upload_media(
        output_io,
        "thumbnail.jpg",
        "image/jpeg"
      )

      # Update progress
      context.progress = 100
    end

    private

    def transform_media(input_path)
      # Your processing logic here
      # Return an IO object (e.g., File.open or StringIO)
    end
  end
end

Step 3: Define Options Schema

Define configurable options for your processor:

backends/media/<plugin_name_underscore>/options.rb

require "verse/schema"

module YourPlugin
  class Options < Verse::Schema::Struct
    attribute :quality, Integer, default: 80
    attribute :format, String, default: "webp"
    attribute :width, Integer, default: 1920
    attribute :height, Integer, default: 1080

    def validate
      super
      errors.add(:quality, "must be between 1 and 100") unless (1..100).cover?(quality)
    end
  end
end

Processor Context API

Download Original Media

input_path = context.download_original

Downloads the original media file and returns a temporary file path.

Upload Processed Media

context.upload_media(
  File.open("output.jpg"),  # IO object
  "thumbnail.jpg",          # Key/identifier
  "image/jpeg"              # MIME type
)

context.upload_media(
  StringIO.new(image_data), # StringIO for raw data
  "preview.jpg",
  "image/jpeg"
)

Parameters:

io - IO object (File.open, StringIO)
key - Identifier for the processed media
mime_type - MIME type (e.g., "image/jpeg", "video/mp4")

Update Progress

context.progress = 50   # 50% complete
context.progress = 100  # Done

Handle Errors

context.error!("Processing failed: #{error_message}")

Reschedule Job

context.reschedule!(after: 10)  # Retry after 10 seconds

Complete Example

Here's a complete implementation that generates thumbnails and previews:

Processor Implementation

backends/media/<plugin_name_underscore>/processor.rb

module YourPlugin
  class Processor
    def initialize(options)
      @options = options
    end

    def process(context)
      Verse.logger.info "Processing #{context.resource}"

      # Download original
      input_path = context.download_original

      # Generate thumbnail
      thumbnail = generate_thumbnail(input_path, @options.quality)
      context.upload_media(thumbnail, "thumbnail.jpg", "image/jpeg")
      context.progress = 50

      # Generate preview
      preview = generate_preview(input_path, @options.width, @options.height)
      context.upload_media(preview, "preview.mp4", "video/mp4")
      context.progress = 100

      Verse.logger.info "Processing complete"
    rescue StandardError => e
      Verse.logger.error "Processing failed: #{e.message}"
      context.error!(e.message)
      raise
    end

    private

    def generate_thumbnail(input, quality)
      # Your thumbnail generation logic
      # Return File.open(output_path)
    end

    def generate_preview(input, width, height)
      # Your preview generation logic
      # Return File.open(output_path)
    end
  end
end

Media Service Registration

backends/media/<plugin_name_underscore>/media.rb

module YourPlugin
  class Media
    def self.init(context)
      context.register_processor(
        "your-plugin",
        class_name: "YourPlugin::Processor",
        options_class_name: "YourPlugin::Options"
      )
    end
  end
end

Options Schema

backends/media/<plugin_name_underscore>/options.rb

require "verse/schema"

module YourPlugin
  class Options < Verse::Schema::Struct
    attribute :quality, Integer, default: 80
    attribute :width, Integer, default: 1920
    attribute :height, Integer, default: 1080

    def validate
      super
      errors.add(:quality, "must be between 1 and 100") unless (1..100).cover?(quality)
    end
  end
end

Implementation Workflow

1. Add Media Backend (if not present)

npx idah-plugin backend add image-processor ./plugins

Select "Media Service" when prompted. This generates the media backend structure.

2. Install Dependencies

cd plugins/image-processor bundle install

3. Implement Processing Logic

Edit backends/media/<plugin_name>/processor.rb and implement your transform_media method:

backends/media/<plugin_name_underscore>/processor.rb

def transform_media(input_path)
  output_path = "#{Dir.tmpdir}/output_#{SecureRandom.hex(8)}.jpg"

  # Example: Use ImageMagick to resize and compress
  system(
    "convert",
    input_path,
    "-resize", "800x600>",
    "-quality", @options.quality.to_s,
    output_path
  )

  File.open(output_path, "rb")
end

4. Configure Options

Customize the options schema for your needs:

backends/media/<plugin_name_underscore>/options.rb

require "verse/schema"

module YourPlugin
  class Options < Verse::Schema::Struct
    attribute :quality, Integer, default: 80
    attribute :format, String, default: "webp"
    attribute :max_width, Integer, default: 1920
    attribute :max_height, Integer, default: 1080
    attribute :generate_thumbnail, Boolean, default: true

    def validate
      super

      errors.add(:quality, "must be between 1 and 100") unless (1..100).cover?(quality)
      errors.add(:format, "must be jpg, png, or webp") unless %w[jpg png webp].include?(format)
      errors.add(:max_width, "must be positive") if max_width <= 0
      errors.add(:max_height, "must be positive") if max_height <= 0
    end
  end
end

5. Write Tests

backends/media/<plugin_name_underscore>/processor_spec.rb

require "spec_helper"
require_relative "processor"
require_relative "options"

RSpec.describe YourPlugin::Processor do
  let(:options) { YourPlugin::Options.new(quality: 90) }
  let(:processor) { described_class.new(options) }

  describe "#process" do
    it "processes media successfully" do
      context = double("context")

      allow(context).to receive(:resource).and_return("test-resource")
      allow(context).to receive(:download_original).and_return("/tmp/input.jpg")
      allow(context).to receive(:upload_media)
      allow(context).to receive(:progress=)

      expect { processor.process(context) }.not_to raise_error
      expect(context).to have_received(:upload_media).at_least(:once)
      expect(context).to have_received(:progress=).with(100)
    end
  end
end

6. Run Tests

bundle exec rspec backends/media/

Common Processing Patterns

Image Processing

def process(context)
  input_path = context.download_original

  # Generate multiple sizes
  generate_size(input_path, "thumbnail", 200, 200, context)
  context.progress = 33

  generate_size(input_path, "medium", 800, 800, context)
  context.progress = 66

  generate_size(input_path, "large", 1920, 1920, context)
  context.progress = 100
end

def generate_size(input, name, width, height, context)
  output = resize_image(input, width, height)
  context.upload_media(output, "#{name}.jpg", "image/jpeg")
end

Video Processing

def process(context)
  input_path = context.download_original

  # Extract frame for thumbnail
  thumb = extract_frame(input_path, "00:00:01")
  context.upload_media(thumb, "thumbnail.jpg", "image/jpeg")
  context.progress = 25

  # Generate 720p preview
  preview_720 = transcode(input_path, 1280, 720)
  context.upload_media(preview_720, "720p.mp4", "video/mp4")
  context.progress = 75

  # Generate 1080p
  preview_1080 = transcode(input_path, 1920, 1080)
  context.upload_media(preview_1080, "1080p.mp4", "video/mp4")
  context.progress = 100
end

Audio Processing

def process(context)
  input_path = context.download_original

  # Generate waveform visualization
  waveform = generate_waveform(input_path)
  context.upload_media(waveform, "waveform.png", "image/png")
  context.progress = 50

  # Convert to web-friendly format
  web_audio = convert_to_web(input_path)
  context.upload_media(web_audio, "audio.mp3", "audio/mpeg")
  context.progress = 100
end

Best Practices

1. Clean Up Temporary Files

def process(context)
  input_path = context.download_original
  temp_file = nil

  begin
    temp_file = create_temp_file
    transform(input_path, temp_file)
    context.upload_media(File.open(temp_file), "output.jpg", "image/jpeg")
  ensure
    File.unlink(temp_file) if temp_file && File.exist?(temp_file)
  end

  context.progress = 100
end

2. Handle Errors Gracefully

def process(context)
  input_path = context.download_original

  begin
    output = transform_media(input_path)
    context.upload_media(output, "result.jpg", "image/jpeg")
    context.progress = 100
  rescue ProcessingError => e
    Verse.logger.warn "Retrying: #{e.message}"
    context.reschedule!(after: 30)
  rescue StandardError => e
    Verse.logger.error "Failed: #{e.message}"
    context.error!(e.message)
    raise
  end
end

3. Provide Progress Updates

def process(context)
  steps = [
    { name: "Download", weight: 10 },
    { name: "Process", weight: 70 },
    { name: "Upload", weight: 20 }
  ]

  progress = 0

  input_path = context.download_original
  progress += steps[0][:weight]
  context.progress = progress

  output = transform_media(input_path)
  progress += steps[1][:weight]
  context.progress = progress

  context.upload_media(output, "result.jpg", "image/jpeg")
  progress += steps[2][:weight]
  context.progress = progress
end

Testing Your Media Backend

Run Tests

bundle exec rspec backends/media/
bundle exec rspec backends/media/<plugin_name>/processor_spec.rb

Test in IDAH Platform

Build your plugin frontend: cd frontend && pnpm build
Restart IDAH platform to load the plugin
Create a dataset using your plugin
Upload media files and verify they're processed correctly
Check logs for any errors or warnings

Real-World Example

See the idah-video plugin for a complete implementation:

View idah-video media backend on GitHub

🎬 Ready to process media! Start by adding a media backend to your plugin and implement your custom processing logic.

Import & Process Media