Persistent Storage

Configure persistent storage for media files and dataset exports in IDAH.

Overview

IDAH uses persistent storage in two services to store files:

Media Service

Stores imported media files such as images, videos, audio files, and other media assets used in datasets.

Sync Service

Stores exported dataset files generated during data export and synchronization operations.

Storage System

IDAH uses the verse-shrine gem for file storage management. This provides a flexible interface to configure different storage backends.

verse-shrine

A file attachment toolkit for Ruby applications with support for multiple storage backends.

View on GitHub →

Supported Storage Backends

  • File System - Local file storage (development)
  • AWS S3 - Amazon S3 storage
  • Google Cloud Storage - GCS
  • S3-Compatible - MinIO, DigitalOcean Spaces, etc.

Configuration

Storage is configured separately for each service using YAML configuration files.

Configuration Files

Media Service

  • app/media/config/config.yml - Production configuration
  • app/media/config/config.development.yml - Development overrides

Sync Service

  • app/sync/config/config.yml - Production configuration
  • app/sync/config/config.development.yml - Development overrides

Development Configuration

In development, IDAH uses local file system storage by default:

app/media/config/config.development.yml 
plugins:
  - name: shrine
    config:
      storages:
        - name: default
          adapter: file_system
          config:
            path: tmp/storage/development

Files are stored in the tmp/storage/development directory within each service.

Production Configuration

In production, configure S3-compatible storage:

Media Service Storage

app/media/config/config.yml 
plugins:
  - name: shrine
    config:
      storages:
        - name: default
          adapter: s3
          config:
            bucket: <%= ENV.fetch("MEDIAS_FILES_BUCKET") %>
            access_key_id: <%= ENV.fetch("MEDIAS_FILES_ACCESS_KEY_ID") %>
            secret_access_key: <%= ENV.fetch("MEDIAS_FILES_SECRET_ACCESS_KEY") %>
            public: false
            region: <%= ENV.fetch("MEDIAS_FILES_REGION") %>
            prefix: files
            endpoint: <%= ENV.fetch("MEDIAS_FILES_ENDPOINT") %>

Sync Service Storage

app/sync/config/config.yml 
plugins:
  - name: shrine
    config:
      storages:
        - name: default
          adapter: s3
          config:
            bucket: <%= ENV.fetch("SYNC_FILES_BUCKET") %>
            access_key_id: <%= ENV.fetch("SYNC_FILES_ACCESS_KEY_ID") %>
            secret_access_key: <%= ENV.fetch("SYNC_FILES_SECRET_ACCESS_KEY") %>
            public: false
            region: <%= ENV.fetch("SYNC_FILES_REGION") %>
            prefix: files
            endpoint: <%= ENV.fetch("SYNC_FILES_ENDPOINT") %>

Environment Variables

Media Service Variables

app/media/.env 
# Media storage configuration MEDIAS_FILES_BUCKET=my-media-bucket MEDIAS_FILES_ACCESS_KEY_ID=your-access-key
        MEDIAS_FILES_SECRET_ACCESS_KEY=your-secret-key MEDIAS_FILES_REGION=us-east-1
        MEDIAS_FILES_ENDPOINT=https://s3.amazonaws.com

Sync Service Variables

app/sync/.env 
# Sync storage configuration SYNC_FILES_BUCKET=my-exports-bucket SYNC_FILES_ACCESS_KEY_ID=your-access-key
        SYNC_FILES_SECRET_ACCESS_KEY=your-secret-key SYNC_FILES_REGION=us-east-1
        SYNC_FILES_ENDPOINT=https://s3.amazonaws.com

Storage Provider Examples

AWS S3

app/media/.env 
# AWS S3 MEDIAS_FILES_BUCKET=my-idah-media MEDIAS_FILES_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
        MEDIAS_FILES_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY MEDIAS_FILES_REGION=us-east-1
        MEDIAS_FILES_ENDPOINT=https://s3.amazonaws.com

Google Cloud Storage

app/media/.env 
# Google Cloud Storage (S3-compatible) MEDIAS_FILES_BUCKET=my-idah-media
        MEDIAS_FILES_ACCESS_KEY_ID=your-gcs-access-key MEDIAS_FILES_SECRET_ACCESS_KEY=your-gcs-secret-key
        MEDIAS_FILES_REGION=us-east1 MEDIAS_FILES_ENDPOINT=https://storage.googleapis.com

MinIO (Self-Hosted)

app/media/.env 
# MinIO MEDIAS_FILES_BUCKET=idah-media MEDIAS_FILES_ACCESS_KEY_ID=minioadmin
        MEDIAS_FILES_SECRET_ACCESS_KEY=minioadmin MEDIAS_FILES_REGION=us-east-1 MEDIAS_FILES_ENDPOINT=http://minio:9000

DigitalOcean Spaces

app/media/.env 
# DigitalOcean Spaces MEDIAS_FILES_BUCKET=my-space MEDIAS_FILES_ACCESS_KEY_ID=your-spaces-key
        MEDIAS_FILES_SECRET_ACCESS_KEY=your-spaces-secret MEDIAS_FILES_REGION=nyc3
        MEDIAS_FILES_ENDPOINT=https://nyc3.digitaloceanspaces.com

Configuration Tips

Use Separate Buckets

It's recommended to use separate buckets for media and exports:

  • Easier to manage lifecycle policies
  • Better access control and permissions
  • Simplified backup and retention strategies

File Prefix

The prefix: files configuration adds a directory prefix to all stored files:

Example S3 Bucket Structure 
my-bucket
   └─ files
      ├─ media-file-1.jpg
      ├─ media-file-2.mp4
      └─ export-dataset-1.json

Security Considerations

  • Keep public: false to prevent public access
  • Use IAM roles instead of access keys when possible (AWS)
  • Rotate access credentials regularly
  • Enable encryption at rest on your storage bucket

Next Steps

🗄️ Storage configured! Your media and exports are now persisted to your chosen storage backend.