Add continuous SQLite replication with Litestream to rvn-ec2-service

[devin-ai-integration]

Summary

Phase 3, stacked on #116. Snapshots and dumps both leave a window: whatever happened since the last interval is lost on unplanned failure, and the termination hook from #116 only covers planned replacement. Litestream closes that window for SQLite by shipping WAL frames continuously, taking RPO from hours to seconds — the self-hosted Executor case that started this thread.

backup_replication_enabled installs a checksum-pinned Litestream, writes a config for one database, and runs litestream replicate as a second supervised program alongside the app, reusing the phase 2 bucket under its own replication/<service> prefix. Restore-on-first-boot reuses the dump path's four-way semantics, deliberately identical so the two features can't behave differently during an incident:

discovery call fails     → block startup, no marker
no replica               → continue, mark   (fresh service)
replica older than max   → block startup
restore fails            → block, no marker

Postgres stays a documented recipe, not an input. WAL-G/pgBackRest need archive_command inside postgresql.conf plus engine credentials this module doesn't own; an input would half-work, so the README carries the recipe over additional_user_data and the phase 1 custom consistency hooks instead.

Two documented caveats that are load-bearing rather than boilerplate: two instances replicating one SQLite database to the same prefix will corrupt the replica (per the earlier decision, documented rather than gated on capacity), and replication supplements snapshots and dumps rather than replacing them, because a replica of a corrupted database is a corrupted replica. Hence all three phases stay independently switchable.

Litestream 0.5 specifics

0.5 reworked both the CLI and the config, and the 0.3-era surface that most examples still show is silently wrong here — the config parser accepts unknown replica keys without complaint, so a clean parse proves nothing. Verified against the pinned release binary and its cmd/litestream/main.go:

• restore no longer has -if-replica-exists, -if-db-not-exists or -json. Discovery is litestream ltx -level all -json <replica-url>, which distinguishes an empty replica ([], exit 0) from a failed lookup (exit 1) — exactly the distinction the blocking logic depends on.
• Retention lives in top-level snapshot.interval / snapshot.retention and retention.enabled. Nested per-replica snapshot-interval / retention keys are accepted and ignored, so exposing them would have handed users a retention policy that silently did nothing.
• dbs[].replica (singular) is current; replicas is deprecated.

The version pin and its checksums live in the module rather than in a variable: a version input whose validation accepts a single value can't be changed anyway, and an unpinned checksum is the real risk. Upgrading Litestream is a module release.

Also folded in, from review feedback on #115: the DLM role gets ec2:CopySnapshot, ec2:ModifySnapshotAttribute and the KMS permissions needed to copy an encrypted snapshot, gated on the cross-region copy case. Without them the copy failed while source snapshots kept succeeding, so the schedule looked healthy.

tofu test covers replication off, replication creating the bucket with dumps disabled, replication reusing a supplied bucket ARN, and the database-path precondition. make publish-local-dev still can't run here (no local API on localhost:8080).

Link to Devin session: https://app.devin.ai/sessions/2b20f3ce4e7744a69642a468e5b7d463
Requested by: @flybayer

Greptile Summary

Adds optional checksum-pinned Litestream replication and first-boot SQLite restoration to the EC2 service module.

• Adds shared S3 bucket selection, IAM access, replication outputs, and module-definition inputs.
• Installs and supervises Litestream during instance bootstrap, with replica discovery, age validation, and restore handling.
• Extends CloudWatch log collection, OpenTofu tests, documentation, and the module release metadata.

Confidence Score: 4/5

The bucket-selection defect should be fixed before merging because a valid EFS-dump configuration can silently replicate SQLite data to the wrong S3 bucket.

The new effective-bucket local ignores the logical-dump destination, allowing an irrelevant dump ARN to override the explicitly configured replication ARN across Litestream configuration, IAM, and outputs.

Files Needing Attention: compute/ec2_service/locals.tf and compute/ec2_service/launch_template.tf

Important Files Changed

Filename	Overview
compute/ec2_service/locals.tf	Adds shared dump/replication bucket selection and Litestream template rendering, but can select the dump ARN incorrectly when dumps use EFS.
compute/ec2_service/templates/backup_replication.sh.tpl	Installs, configures, restores, and supervises Litestream with explicit discovery and restore failure handling.
compute/ec2_service/iam.tf	Extends instance-role S3 list and object permissions to the replication prefix.
compute/ec2_service/launch_template.tf	Adds replication storage and database-path preconditions, though its bucket-equality guard does not cover the EFS-dump combination implicated by the finding.
compute/ec2_service/variables.tf	Introduces typed replication settings with ARN, duration, engine, and maximum-age validation.
compute/ec2_service/rvn-ec2-service-definition.yml	Publishes the replication controls, Terraform mappings, documentation, and v1.7.0 release metadata.
compute/ec2_service/tests/backups.tftest.hcl	Covers disabled replication, created and supplied buckets, and database-path validation, but not the failing EFS-dump and replication bucket combination.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  UI["Replication inputs"] --> TF["Terraform locals and IAM"]
  TF --> Boot["EC2 user data"]
  Boot --> Install["Install checksum-pinned Litestream"]
  Install --> Discover{"Restore enabled and marker absent?"}
  Discover -->|No| Replicate["Supervisor: litestream replicate"]
  Discover -->|Yes| List["List replica LTX files in S3"]
  List -->|Empty| Marker["Write restore marker"]
  List -->|Found and fresh| Restore["Restore SQLite database"]
  List -->|Failure or stale| Block["Block application startup"]
  Restore --> Marker
  Marker --> Replicate
  Replicate --> S3[("S3 replication prefix")]

Loading

Prompt To Fix All With AI

### Issue 1
compute/ec2_service/locals.tf:28
**Replication bucket selection is overridden**

When logical dumps use EFS and both bucket ARN inputs are set, this expression selects the otherwise-unused dump bucket ARN instead of `backup_replication_s3_bucket_arn`, causing Litestream, its IAM resources, and the replication outputs to target the wrong S3 bucket.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (1): Last reviewed commit: "Merge branch 'devin/1786976726-ec2-servi..." | Re-trigger Greptile}

Greptile also left 1 inline comment on this PR.

Context used:

• Knowledge Base — EC2 Service deployment and durability
• Knowledge Base — tools/ravion-modules

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

Ravion Module Publish Plan

Dry run only. No Ravion API mutations were made.

Module	Current Version	New Version	Description
rvn-ec2-service	1.4.1	1.7.0	Add checksum-verified Litestream replication, safe logical restore, and planned-termination backups for on-disk databases.

Diffs

rvn-ec2-service n/a -> 1.7.0

--- remote
+++ compiled
-description: Runs supervised workloads on a stable EC2 Auto Scaling Group, with optional shared ALB routing and switchable container or manual in-place deploys.
+description: Runs supervised workloads on a stable EC2 Auto Scaling Group, with optional shared ALB routing, local EBS data, and switchable container or manual in-place deploys.
 name: EC2 Service
 type: rvn-ec2-service

rvn-ec2-service 1.4.1 -> 1.7.0

--- remote
+++ compiled
     show_when:
       data_volume_creation_enabled: true
     type: string
+  - collapsible: true
+    description: Optional EBS snapshot ID to restore into the data volume when a replacement instance boots. Clear this after the restore is complete.
+    id: data_volume_snapshot_id
+    label: Data volume snapshot ID
+    patterns:
+      - message: Enter an EBS snapshot ID.
+        pattern: ^snap-[a-z0-9]+$
+    placeholder: snap-...
+    required: false
+    show_when:
+      data_volume_creation_enabled: true
+    type: string
+  - description: EBS snapshots and engine-native logical dumps for on-disk application data.
+    id: section_backups
+    label: Backups
+    type: section
   - default: false
+    description: Schedule Amazon Data Lifecycle Manager snapshots for this service's instances. Opt in for on-disk databases such as SQLite or Postgres.
+    id: backup_enabled
+    label: Enable EBS backups
+    type: boolean
+  - default: 24
+    description: Maximum age of a scheduled snapshot, and therefore the honest backup RPO.
+    id: backup_interval_hours
+    label: Backup interval (hours)
+    max: 24
+    min: 1
+    show_when:
+      backup_enabled: true
+    type: number
+    values:
+      - label: Every hour
+        value: 1
+      - label: Every 2 hours
+        value: 2
+      - label: Every 3 hours
+        value: 3
+      - label: Every 4 hours
+        value: 4
+      - label: Every 6 hours
+        value: 6
+      - label: Every 8 hours
+        value: 8
+      - label: Every 12 hours
+        value: 12
+      - label: Daily
+        value: 24
+  - default: 05:00
+    description: UTC time in HH:MM format when the daily schedule begins.
+    id: backup_start_time
+    label: Backup start time (UTC)
+    patterns:
+      - message: Use HH:MM in UTC.
+        pattern: ^(0[0-9]|1[0-9]|2[0-3]):[0-5][0-9]$
+    show_when:
+      backup_enabled: true
+    type: string
+  - default: 7
+    description: Number of snapshots retained for this schedule.
+    id: backup_retention_count
+    label: Snapshots to retain
+    min: 1
+    show_when:
+      backup_enabled: true
+    type: number
+  - default: false
+    description: Include the operating system volume in each snapshot set. It is included automatically when no data volume exists.
+    id: backup_root_volume_included
+    label: Include root volume
+    show_when:
+      backup_enabled: true
+    type: boolean
+  - default: filesystem_freeze
+    description: Freeze the data filesystem before snapshots, take crash-consistent snapshots, or provide database-specific pre/post commands.
+    id: backup_consistency_mode
+    label: Backup consistency mode
+    show_when:
+      backup_enabled: true
+    type: string
+    values:
+      - description: Sync and freeze the data volume without touching / or /boot.
+        label: Filesystem freeze
+        value: filesystem_freeze
+      - description: Snapshot without scripts; database engines must recover as after a power loss.
+        label: Crash consistent
+        value: crash_consistent
+      - description: Run your own database quiesce commands before and after the snapshot.
+        label: Custom
+        value: custom
+  - description: Command that quiesces the database before each snapshot. Required for Custom consistency mode.
+    id: backup_pre_script_command
+    label: Backup pre-script command
+    required: true
+    show_when:
+      backup_consistency_mode: custom
+      backup_enabled: true
+    type: text
+  - description: Command that resumes the database after each snapshot. Required for Custom consistency mode.
+    id: backup_post_script_command
+    label: Backup post-script command
+    required: true
+    show_when:
+      backup_consistency_mode: custom
+      backup_enabled: true
+    type: text
+  - collapsible: true
+    description: Optional AWS region for a second snapshot copy. Cross-region copies incur additional storage and transfer costs.
+    id: backup_cross_region_copy_destination
+    label: Cross-region copy destination
+    required: false
+    show_when:
+      backup_enabled: true
+    type: string
+    values: $values:aws/regions
+  - description: Engine-native logical backups shipped to S3 or EFS, with optional restore and termination-time automation.
+    id: section_logical_backups
+    label: Logical dumps
+    type: section
+  - default: false
+    description: Run engine-native dumps on a systemd schedule and ship them off the instance.
+    id: backup_dump_enabled
+    label: Enable logical dumps
+    type: boolean
+  - description: Root command that writes logical backup artifacts into the RAVION_BACKUP_DIR directory.
+    id: backup_dump_command
+    label: Dump command
+    required: true
+    show_when:
+      backup_dump_enabled: true
+    type: text
+  - description: Root command that reads logical backup artifacts from RAVION_BACKUP_DIR.
+    id: backup_dump_restore_command
+    label: Restore command
+    required: true
+    show_when:
+      backup_dump_restore_on_first_boot_enabled: true
+    type: text
+  - default: "*-*-* 04:00:00 UTC"
+    description: systemd OnCalendar expression for logical dumps, not cron syntax.
+    id: backup_dump_schedule
+    label: Dump schedule
+    show_when:
+      backup_dump_enabled: true
+    type: string
+  - default: s3
+    description: Store logical dumps in a module-created or supplied S3 bucket, or on the existing EFS mount.
+    id: backup_dump_destination
+    label: Dump destination
+    show_when:
+      backup_dump_enabled: true
+    type: string
+    values:
+      - label: S3
+        value: s3
+      - label: EFS
+        value: efs
+  - description: Optional existing S3 bucket ARN. Leave blank to create a dedicated encrypted bucket.
+    id: backup_dump_s3_bucket_arn
+    label: S3 bucket ARN
+    placeholder: arn:aws:s3:::bucket-name
+    required: false
+    show_when:
+      backup_dump_destination: s3
+      backup_dump_enabled: true
+    type: string
+  - default: backups/
+    description: Prefix under which this service's timestamped manifests and dump artifacts are stored.
+    id: backup_dump_s3_prefix
+    label: S3 prefix
+    show_when:
+      backup_dump_destination: s3
+      backup_dump_enabled: true
+    type: string
+  - default: 30
+    description: Number of days to retain logical dump artifacts.
+    id: backup_dump_retention_days
+    label: Logical dump retention (days)
+    min: 1
+    show_when:
+      backup_dump_enabled: true
+    type: number
+  - default: 48
+    description: Maximum expected interval between successful dumps; set this higher than the dump schedule interval so the freshness alarm is meaningful.
+    id: backup_dump_max_interval_hours
+    label: Maximum dump interval (hours)
+    min: 1
+    show_when:
+      backup_dump_enabled: true
+    type: number
+  - default: false
+    description: Allow Terraform to delete a module-created S3 backup bucket that still contains backups.
+    id: backup_dump_force_deletion_enabled
+    label: Allow backup bucket deletion
+    show_when:
+      backup_dump_destination: s3
+      backup_dump_enabled: true
+    type: boolean
+  - default: false
+    description: Discover and restore the newest manifest-based logical dump before starting the application on a replacement instance.
+    id: backup_dump_restore_on_first_boot_enabled
+    label: Restore latest dump on first boot
+    show_when:
+      backup_dump_enabled: true
+    type: boolean
+  - description: Refuse to restore and block application startup when the newest dump is older than this limit.
+    id: backup_max_age_hours
+    label: Maximum restore age (hours)
+    min: 1
+    required: false
+    show_when:
+      backup_dump_enabled: true
+      backup_dump_restore_on_first_boot_enabled: true
+    type: number
+  - default: true
+    description: Run a final logical dump through an Auto Scaling lifecycle hook before planned instance termination.
+    id: backup_on_termination_enabled
+    label: Backup on planned termination
+    show_when:
+      backup_dump_enabled: true
+    type: boolean
+  - default: 1800
+    description: Maximum time allowed for the planned-termination dump; the automation step keeps 60 seconds of slack before the lifecycle hook expires.
+    id: backup_on_termination_timeout_seconds
+    label: Planned termination timeout (secs)
+    max: 7200
+    min: 300
+    show_when:
+      backup_dump_enabled: true
+    type: number
+  - default: true
+    description: Alarm when the shared CloudWatch log group has no recent successful logical dump record.
+    id: backup_dump_failure_alarm_enabled
+    label: Logical dump freshness alarm
+    show_when:
+      backup_dump_enabled: true
+    type: boolean
+  - description: Continuous SQLite replication to S3 with Litestream.
+    id: section_replication
+    label: Continuous replication
+    type: section
+  - default: false
+    description: Continuously replicate one SQLite database to S3 with a pinned Litestream release.
+    id: backup_replication_enabled
+    label: Enable continuous replication
+    type: boolean
+  - default: litestream
+    description: The replication engine. Litestream is currently the only supported engine.
+    id: backup_replication_engine
+    label: Replication engine
+    show_when:
+      backup_replication_enabled: true
+    type: string
+    values:
+      - label: Litestream (SQLite)
+        value: litestream
+  - description: Absolute path to the SQLite database on the data volume, such as /data/app.db.
+    id: backup_replication_database_path
+    label: SQLite database path
+    required: true
+    show_when:
+      backup_replication_enabled: true
+    type: string
+  - description: Optional existing S3 bucket ARN for the replica. Leave blank to create a dedicated encrypted bucket.
+    id: backup_replication_s3_bucket_arn
+    label: Replication S3 bucket ARN
+    placeholder: arn:aws:s3:::bucket-name
+    required: false
+    show_when:
+      backup_replication_enabled: true
+    type: string
+  - default: false
+    description: Restore the newest Litestream replica before starting the application on a replacement instance.
+    id: backup_replication_restore_on_first_boot_enabled
+    label: Restore replica on first boot
+    show_when:
+      backup_replication_enabled: true
+    type: boolean
+  - default: 1m
+    description: Litestream full snapshot interval, such as 1m or 1h. Full snapshots are expensive, so choose a value shorter than the retention duration.
+    id: backup_replication_snapshot_interval
+    label: Replica snapshot interval
+    show_when:
+      backup_replication_enabled: true
+    type: string
+  - default: 24h
+    description: Litestream snapshot retention duration, such as 24h or 168h. It must exceed the snapshot interval.
+    id: backup_replication_retention
+    label: Replica retention
+    show_when:
+      backup_replication_enabled: true
+    type: string
+  - description: Refuse to restore and block application startup when the newest replica is older than this limit.
+    id: backup_replication_max_age_hours
+    label: Maximum replica age (hours)
+    min: 1
+    required: false
+    show_when:
+      backup_replication_enabled: true
+      backup_replication_restore_on_first_boot_enabled: true
+    type: number
+  - default: false
     description: Mount a shared network filesystem that every instance can access. Use it when multiple instances need the same files, or when files must survive instance replacement without a restore step.
     id: efs_enabled
     label: EFS file system
@@
 
   Instances are as stable as an EC2 instance you launch yourself in the AWS console. Deploys, app restarts, and stack updates do not replace them, so each instance keeps its root and optional data volume, and everything on those disks, for its whole life. Even changing the AMI leaves running instances alone: the change becomes a new launc
... diff truncated ...

[greptile-apps]

Replication bucket selection is overridden

When logical dumps use EFS and both bucket ARN inputs are set, this expression selects the otherwise-unused dump bucket ARN instead of backup_replication_s3_bucket_arn, causing Litestream, its IAM resources, and the replication outputs to target the wrong S3 bucket.

Prompt To Fix With AI

This is a comment left during a code review.
Path: compute/ec2_service/locals.tf
Line: 28

Comment:
**Replication bucket selection is overridden**

When logical dumps use EFS and both bucket ARN inputs are set, this expression selects the otherwise-unused dump bucket ARN instead of `backup_replication_s3_bucket_arn`, causing Litestream, its IAM resources, and the replication outputs to target the wrong S3 bucket.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[devin-ai-integration]

Valid — fixing. The same-bucket precondition in launch_template.tf only fires when backup_dump_destination == "s3", so the EFS-dumps case the bot describes slips past it: backup_dump_s3_bucket_arn is inert for dumps but still wins the ternary, and replication silently points at it. backup_s3_bucket_arn will gate the dump ARN on the dump destination actually being s3, so a supplied replication bucket is used whenever dumps aren't the S3 consumer.