Skip to content

video-generation.md

Latest commit

 

History

History
1312 lines (1099 loc) · 46.2 KB

video-generation.md

File metadata and controls

1312 lines (1099 loc) · 46.2 KB
Raw
title Video Generation with Veo 3.1
description Generate professional video content with audio using Google's Veo 3.1 model through image and prompt inputs
keywords video generation, veo 3.1, google ai, multimodal, video synthesis, ai video, vertex ai

Video Generation with Veo 3.1

NeuroLink integrates multiple video-generation providers — Google's Veo 3.1 (default), Kling, Runway, and Replicate-hosted models — behind a single nl.generate({ output: { mode: "video" } }) call. Transform static images into dynamic, professional-quality video content with synchronized audio (where the provider supports it).

Overview

Video generation in NeuroLink dispatches through the central VideoProcessor registry. The system uses the existing generate() function with video-specific options:

  1. Accepts an input image via input.images and text prompt via input.text
  2. Validates image format, size, and aspect ratio requirements
  3. Selects the handler matching output.video.provider (default "vertex") — see Routing Across Providers below
  4. Generates a clip (length / audio / resolution support varies per provider)
  5. Returns a VideoGenerationResult containing video buffer and metadata
graph LR
    A[Input Image] --> B[NeuroLink SDK]
    C[Text Prompt] --> B
    B --> D{VideoProcessor}
    D -->|"provider:vertex (default)"| E[Vertex Veo 3.1]
    D -->|"provider:kling"| F[PiAPI Kling]
    D -->|"provider:runway"| G[Runway gen3]
    D -->|"provider:replicate"| H[Replicate model]
    E --> I[VideoGenerationResult]
    F --> I
    G --> I
    H --> I
    I --> J[Save / Stream / Process]
Loading

What You Get

  • Video with audio – Generate 8-second video clips with synchronized audio from a single image and text prompt
  • SDK integration – Use existing neurolink.generate() with output.mode: "video" to create videos
  • CLI support – Generate videos directly from the command line with --outputMode video
  • Buffer-based output – Receive video as Buffer objects via VideoGenerationResult for flexible post-processing
  • Multiple resolutions – Support for 720p and 1080p output
  • Aspect ratio control – Choose between 9:16 (portrait) and 16:9 (landscape) formats
  • Director Mode – Chain multiple segments into one continuous video with AI-generated transitions (see Video Director Mode)

Supported Providers & Models

nl.generate({ output: { mode: "video", video: { provider } } }) dispatches through the central VideoProcessor registry, which knows four shipped handlers. The default when provider is omitted is vertex.

Provider Compatibility

Provider Default Model Length Audio Input Auth Notes
vertex veo-3.1 4 / 6 / 8 s Yes image + text prompt GOOGLE_APPLICATION_CREDENTIALS Default — best fit for GCP / Vertex tenants
kling PiAPI Kling v1.6 5 / 10 s No publicly accessible image URL + text KLING_API_KEY (PiAPI token) Requires imageUrl (handler rejects inline base64). See provider guide.
runway Runway gen3 5 / 10 s No image + text RUNWAY_API_KEY Length must be 5 or 10 (rejected at upstream if 4). See provider guide.
replicate meta/wan-2-2/i2v (override via model) Model-specific Model-specific image + text REPLICATE_API_TOKEN Any image-to-video model on Replicate via video.model. See provider guide.

Model Versions & Capabilities

Model Version Release Date Key Features Notes
veo-3.1 2024 Audio generation, 8s duration Default for vertex
kling-v1.6 2024 Sharp motion, 720p / 1080p Routed via PiAPI
gen3-turbo 2024 5 s / 10 s clips, 720p+ Runway's faster generation tier

Note: Use the per-provider setup pages (Vertex Veo, Kling, Runway, Replicate) for credential and quota details.

Known Limitations

  • vertex: max video duration 8 seconds per clip (4 / 6 / 8 s); image required; audio auto-generated; concurrent request limit 5 per project; processing 30–120 s
  • kling: image must be a publicly accessible URL — pass video.imageUrl. Inline Buffer images are rejected at the handler
  • runway: length validated upstream as 5 or 10 seconds — using 4 will surface a Runway 400. The local 4 / 6 / 8 schema gate matches Vertex's contract; future versions may widen the type
  • replicate: per-model quirks (some models have token caps, some return WebP/GIF). Override video.model to pick a specific model checkpoint
  • For multi-segment videos with transitions, see Video Director Mode (currently Vertex-only)

Routing Across Providers

Pick a provider per-call by setting output.video.provider:

// Vertex (default — same as omitting `provider`)
await nl.generate({
  input: { text: "...", images: [imageBuffer] },
  output: {
    mode: "video",
    video: { provider: "vertex", length: 6, resolution: "720p" },
  },
});

// Kling (PiAPI) — requires a publicly accessible image URL, not a Buffer
await nl.generate({
  input: { text: "...", images: [imageBuffer] },
  output: {
    mode: "video",
    video: {
      provider: "kling",
      imageUrl: "https://your-cdn.example.com/source.jpg",
      length: 5,
      resolution: "720p",
    },
  },
});

// Runway gen3 — length must be 5 or 10 upstream
await nl.generate({
  input: { text: "...", images: [imageBuffer] },
  output: {
    mode: "video",
    video: { provider: "runway", length: 5, resolution: "720p" },
  },
});

// Replicate (any image-to-video model)
await nl.generate({
  input: { text: "...", images: [imageBuffer] },
  output: {
    mode: "video",
    video: { provider: "replicate", model: "meta/wan-2-2/i2v", length: 4 },
  },
});

result.provider echoes the chosen handler — useful for logging and multi-provider A/B harnesses.

Unknown Provider Behavior

Passing an unregistered provider name throws a typed VideoError(VIDEO_ERROR_CODES.PROVIDER_NOT_SUPPORTED) with the list of known names. NeuroLink does not silently fall back to Vertex when the requested provider is unknown — a misspelled provider is always surfaced as an error.

try {
  await nl.generate({
    input: { text: "...", images: [imageBuffer] },
    output: { mode: "video", video: { provider: "klng" /* typo */ } },
  });
} catch (err) {
  // err.code === "VIDEO_PROVIDER_NOT_SUPPORTED"
  // err.context.available === ["vertex", "kling", "runway", "replicate"]
}

Prerequisites

  1. Vertex AI credentials with Veo access enabled
  2. Google Cloud project with billing enabled
  3. Service account with aiplatform.user role
  4. Sufficient storage for video buffers (each 8-second video is approximately 2-5 MB)

Quick Start

SDK Usage

import { NeuroLink } from "@juspay/neurolink";
import { readFileSync, writeFileSync } from "fs";

const neurolink = new NeuroLink();

// Basic video generation using generate() with video output mode
const result = await neurolink.generate({
  input: {
    text: "Camera slowly zooms in on the product with soft lighting",
    images: [readFileSync("./product-image.jpg")],
  },
  provider: "vertex",
  model: "veo-3.1",
  output: {
    mode: "video",
    video: {
      resolution: "720p",
      length: 8,
      aspectRatio: "16:9",
      audio: true,
    },
  },
});

// Access video data from VideoGenerationResult
if (result.video) {
  writeFileSync("output.mp4", result.video.data);
  console.log(`Video generated: ${result.video.metadata?.duration}s`);
}

With Full Options

import { NeuroLink } from "@juspay/neurolink";
import { readFile, writeFile } from "fs/promises";

const neurolink = new NeuroLink();

const result = await neurolink.generate({
  input: {
    text: "Dynamic camera movement showcasing the product from multiple angles",
    images: [await readFile("./input.jpg")],
  },
  provider: "vertex",
  model: "veo-3.1",
  output: {
    mode: "video",
    video: {
      resolution: "1080p",
      length: 8,
      aspectRatio: "16:9",
      audio: true,
    },
  },
});

if (result.video) {
  await writeFile("output.mp4", result.video.data);

  console.log("Video metadata:", {
    duration: result.video.metadata?.duration,
    dimensions: result.video.metadata?.dimensions,
    format: result.video.mediaType,
  });
}

Image URL Input

import { NeuroLink } from "@juspay/neurolink";
import { writeFile } from "fs/promises";

const neurolink = new NeuroLink();

// Use image URL instead of Buffer
const result = await neurolink.generate({
  input: {
    text: "Elegant rotation revealing product details",
    images: ["https://example.com/product.png"],
  },
  provider: "vertex",
  model: "veo-3.1",
  output: {
    mode: "video",
    video: {
      resolution: "720p",
      length: 8,
    },
  },
});

if (result.video) {
  await writeFile("output.mp4", result.video.data);
}

CLI Usage

# Basic video generation
npx @juspay/neurolink generate "Create a product showcase video" \
  --image ./input.jpg \
  --videoOutput ./output.mp4

# Full options
npx @juspay/neurolink generate "Dynamic camera movement" \
  --image ./input.jpg \
  --provider vertex \
  --model veo-3.1 \
  --videoResolution 1080p \
  --videoLength 8 \
  --videoAspectRatio 16:9 \
  --videoAudio true \
  --videoOutput ./output.mp4

# JSON output mode (for scripting)
npx @juspay/neurolink generate "prompt" \
  --image input.jpg \
  --videoOutput output.mp4 \
  --format json

# With analytics
npx @juspay/neurolink generate "Camera pans across futuristic city" \
  --image ./input-city.jpg \
  --videoResolution 1080p \
  --videoOutput ./city-video.mp4 \
  --enable-analytics

CLI Arguments

Argument Type Default Description
--image string Required Path to the input image file
--videoOutput string ./output.mp4 Path to save the generated video
--provider string vertex AI provider to use
--model string veo-3.1 Model version
--videoResolution string 720p Output resolution (720p or 1080p)
--videoLength number 4 Video duration in seconds (4, 6, or 8)
--videoAspectRatio string 16:9 Aspect ratio (9:16 or 16:9)
--videoAudio boolean true Enable audio generation

Comprehensive Examples

Example 1: Basic Video Generation

import { NeuroLink } from "@juspay/neurolink";
import { readFile, writeFile } from "fs/promises";

const neurolink = new NeuroLink();

async function generateSingleVideo() {
  const result = await neurolink.generate({
    input: {
      text: "Smooth camera pan revealing the product with ambient lighting",
      images: [await readFile("./product-hero.jpg")],
    },
    provider: "vertex",
    model: "veo-3.1",
    output: {
      mode: "video",
      video: { resolution: "720p", length: 8 },
    },
  });

  if (result.video) {
    await writeFile("product-video.mp4", result.video.data);

    console.log({
      duration: result.video.metadata?.duration,
      dimensions: result.video.metadata?.dimensions,
      mediaType: result.video.mediaType,
      size: result.video.data.length,
    });
  }
}

Example 2: Batch Video Generation

import { NeuroLink } from "@juspay/neurolink";
import { readdir, readFile, writeFile } from "fs/promises";
import path from "path";

const neurolink = new NeuroLink();

async function batchGenerateVideos(
  inputDir: string,
  outputDir: string,
  prompt: string,
) {
  const files = await readdir(inputDir);
  const imageFiles = files.filter((f) =>
    [".jpg", ".jpeg", ".png", ".webp"].includes(path.extname(f).toLowerCase()),
  );

  const results = [];

  for (const imageFile of imageFiles) {
    console.log(`Processing: ${imageFile}`);

    try {
      const imageBuffer = await readFile(path.join(inputDir, imageFile));

      const result = await neurolink.generate({
        input: {
          text: prompt,
          images: [imageBuffer],
        },
        provider: "vertex",
        model: "veo-3.1",
        output: {
          mode: "video",
          video: { resolution: "720p", length: 8 },
        },
      });

      if (result.video) {
        const outputPath = path.join(
          outputDir,
          `${path.basename(imageFile, path.extname(imageFile))}.mp4`,
        );
        await writeFile(outputPath, result.video.data);

        results.push({
          input: imageFile,
          output: outputPath,
          duration: result.video.metadata?.duration,
          success: true,
        });
      }
    } catch (error) {
      results.push({
        input: imageFile,
        error: error instanceof Error ? error.message : "Unknown error",
        success: false,
      });
    }
  }

  return results;
}

// Usage
const results = await batchGenerateVideos(
  "./product-images",
  "./product-videos",
  "Dynamic product showcase with smooth camera movement",
);
console.table(results);

Example 3: Different Aspect Ratios

import { NeuroLink } from "@juspay/neurolink";
import { readFile, writeFile } from "fs/promises";

const neurolink = new NeuroLink();

// Portrait video for social media stories/reels
const portrait = await neurolink.generate({
  input: {
    text: "Vertical video with upward camera movement",
    images: [await readFile("./portrait-image.jpg")],
  },
  provider: "vertex",
  model: "veo-3.1",
  output: {
    mode: "video",
    video: {
      resolution: "1080p",
      aspectRatio: "9:16",
      length: 8,
    },
  },
});

// Landscape video for YouTube/websites
const landscape = await neurolink.generate({
  input: {
    text: "Cinematic horizontal pan across the scene",
    images: [await readFile("./landscape-image.jpg")],
  },
  provider: "vertex",
  model: "veo-3.1",
  output: {
    mode: "video",
    video: {
      resolution: "1080p",
      aspectRatio: "16:9",
      length: 8,
    },
  },
});

Example 4: Integration with Image Analysis

import { NeuroLink } from "@juspay/neurolink";
import { readFile, writeFile } from "fs/promises";

const neurolink = new NeuroLink();

// Step 1: Analyze product image and generate video concept
const analysis = await neurolink.generate({
  input: {
    text: `Analyze this product image and suggest a compelling video concept.
           Focus on key visual features and motion opportunities.`,
    images: [await readFile("product-image.jpg")],
  },
  provider: "vertex",
  model: "gemini-3-flash-preview",
});

console.log("AI Video Concept:", analysis.content);

// Step 2: Generate video using AI-suggested prompt
const result = await neurolink.generate({
  input: {
    text: analysis.content, // Use AI-generated prompt
    images: [await readFile("product-image.jpg")],
  },
  provider: "vertex",
  model: "veo-3.1",
  output: {
    mode: "video",
    video: {
      resolution: "1080p",
      aspectRatio: "16:9",
      length: 8,
    },
  },
});

if (result.video) {
  await writeFile("ai-directed-video.mp4", result.video.data);
  console.log("AI-driven video generation complete!");
}

Example 5: Error Handling

import { NeuroLink } from "@juspay/neurolink";
import { NeuroLinkError } from "@juspay/neurolink";
import { readFile, writeFile } from "fs/promises";

const neurolink = new NeuroLink();

async function generateVideoWithErrorHandling(
  imagePath: string,
  prompt: string,
) {
  const maxRetries = 3;
  let lastError: Error | null = null;

  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      const result = await neurolink.generate({
        input: {
          text: prompt,
          images: [await readFile(imagePath)],
        },
        provider: "vertex",
        model: "veo-3.1",
        output: {
          mode: "video",
          video: { resolution: "720p", length: 8 },
        },
        timeout: 120000, // Video generation can take longer
      });

      if (result.video) {
        return result.video;
      }
      throw new Error("No video generated");
    } catch (error) {
      lastError = error;

      if (error instanceof NeuroLinkError) {
        // Check error category and handle accordingly
        if (
          error.category === "configuration" ||
          error.category === "permission"
        ) {
          console.error(
            "Configuration/permission error. Check your Vertex AI credentials.",
          );
          throw error; // Don't retry config/permission errors
        }

        if (error.code.includes("RATE_LIMIT")) {
          const waitTime = Math.pow(2, attempt) * 1000; // Exponential backoff
          console.log(
            `Rate limited. Waiting ${waitTime / 1000}s before retry...`,
          );
          await new Promise((resolve) => setTimeout(resolve, waitTime));
          continue;
        }

        if (error.category === "network" && error.retriable) {
          console.log(`Network error on attempt ${attempt}. Retrying...`);
          await new Promise((resolve) => setTimeout(resolve, 2000));
          continue;
        }

        if (error.category === "execution") {
          console.error(`Execution error: ${error.message}`);
          throw error;
        }
      }

      throw error;
    }
  }

  throw lastError || new Error("Max retries exceeded");
}

Example 6: Video Generation Pipeline

import { NeuroLink } from "@juspay/neurolink";
import { readdir, readFile, writeFile, mkdir } from "fs/promises";
import path from "path";
import pLimit from "p-limit";

type PipelineConfig = {
  inputDir: string;
  outputDir: string;
  prompts: Record<string, string>; // filename pattern -> prompt
  defaultPrompt: string;
  resolution: "720p" | "1080p";
  aspectRatio: "9:16" | "16:9";
  concurrency: number;
};

async function videoPipeline(config: PipelineConfig) {
  const neurolink = new NeuroLink();
  const limit = pLimit(config.concurrency);

  // Ensure output directory exists
  await mkdir(config.outputDir, { recursive: true });

  // Get all image files
  const files = await readdir(config.inputDir);
  const imageFiles = files.filter((f) => /\.(jpg|jpeg|png|webp)$/i.test(f));

  // Process with concurrency limit
  const results = await Promise.all(
    imageFiles.map((imageFile) =>
      limit(async () => {
        // Find matching prompt pattern or use default
        const prompt =
          Object.entries(config.prompts).find(([pattern]) =>
            imageFile.startsWith(pattern),
          )?.[1] || config.defaultPrompt;

        try {
          const imageBuffer = await readFile(
            path.join(config.inputDir, imageFile),
          );

          const result = await neurolink.generate({
            input: {
              text: prompt,
              images: [imageBuffer],
            },
            provider: "vertex",
            model: "veo-3.1",
            output: {
              mode: "video",
              video: {
                resolution: config.resolution,
                aspectRatio: config.aspectRatio,
                length: 8,
              },
            },
          });

          if (result.video) {
            const outputPath = path.join(
              config.outputDir,
              `${path.basename(imageFile, path.extname(imageFile))}.mp4`,
            );
            await writeFile(outputPath, result.video.data);

            return {
              input: imageFile,
              output: outputPath,
              duration: result.video.metadata?.duration,
              success: true,
            };
          }
          return {
            input: imageFile,
            success: false,
            error: "No video generated",
          };
        } catch (error) {
          return {
            input: imageFile,
            success: false,
            error: error instanceof Error ? error.message : "Unknown error",
          };
        }
      }),
    ),
  );

  return results;
}

// Usage
const pipelineResults = await videoPipeline({
  inputDir: "./raw-images",
  outputDir: "./generated-videos",
  prompts: {
    "product-": "Elegant product rotation with soft lighting",
    "hero-": "Dramatic zoom with cinematic lighting",
    "lifestyle-": "Natural movement with ambient atmosphere",
  },
  defaultPrompt: "Smooth camera movement showcasing the subject",
  resolution: "1080p",
  aspectRatio: "16:9",
  concurrency: 3,
});

console.table(pipelineResults);

Type Definitions

VideoGenerationInput

Extended input type for video generation requests:

// Part of GenerateOptions input - uses existing multimodal types
type VideoGenerationInput = {
  text: string; // Prompt describing desired video motion/style
  images: Array<Buffer | string | ImageWithAltText>; // Input image (required)
};

VideoOutputOptions

Options for video output configuration:

type VideoOutputOptions = {
  /** Output resolution - "720p" (1280x720) or "1080p" (1920x1080) */
  resolution?: "720p" | "1080p";
  /** Video duration in seconds (4, 6, or 8 seconds supported) */
  length?: 4 | 6 | 8;
  /** Aspect ratio - "9:16" for portrait or "16:9" for landscape */
  aspectRatio?: "9:16" | "16:9";
  /** Enable audio generation (default: true) */
  audio?: boolean;
};

VideoGenerationResult

Result type for generated video:

type VideoGenerationResult = {
  /** Raw video data as Buffer */
  data: Buffer;
  /** Video media type */
  mediaType: "video/mp4" | "video/webm";
  /** Video metadata */
  metadata?: {
    /** Original filename if applicable */
    filename?: string;
    /** Video duration in seconds */
    duration?: number;
    /** Video dimensions */
    dimensions?: {
      width: number;
      height: number;
    };
    /** Frame rate in fps */
    frameRate?: number;
    /** Video codec used */
    codec?: string;
    /** Model used for generation */
    model?: string;
    /** Provider used for generation */
    provider?: string;
    /** Aspect ratio of the video */
    aspectRatio?: string;
    /** Whether audio was enabled during generation */
    audioEnabled?: boolean;
    /** Processing time in milliseconds */
    processingTime?: number;
  };
};

Extended GenerateResult

The generate() function returns an extended result when video mode is enabled:

type GenerateResult = {
  content: string; // Text content (prompt echoed back)
  provider?: string;
  model?: string;
  usage?: TokenUsage;
  responseTime?: number;

  // Video-specific field (present when output.mode === "video")
  video?: VideoGenerationResult;

  // Other optional fields
  toolsUsed?: string[];
  analytics?: AnalyticsData;
  evaluation?: EvaluationData;
};

Configuration & Best Practices

Configuration Options

Option Type Default Required Description
input.images[0] Buffer | string - Yes Image buffer, file path, or URL
input.text string - Yes Text description of desired video
output.video.provider VideoProviderName "vertex" No Video handler: vertex (default), kling, runway, replicate. See Routing Across Providers.
output.video.model string provider-default No Model version / checkpoint id (e.g. veo-3.1-generate-001, meta/wan-2-2/i2v)
output.mode string "text" Yes Must be "video" for video output
output.video.resolution string "720p" No Output resolution (720p or 1080p)
output.video.length number 6 No Duration in seconds — provider-dependent: vertex accepts 4/6/8, kling and runway accept 5/10, replicate is model-specific. See Routing Across Providers for per-provider details.
output.video.aspectRatio string "16:9" No Aspect ratio (9:16 or 16:9)
output.video.audio boolean true No Enable audio generation

Video Quality Settings

// High quality for professional content
const professional = await neurolink.generate({
  input: {
    text: "Cinematic product showcase with dramatic lighting",
    images: [await readFile("./product.jpg")],
  },
  provider: "vertex",
  model: "veo-3.1",
  output: {
    mode: "video",
    video: {
      resolution: "1080p",
      length: 8,
      aspectRatio: "16:9",
      audio: true,
    },
  },
});

// Optimized for social media
const social = await neurolink.generate({
  input: {
    text: "Quick product reveal",
    images: [await readFile("./input.jpg")],
  },
  provider: "vertex",
  model: "veo-3.1",
  output: {
    mode: "video",
    video: {
      resolution: "720p",
      length: 4,
      aspectRatio: "9:16",
      audio: true,
    },
  },
});

Best Practices

1. Prompt Engineering

// ❌ Vague and unclear
const vaguePrompt = "Make a video of this product";
// ✅ Specific and actionable
const specificPrompt =
  "Smooth 360-degree rotation of the product with soft studio lighting, camera slowly zooms out";
// ✅ Include camera direction
const cameraDirectionPrompt =
  "Camera slowly pans from left to right, revealing product details with cinematic depth of field";
// ✅ Describe motion and atmosphere
const atmospherePrompt =
  "Dynamic product showcase with subtle particle effects, ambient lighting transitions from warm to cool";

Prompt Template Examples:

Use Case Template
Product Rotation "Elegant 360-degree rotation of [product] with [lighting style] lighting"
Hero Shot "Cinematic zoom from [distance] to [detail] with [motion style] camera movement"
Lifestyle "Natural scene with [subject] in [environment], subtle ambient movement"
Social Media "Quick dynamic reveal of [product] with energetic transitions"

2. Image Preparation

// Image requirements
const imageRequirements = {
  minResolution: "720p", // 1280x720 minimum
  recommendedResolution: "1080p", // 1920x1080 for best results
  formats: ["JPEG", "PNG", "WebP"],
  maxSize: "10MB",
  aspectRatio: "Match desired video output",
};

// Preprocessing recommendations
import sharp from "sharp";

async function prepareImage(inputPath: string, outputRatio: "9:16" | "16:9") {
  const targetWidth = outputRatio === "16:9" ? 1920 : 1080;
  const targetHeight = outputRatio === "16:9" ? 1080 : 1920;

  return sharp(inputPath)
    .resize(targetWidth, targetHeight, {
      fit: "cover",
      position: "center",
    })
    .jpeg({ quality: 90 })
    .toBuffer();
}

3. Performance Optimization

// Parallel processing with rate limiting
import pLimit from "p-limit";

const limit = pLimit(3); // Max 3 concurrent requests (within provider limits)

const images = ["img1.jpg", "img2.jpg", "img3.jpg", "img4.jpg", "img5.jpg"];

const videos = await Promise.all(
  images.map((img) =>
    limit(async () => {
      const result = await neurolink.generate({
        input: {
          text: "Product showcase",
          images: [await readFile(img)],
        },
        provider: "vertex",
        model: "veo-3.1",
        output: { mode: "video", video: { resolution: "720p", length: 8 } },
      });
      return result.video;
    }),
  ),
);

4. Quality vs. Cost Tradeoffs

Setting Quality Cost Use Case
720p, 4s Good Low Quick previews, drafts
720p, 8s Good Medium Social media content
1080p, 6s High High Marketing materials
1080p, 8s Highest Highest Professional productions

Error Handling & Validation

Validation Rules

Parameter Validation Error Type Example Message
input.images[0] Must be valid image file/buffer NeuroLinkError Invalid image format. Supported: JPEG, PNG, WebP
input.images[0] Max 10MB NeuroLinkError Image size exceeds 10MB limit
input.text 1-500 characters NeuroLinkError Prompt must be between 1 and 500 characters
output.video.resolution 720p or 1080p NeuroLinkError Invalid resolution. Use '720p' or '1080p'
output.video.length 4, 6, or 8 NeuroLinkError Invalid length. Use 4, 6, or 8 seconds
output.video.aspectRatio 9:16 or 16:9 NeuroLinkError Invalid aspect ratio. Use '9:16' or '16:9'

Error Types

NeuroLink uses a unified error handling system with error categories:

import { NeuroLinkError } from "@juspay/neurolink";

// Error categories (from ErrorCategory enum)
type ErrorCategory =
  | "validation"
  | "timeout"
  | "network"
  | "resource"
  | "permission"
  | "configuration"
  | "execution"
  | "system";

// Video-specific error codes (standard single-clip generation)
const VIDEO_ERROR_CODES = {
  GENERATION_FAILED: "VIDEO_GENERATION_FAILED",
  PROVIDER_NOT_CONFIGURED: "VIDEO_PROVIDER_NOT_CONFIGURED",
  POLL_TIMEOUT: "VIDEO_POLL_TIMEOUT",
  INVALID_INPUT: "VIDEO_INVALID_INPUT",
  // Director Mode adds additional codes (DIRECTOR_CLIP_FAILED, etc.)
  // See Video Director Mode docs for the full list
};

Note: Video errors are thrown as VideoError (extends NeuroLinkError) with the codes above. Director Mode introduces additional error codes — see Video Director Mode – Error Types for the complete list.

Error Handling Example

import { NeuroLinkError } from "@juspay/neurolink";

try {
  const result = await neurolink.generate({
    input: {
      text: prompt,
      images: [imageBuffer],
    },
    provider: "vertex",
    model: "veo-3.1",
    output: { mode: "video", video: { resolution: "720p" } },
  });
} catch (error) {
  if (error instanceof NeuroLinkError) {
    console.error(`Error [${error.code}]:`, error.message);
    console.error("Category:", error.category);
    console.error("Severity:", error.severity);
    console.error("Retriable:", error.retriable);

    // Handle specific error categories
    switch (error.category) {
      case "validation":
        console.error("Validation issues:");
        // - Unsupported image format (use JPEG, PNG, or WebP)
        // - Image too large (max 10MB)
        // - Invalid prompt length (1-500 characters)
        // - Invalid resolution, length, or aspect ratio
        break;

      case "timeout":
        console.error("Request timed out - retry with backoff");
        break;

      case "configuration":
      case "permission":
        console.error(
          "Config/auth failed - check GOOGLE_APPLICATION_CREDENTIALS",
        );
        break;

      case "network":
        console.error("Network error - retry with backoff");
        break;

      case "execution":
        console.error("Execution error - check status and quotas");
        // Detect rate limiting via error code
        if (error.code.includes("RATE_LIMIT")) {
          console.error("Rate limited - implement exponential backoff");
        }
        break;
    }
  }
}

Token & Cost Information

Pricing Structure

Resolution Duration Estimated Cost Notes
720p 4 seconds ~$1.60 Best for previews
720p 8 seconds ~$3.20 Standard quality
1080p 4 seconds ~$2.00 High quality short
1080p 8 seconds ~$4.00 Professional quality

Note: Pricing is approximate and subject to change (as of October 2025). Check Google Cloud pricing for current rates.

Storage Costs

Resolution Duration Approx. File Size
720p 4 seconds ~1-2 MB
720p 8 seconds ~2-4 MB
1080p 4 seconds ~2-3 MB
1080p 8 seconds ~4-6 MB

Working with Video Results

import { NeuroLink } from "@juspay/neurolink";
import { readFile, writeFile } from "fs/promises";

const neurolink = new NeuroLink();

// Generate video
const result = await neurolink.generate({
  input: {
    text: "Product showcase video",
    images: [await readFile("./product.jpg")],
  },
  provider: "vertex",
  model: "veo-3.1",
  output: { mode: "video" },
});

// Check for video result
if (result.video) {
  // Save to file
  await writeFile("output.mp4", result.video.data);

  // Access metadata
  console.log({
    duration: result.video.metadata?.duration,
    resolution: result.video.metadata?.dimensions,
    model: result.video.metadata?.model,
    size: result.video.data.length,
  });
}

Troubleshooting

Symptom Cause Solution
Authentication error Invalid or missing credentials Verify GOOGLE_APPLICATION_CREDENTIALS is set correctly
Authorization error Service account lacks permissions Add aiplatform.user role to service account
Validation error (format) Unsupported image type Convert image to JPEG, PNG, or WebP
Validation error (size) Image exceeds 10MB limit Compress or resize image before upload
Rate limit error Too many requests Implement exponential backoff
Network timeout Processing took too long Try lower resolution or shorter duration
Provider quota exceeded Monthly quota reached Request quota increase or wait for reset
Connection error Network issues Check network connectivity; retry with backoff
Video quality is poor Low resolution input image Use minimum 720p source images
Audio not matching video Complex scene Simplify prompt; focus on visual elements
Unexpected aspect ratio Input image ratio mismatch Preprocess image to match target aspect ratio

Debug Mode

// Enable verbose logging for debugging
const neurolink = new NeuroLink({
  debug: true,
  logLevel: "verbose",
});

// Or via environment variable
// export NEUROLINK_DEBUG=true

Limitations

Current Limitations

Limitation Description Workaround
Max duration Provider-specific: Vertex 4/6/8 s, Runway 5/10 s, Kling 5/10 s, Replicate per-model. Chain multiple clips via Video Director Mode (Vertex)
Audio input No custom audio supported on any provider Audio is auto-generated (Vertex Veo) or absent (Kling / Runway / Replicate)
Text-only prompts All four providers require an image (and Kling needs a public URL — see Routing Across Providers) Generate an image first, then pass it as input.images[0]
Director Mode Currently Vertex-only Generate per-segment clips with non-Vertex providers and stitch externally
Concurrent requests Provider-specific (Vertex: 5/project; Replicate: 6/min on free tier) Implement request queuing

Testing

Unit Test Examples

import { describe, it, expect, vi } from "vitest";
import { NeuroLink } from "@juspay/neurolink";

describe("Video Generation", () => {
  it("should generate video with valid inputs", async () => {
    const neurolink = new NeuroLink();
    const imageBuffer = Buffer.from("fake-image-data");

    const result = await neurolink.generate({
      input: {
        text: "Test video generation",
        images: [imageBuffer],
      },
      provider: "vertex",
      model: "veo-3.1",
      output: { mode: "video", video: { resolution: "720p", length: 8 } },
    });

    expect(result.video).toBeDefined();
    expect(result.video?.data).toBeInstanceOf(Buffer);
    expect(result.video?.metadata?.duration).toBe(8);
  });

  it("should throw error for invalid image format", async () => {
    const neurolink = new NeuroLink();

    await expect(
      neurolink.generate({
        input: {
          text: "Test",
          images: ["invalid-file.txt"],
        },
        provider: "vertex",
        model: "veo-3.1",
        output: { mode: "video" },
      }),
    ).rejects.toThrow(); // Should throw ValidationError
  });

  it("should respect resolution settings", async () => {
    const neurolink = new NeuroLink();
    const imageBuffer = Buffer.from("fake-image-data");

    const result = await neurolink.generate({
      input: {
        text: "Test",
        images: [imageBuffer],
      },
      provider: "vertex",
      model: "veo-3.1",
      output: { mode: "video", video: { resolution: "1080p" } },
    });

    expect(result.video?.metadata?.dimensions?.width).toBe(1920);
    expect(result.video?.metadata?.dimensions?.height).toBe(1080);
  });
});

Mock Strategy for CI/CD

import { vi } from "vitest";

// Mock the NeuroLink class to return video generation results
vi.mock("@juspay/neurolink", () => ({
  NeuroLink: vi.fn().mockImplementation(() => ({
    generate: vi.fn().mockResolvedValue({
      content: "",
      provider: "vertex",
      model: "veo-3.1",
      video: {
        data: Buffer.from("mock-video-data"),
        mediaType: "video/mp4",
        metadata: {
          duration: 8,
          dimensions: { width: 1920, height: 1080 },
          model: "veo-3.1",
        },
      },
    }),
  })),
}));

Integration Test Pattern

import { describe, it, expect } from "vitest";
import { NeuroLink } from "@juspay/neurolink";
import { readFile } from "fs/promises";

describe("Video Generation Integration", () => {
  it("should complete full generation workflow", async () => {
    // Skip in CI without credentials
    if (!process.env.GOOGLE_APPLICATION_CREDENTIALS) {
      console.log("Skipping: No Google credentials");
      return;
    }

    const neurolink = new NeuroLink();
    const imageBuffer = await readFile("./test-fixtures/sample-image.jpg");

    const result = await neurolink.generate({
      input: {
        text: "Smooth camera pan for product showcase",
        images: [imageBuffer],
      },
      provider: "vertex",
      model: "veo-3.1",
      output: {
        mode: "video",
        video: { resolution: "720p", length: 4 },
      },
    });

    expect(result.video).toBeDefined();
    expect(result.video?.data).toBeInstanceOf(Buffer);
    expect(result.video?.data.length).toBeGreaterThan(0);
    expect(result.video?.metadata?.duration).toBe(4);
  }, 180000); // 3 minute timeout for video generation
});

Related Features

Implementation Files

The video generation feature is implemented across these files:

File Purpose
src/lib/types/multimodal.ts Core types: VideoOutputOptions, VideoGenerationResult
src/lib/types/generateTypes.ts Extended GenerateOptions with video output mode
src/lib/adapters/video/vertexVideoHandler.ts Vertex AI Veo 3.1 video generation handler, VideoError, VIDEO_ERROR_CODES
src/lib/adapters/video/ffmpegAdapter.ts Shared FFmpeg adapter (binary resolution, temp files, buffer validation)
src/lib/adapters/video/frameExtractor.ts Frame extraction from MP4 buffers (used by Director Mode)
src/lib/adapters/video/videoMerger.ts MP4 concatenation via FFmpeg concat demuxer (used by Director Mode)
src/lib/adapters/video/directorPipeline.ts Director Mode pipeline orchestrator (multi-segment generation)
src/lib/core/baseProvider.ts Video generation routing in generate() method
src/lib/neurolink.ts Main SDK interface with video result handling
src/lib/utils/parameterValidation.ts Input validation: validateVideoGenerationInput(), validateImageForVideo()
src/lib/utils/errorHandling.ts Error factory methods for video generation errors

Key Functions

  • generateVideoWithVertex() - Main video generation function in vertexVideoHandler.ts
  • generateTransitionWithVertex() - Transition generation with first-and-last-frame API (Director Mode)
  • validateVideoGenerationInput() - Comprehensive input validation in parameterValidation.ts
  • validateImageForVideo() - Image format and size validation in parameterValidation.ts
  • handleVideoGeneration() - Private method in BaseProvider that orchestrates the video generation flow
  • executeDirectorPipeline() - Director Mode orchestrator (parallel clips, transitions, merge)

Next: Multimodal Chat Guide | Video Director Mode