OpenMontage 中 HeyGen 数字人视频与 Remotion 合成集成实战从 /v2/video/generate 到帧精确渲染【免费下载链接】OpenMontageWorlds first open-source, agentic video production system. 12 production pipelines, 100 tools, 700 agent skill and production-knowledge files. Turn your AI coding assistant into a full video production studio.项目地址: https://gitcode.com/GitHub_Trending/op/OpenMontage本篇基于 OpenMontage 仓库中 avatar-video 技能包下的参考文档 remotion-integration.md系统讲解如何调用 HeyGen 生成数字人avatar视频并将其以帧精确的方式嵌入 Remotion 合成工程中。读完后你将掌握MP4 背景版与 WebM 透明版两种输出格式的选择依据、尺寸/帧率对齐方法、OffthreadVideo的帧精确渲染原理、calculateMetadata动态时长方案以及完整的“生成—轮询—bundle—渲染”生产工作流。1. 集成定位与整体工作流该参考文档是 avatar-video 技能SKILL.md的“Integration”部分专门回答一个问题HeyGen 生成的数字人视频如何进入 Remotion 程序化合成流水线。在 OpenMontage 中Remotion 工程位于 remotion-composer 目录其Root.tsx注册了TalkingHead、Explainer、CinematicRenderer等多个Composition其中TalkingHead合成正是“数字人视频 叠加层 字幕”的三层结构与本文档描述的合成模式一致。典型工作流四步调用 HeyGen 生成数字人视频轮询状态直至完成获取视频 URL下载视频或直接在 Remotion 中引用 URL在 Remotion 中与其他元素背景、叠加层、动效合成。快速上手// 1. Get avatar with default voice const avatar await getAvatarDetails(avatarId); // 2. Generate video (MP4 with background - most common) const videoId await generateVideo({ video_inputs: [{ character: { type: avatar, avatar_id: avatar.id, avatar_style: normal }, voice: { type: text, input_text: script, voice_id: avatar.default_voice_id }, background: { type: color, value: #1a1a2e }, }], dimension: { width: 1920, height: 1080 }, }); // 3. Poll for completion (10-15 min) // 4. Use in Remotion with motion graphics overlaid on top生成接口为POST https://api.heygen.com/v2/video/generate请求需携带X-Api-Key头技能包的 frontmatter 声明了环境变量要求HEYGEN_API_KEY。完整的请求字段表、多场景视频与video_inputs结构见同目录参考文档 video-generation.md轮询实现、状态类型与下载重试见 video-status.md。注意区分仓库中的 heygen_video.py 是一个以 HeyGen 为通道调度 VEO/Sora/Kling 等模型的云端视频生成工具与本文讨论的 HeyGen 原生 avatar 接口是两回事本地的照片驱动说话头工具则见 talking_head.py基于 SadTalker。avatar 集成工作流完全走 avatar-video 技能包的 HTTP API 路径。2. 选择正确的输出格式MP4 背景版 vs WebM 透明版这是本集成中最关键的一个决策。参考文档给出的选择矩阵你的合成形态推荐格式原因数字人作为主讲人动效叠加在其上MP4 背景色更简单叠加层直接放上层Loom 风格数字人悬浮于录屏之上WebM closeUpRemotion 中加遮罩需要透明背景用 CSS 做圆形遮罩数字人叠加在其他视频/内容之上WebM透明需要“看穿”到数字人背后的内容全屏数字人MP4 背景色标准做法多数场景直接用带背景的 MP4只有当需要透过数字人看到其背后内容时才用 WebM。两个端点的差异POST /v2/video/generate→ 输出 MP4支持normal/closeUp/circle三种avatar_stylePOST /v1/video.webm→ 输出透明 WebM仅支持normal和closeUp圆形取景需在 Remotion 中用 CSSborder-radius: 50%实现。/v1/video.webm与/v2/video/generate的请求结构不同前者是扁平字段avatar_pose_id、avatar_style、input_text、voice_id等字段约束与“input_textvoice_id二选一搭配input_audio”的规则见 video-generation.md 的 WebM 章节。3. 并行开发工作流不要干等 10–15 分钟HeyGen 视频生成通常需要10–15 分钟以上video-status.md 给出的超时建议是 15–20 分钟。文档推荐的工作方式是提交即退出、并行开发启动 HeyGen 生成— 把video_id存到文件进程立即退出video-status.md中的“Resumable Status Checking”章节给出了pending-video.json存状态 稍后查询的完整实现搭建 Remotion 合成— 使用占位视频或数字人的preview_video_url一段短循环片段定期查询 HeyGen 状态— 构建完成后或周期性轮询就绪后替换占位— 换成真实视频 URL。两个实用的工程技巧按文稿估算时长按约 150 词/分钟的语速wordCount / 150 * 60 * fps即可得到近似的帧数用于在视频就绪前就搭好合成骨架组件解耦设计让组件在“有/没有数字人视频”两种状态下都能工作这样动效部分可以独立测试与预览。4. 尺寸对齐HeyGen 与 Remotion 共用一份常量关键原则HeyGen 的输出尺寸必须与 Remotion 合成完全一致。文档给出共享的维度常量// Shared dimension constants for both HeyGen and Remotion const DIMENSIONS { landscape_1080p: { width: 1920, height: 1080 }, landscape_720p: { width: 1280, height: 720 }, portrait_1080p: { width: 1080, height: 1920 }, portrait_720p: { width: 720, height: 1280 }, square_1080p: { width: 1080, height: 1080 }, square_720p: { width: 720, height: 720 }, } as const; type DimensionPreset keyof typeof DIMENSIONS;这套预设与技能包 dimensions.md 的官方分辨率表一致横屏 16:91280×720 / 1920×1080、竖屏 9:16720×1280 / 1080×1920、方形 1:1720×720 / 1080×1080。对应到 Remotion 侧的Root.tsx注册// remotion/src/Root.tsx import { Composition } from remotion; import { AvatarComposition } from ./AvatarComposition; export const RemotionRoot: React.FC () { return ( Composition idAvatarVideo component{AvatarComposition} durationInFrames{300} // Will be set dynamically fps{30} width{DIMENSIONS.landscape_1080p.width} height{DIMENSIONS.landscape_1080p.height} defaultProps{{ avatarVideoUrl: }} / / ); };仓库内的真实工程印证了这个模式Root.tsx 注册TalkingHead合成时采用竖屏1080×1920、fps{30}而 TalkingHead.tsx 内部的POSITION_STYLESlower_third、left_panel等位置预设正是按 9:16 的 1080×1920 画布坐标设计的——尺寸常量一旦错位所有像素级定位都会跟着错位。5. 生成带背景的 MP4标准生成函数async function generateHeyGenVideo( script: string, avatarId: string, voiceId: string, preset: DimensionPreset ): Promisestring { const dimension DIMENSIONS[preset]; const response await fetch(https://api.heygen.com/v2/video/generate, { method: POST, headers: { X-Api-Key: process.env.HEYGEN_API_KEY!, Content-Type: application/json, }, body: JSON.stringify({ video_inputs: [ { character: { type: avatar, avatar_id: avatarId, avatar_style: normal, }, voice: { type: text, input_text: script, voice_id: voiceId, }, background: { type: color, value: #00FF00, // Green screen for compositing }, }, ], dimension, }), }); const { data } await response.json(); return data.video_id; }封装成面向 Remotion 的生成函数时把风格与背景色参数化async function generateAvatarForRemotion( script: string, avatarId: string, voiceId: string, options: { style?: normal | closeUp | circle; backgroundColor?: string; } {} ): Promisestring { const { style normal, backgroundColor #1a1a2e } options; const response await fetch(https://api.heygen.com/v2/video/generate, { method: POST, headers: { X-Api-Key: process.env.HEYGEN_API_KEY!, Content-Type: application/json, }, body: JSON.stringify({ video_inputs: [{ character: { type: avatar, avatar_id: avatarId, avatar_style: style, }, voice: { type: text, input_text: script, voice_id: voiceId, }, background: { type: color, value: backgroundColor }, }], dimension: { width: 1920, height: 1080 }, }), }); const { data } await response.json(); return data.video_id; }请求字段补充说明源自 video-generation.md 的字段表character.avatar_style可取normal/closeUp/circlevoice.type除text外还支持audio需audio_url与silence需durationspeed取值 0.5–2.0pitch取值 -20 到 20background.type除color外还支持image/video用urlfit字段顶层还支持test: true测试模式不扣额度、带水印、caption自动字幕、callback_urlWebhook 通知——生产工作流建议开发阶段一律开测试模式。6. 透明背景 WebM/v1/video.webm端点只有在需要“看到数字人背后的内容”时才用 WebM例如数字人悬浮在录屏之上// Use /v1/video.webm endpoint for transparent background // Note: Different structure than /v2/video/generate const response await fetch(https://api.heygen.com/v1/video.webm, { method: POST, headers: { X-Api-Key: process.env.HEYGEN_API_KEY!, Content-Type: application/json, }, body: JSON.stringify({ avatar_pose_id: avatarPoseId, // Required: avatar pose ID avatar_style: normal, // Required: normal or closeUp only input_text: script, // Required (with voice_id) voice_id: voiceId, // Required (with input_text) dimension: { width: 1920, height: 1080 }, }), });注意avatar_pose_id来自数字人详情接口GET /v2/avatar/{avatar_id}/detailsavatar_style只有normal/closeUp两个合法值。WebM 与 MP4 共用同一状态查询端点GET /v2/videos/{video_id}完成后video_url指向.webm文件。7. 在 Remotion 中使用 HeyGen 视频务必使用 OffthreadVideo核心结论数字人视频必须用OffthreadVideo而非Video。基础Video组件依赖浏览器解码器渲染时并非帧精确会产生抖动jitterOffthreadVideo通过 FFmpeg 逐帧抽帧得到平滑、帧精确的输出。它包含在remotion核心包中无需额外安装。仓库源码可以直接印证这一实践TalkingHead.tsx 的第一层就是OffthreadVideo并且素材先经过resolveAsset(videoSrc)归一化为本地路径对应后文“先下载再用”策略// Layer 1: Video background OffthreadVideo src{resolveAsset(videoSrc)} style{{ width: 100%, height: 100%, objectFit: cover }} /基本用法// remotion/src/AvatarComposition.tsx import { OffthreadVideo, useVideoConfig } from remotion; interface AvatarCompositionProps { avatarVideoUrl: string; } export const AvatarComposition: React.FCAvatarCompositionProps ({ avatarVideoUrl, }) { return ( div style{{ flex: 1, backgroundColor: #1a1a2e }} OffthreadVideo src{avatarVideoUrl} style{{ width: 100%, height: 100%, objectFit: contain, }} / /div ); };WebM 透明背景三层结构使用/v1/video.webm产物时不需要任何色度键chroma key处理加transparent属性即可import { OffthreadVideo, AbsoluteFill, Sequence } from remotion; export const AvatarWithMotionGraphics: React.FC{ avatarWebmUrl: string } ({ avatarWebmUrl }) { return ( AbsoluteFill {/* Layer 1: Your background/content */} AbsoluteFill style{{ backgroundColor: #1a1a2e }} YourMotionGraphics / /AbsoluteFill {/* Layer 2: Avatar with transparent background */} OffthreadVideo src{avatarWebmUrl} transparent style{{ position: absolute, bottom: 0, right: 0, width: 50%, height: auto, }} / {/* Layer 3: Overlays on top of avatar */} Sequence from{30} AnimatedTitle textWelcome! / /Sequence /AbsoluteFill ); };Loom 风格圆形数字人悬浮于录屏之上closeUp风格 WebM圆形取景在 Remotion 侧用 CSS 实现import { OffthreadVideo, AbsoluteFill } from remotion; export const LoomStyleComposition: React.FC{ screenRecordingUrl: string; avatarWebmUrl: string; // Generated with avatar_style: closeUp via /v1/video.webm } ({ screenRecordingUrl, avatarWebmUrl }) { return ( AbsoluteFill {/* Screen recording fills the frame */} OffthreadVideo src{screenRecordingUrl} style{{ width: 100%, height: 100% }} / {/* Avatar with circular mask - transparent bg shows screen behind */} OffthreadVideo src{avatarWebmUrl} transparent style{{ position: absolute, bottom: 40, left: 40, width: 180, height: 180, borderRadius: 50%, // Circular mask applied in CSS overflow: hidden, objectFit: cover, }} / /AbsoluteFill ); };再次强调WebM 不支持circle风格圆形必须靠borderRadius: 50%遮罩。分层合成背景图 数字人 标题 Logoimport { OffthreadVideo, Sequence, useVideoConfig, Img } from remotion; interface LayeredAvatarProps { avatarVideoUrl: string; backgroundUrl: string; logoUrl: string; title: string; } export const LayeredAvatarComposition: React.FCLayeredAvatarProps ({ avatarVideoUrl, backgroundUrl, logoUrl, title, }) { const { fps } useVideoConfig(); return ( div style{{ position: relative, width: 100%, height: 100% }} {/* Layer 1: Background */} Img src{backgroundUrl} style{{ position: absolute, width: 100%, height: 100%, objectFit: cover, }} / {/* Layer 2: Avatar video - use OffthreadVideo to prevent jitter */} OffthreadVideo src{avatarVideoUrl} style{{ position: absolute, bottom: 0, right: 0, width: 40%, height: auto, }} / {/* Layer 3: Title (appears after 1 second) */} Sequence from{fps} div style{{ position: absolute, top: 50, left: 50, color: white, fontSize: 48, fontWeight: bold, }}{title}/div /Sequence {/* Layer 4: Logo */} Img src{logoUrl} style{{ position: absolute, top: 20, right: 20, width: 100, height: auto, }} / /div ); };这种“底层视频 中层人物 上层动效”的三层结构与 TalkingHead.tsx 的实现完全同构Layer 1 是OffthreadVideo视频底Layer 2 是带Sequence时间轴的叠加卡片通过in_seconds/out_seconds换算成帧Layer 3 是最顶层的字幕CaptionOverlay。仓库还实现了lower_third/upper_third/left_panel/right_panel/full_overlay五种位置预设每个叠加层带 8 帧淡入淡出——这些细节正是“叠加层放在数字人视频之上”这一思路的完整落地。旧方案绿幕 色度键不推荐若你手里只有绿底 MP4文档给出的“旧方案”仅是基础混合模式// Note: True chroma key requires WebGL or post-processing // WebM transparent background is much simpler OffthreadVideo src{avatarVideoUrl} style{{ mixBlendMode: multiply }} // Basic compositing only /文档明确建议真正的色度键需要 WebGL 或后处理优先改用 WebM 透明背景。8. 完整工作流生成、轮询、Bundle、渲染把生成与渲染串起来的端到端函数import { bundle } from remotion/bundler; import { renderMedia, selectComposition } from remotion/renderer; async function generateAvatarVideoForRemotion( script: string, outputPath: string ) { // 1. Generate HeyGen video console.log(Generating HeyGen avatar video...); const videoId await generateHeyGenVideo( script, josh_lite3_20230714, 1bd001e7e50f421d891986aad5158bc8, landscape_1080p ); // 2. Wait for completion console.log(Waiting for HeyGen video...); const avatarVideoUrl await waitForVideo(videoId); console.log(HeyGen video ready: ${avatarVideoUrl}); // 3. Get video duration for Remotion const avatarDuration await getVideoDuration(avatarVideoUrl); const durationInFrames Math.ceil(avatarDuration * 30); // 30 fps // 4. Bundle Remotion project console.log(Bundling Remotion project...); const bundleLocation await bundle({ entryPoint: ./remotion/src/index.ts, }); // 5. Select composition const composition await selectComposition({ serveUrl: bundleLocation, id: AvatarVideo, inputProps: { avatarVideoUrl }, }); // 6. Render final video console.log(Rendering final composition...); await renderMedia({ composition: { ...composition, durationInFrames }, serveUrl: bundleLocation, codec: h264, outputLocation: outputPath, inputProps: { avatarVideoUrl }, }); console.log(Final video rendered: ${outputPath}); return outputPath; }其中waitForVideo即 video-status.md 中的轮询实现每 5 秒查询一次GET /v2/videos/{video_id}状态为pending/processing时继续等待completed时返回video_urlfailed时抛出failure_message。状态机四态pending→processing→completed/failed以及“状态显示 completed 后 URL 可能还不可立即访问、下载要带指数退避重试”的细节都在该文档中。动态时长calculateMetadata数字人视频的实际时长取决于文稿长度硬编码durationInFrames并不合理。Remotion 的calculateMetadata可以在渲染前根据inputProps动态计算合成元数据// remotion/src/AvatarComposition.tsx import { CalculateMetadataFunction } from remotion; export const calculateAvatarMetadata: CalculateMetadataFunction AvatarCompositionProps async ({ props }) { // Fetch video duration from HeyGen video const duration await getVideoDurationInSeconds(props.avatarVideoUrl); return { durationInFrames: Math.ceil(duration * 30), fps: 30, width: 1920, height: 1080, }; }; // In Root.tsx Composition idAvatarVideo component{AvatarComposition} calculateMetadata{calculateAvatarMetadata} defaultProps{{ avatarVideoUrl: }} /这个模式在仓库真实工程中同样是标准做法Root.tsx 中Explainer合成的calculateMetadata依据cuts的最后一个out_seconds计算durationInFrames外加 1 秒收尾淡出CinematicRenderer、TitledVideo也各自挂载了calculateCinematicMetadata、calculateTitledVideoMetadata——可见“按素材动态算时长”是 OpenMontage Remotion 工程的一贯范式。9. 最佳实践9.1 用绿幕换取合成灵活性希望后期合成时生成阶段直接给绿底background: { type: color, value: #00FF00, // Pure green for chroma key }配合 7 节末尾的说明真色度键成本较高能走 WebM 透明背景就走 WebM。9.2 帧率对齐HeyGen 默认 25 fpsHeyGen 输出默认 25 fpsRemotion 合成帧率与之不一致时需要显式处理// Option 1: Match HeyGens 25 fps fps: 25 // Option 2: Use 30 fps with playback rate adjustment OffthreadVideo src{avatarVideoUrl} playbackRate{25/30} // Slow down slightly to match /9.3 URL 直连 vs 先下载直接用 URL的适用条件Remotion Studio 预览npm run dev、URL 在渲染完成前不会过期、追求开发期快速迭代// Direct URL usage - simpler, faster for dev OffthreadVideo src{avatarVideoUrl} /先下载的适用条件HeyGen 的 URL 约24 小时后过期渲染会延后或重复进行网络可靠性存疑需要离线渲染。带指数退避的下载实现// Download with retry for reliability async function downloadVideoWithRetry( url: string, outputPath: string, maxRetries 5 ): Promisestring { for (let attempt 0; attempt maxRetries; attempt) { try { const response await fetch(url); if (!response.ok) throw new Error(HTTP ${response.status}); const buffer await response.arrayBuffer(); await fs.promises.writeFile(outputPath, Buffer.from(buffer)); return outputPath; } catch (error) { const delay 2000 * Math.pow(2, attempt); console.log(Retry ${attempt 1}/${maxRetries} in ${delay}ms...); await new Promise((r) setTimeout(r, delay)); } } throw new Error(Download failed after retries); } // Use local file in Remotion const localPath await downloadVideoWithRetry(avatarVideoUrl, ./public/avatar.mp4);混合方案生产推荐元数据中同时保存 URL 与本地路径本地优先// Save both URL and local path in metadata const metadata { videoUrl: result.video_url, // For quick preview localPath: ./public/avatar.mp4, // For reliable rendering expiresAt: Date.now() 24 * 60 * 60 * 1000, // URL expiration }; // In Remotion component, prefer local if available const videoSrc fs.existsSync(localPath) ? staticFile(avatar.mp4) : avatarVideoUrl;resolveAsset.ts 在仓库中扮演的正是“URL/本地路径归一化”的角色TalkingHead合成入口的resolveAsset(videoSrc)保证渲染时拿到的始终是可访问的资源引用。9.4 数字人位置预设const AVATAR_POSITIONS { fullscreen: { width: 100%, height: 100%, position: center }, bottomRight: { width: 40%, bottom: 0, right: 0 }, bottomLeft: { width: 40%, bottom: 0, left: 0 }, pictureInPicture: { width: 25%, bottom: 20, right: 20 }, leftThird: { width: 33%, left: 0, height: 100% }, };10. 输出格式与质量参数HeyGen 输出MP4H.264音频 AAC分辨率即请求时指定的dimensionRemotion 输出可选 H.264默认、VP8、VP9、ProRes质量设置应不低于 HeyGen 源await renderMedia({ codec: h264, crf: 18, // High quality // ... });11. 故障排查视频在 Remotion 中不播放检查 URL 可访问性CORS 问题确认视频格式兼容先下载到本地再试。尺寸不匹配HeyGen 与 Remotion 必须使用完全相同的尺寸建议共用一份配置// Shared config const VIDEO_CONFIG { width: 1920, height: 1080, fps: 30 }; // HeyGen dimension: { width: VIDEO_CONFIG.width, height: VIDEO_CONFIG.height } // Remotion Composition width{VIDEO_CONFIG.width} height{VIDEO_CONFIG.height} /渲染时视频抖动用OffthreadVideo替换Video——基础Video使用浏览器解码器非帧精确只需改导入核心包已包含无需额外安装// Before (causes jitter) import { Video } from remotion; // After (frame-accurate) import { OffthreadVideo } from remotion;透明 WebM 记得加transparent属性OffthreadVideo src{avatarWebmUrl} transparent /。音画不同步音频漂移核对源视频帧率检查是否存在编码问题考虑用一致参数重新编码。12. 小结把该文档放回仓库语境这篇参考文档在 avatar-video 技能中承担“Integration”角色与技能包其他参考文档形成完整闭环avatars.md 选数字人与默认音色、video-generation.md 管请求构造、video-status.md 管轮询与下载、dimensions.md 管分辨率而本文聚焦的 remotion-integration 则负责“最后一公里”——把云生成的 MP4/WebM 以帧精确、透明安全、尺寸一致的方式嵌进 Remotion 程序化合成。OpenMontage 的 remotion-composer 工程OffthreadVideo底座、位置预设叠加层、calculateMetadata动态时长与 talking_head.py 本地工具共同构成了数字人视频从生成到合成的完整技术栈照本文档的步骤即可复制出一套可运行的 avatar 成片流水线。【免费下载链接】OpenMontageWorlds first open-source, agentic video production system. 12 production pipelines, 100 tools, 700 agent skill and production-knowledge files. Turn your AI coding assistant into a full video production studio.项目地址: https://gitcode.com/GitHub_Trending/op/OpenMontage创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考