Reference-to-Video (ref2va) with Sound in ComfyUI (Workflow Included) – MiniMax H3 Reference-to-Video
Keeping the same character face, outfit, or voice across multiple shots is one of the hardest parts of AI video generation. With MiniMax H3 Reference-to-Video in ComfyUI, you can combine up to 9 reference images, 3 reference videos, and 3 audio clips to generate multi-shot 24fps video with…
Annons
Annons
Keeping the same character face, outfit, or voice across multiple shots is one of the hardest parts of AI video generation. With MiniMax H3 Reference-to-Video in ComfyUI, you can combine up to 9 reference images, 3 reference videos, and 3 audio clips to generate multi-shot 24fps video with synchronized stereo dialogue, sound effects, and music in a single pass. Generated from two separate character reference images in one 5-second pass, complete with lip-synced English dialogue, rain ambience, plasma sound effects, and score. From left to right: Reference Image 1 (), Reference Image 2 (), and the two generated camera shots combining both subjects. In this tutorial, you will learn how to: Set up the MiniMax H3 reference-to-video (ref2va) workflow in ComfyUI. Combine multiple character and creature reference images across camera cuts. Connect reference videos for character replacement and reference audio for voice cloning or lip-sync. Structure a full-reference prompt that avoids the last-frame morphing bug. Software needed We will use ComfyUI, a free AI image and video generator. You can use it on Windows, Mac, or Google Colab. Think Diffusion provides an online ComfyUI service if you don’t have a capable local GPU. (20% extra credit) What is MiniMax H3 Reference-to-Video? MiniMax H3 (known commercially as Hailuo 3.0) is an open-weights, omni-modal diffusion transformer that generates video and native stereo audio simultaneously. While the MiniMax H3 image-to-video (fl2va) model locks onto an opening or closing keyframe, the Reference-to-Video (ref2va) model injects encoded reference tokens into every denoising step so you can direct new scenes while preserving identity, style, motion, or voice. Using the built-in MiniMaxH3ReferenceToVideo node, the same workflow handles four core tasks: Multi-image character and style consistency: Feed up to 9 reference images ( to ) to keep character faces, outfits, props, and environments consistent across multi-shot camera cuts. Character replacement and motion transfer: Connect a reference video-to-video clip () alongside character photos to swap the actor while keeping the original camera movement, lighting, and choreography. Zero-shot voice cloning and lip-sync: Pass a reference audio clip () either to clone a speaker’s vocal timbre for newly written dialogue or to copy a voiceover or song track so the character lip-syncs to the exact waveform. Joint stereo sound design: Generate spoken lines, physical Foley, ambient room tone, and background music aligned frame-by-frame with the visual action. MiniMax H3 reference-to-video workflow Step 0: Update ComfyUI MiniMax H3 requires ComfyUI 0.30.0 or newer (0.36.0+ is recommended for faster VAE decoding). Update ComfyUI before loading the workflow, ideally using ComfyUI Manager. Click the Manager button on the top toolbar. Select Update ComfyUI. Restart ComfyUI. Step 1: Download the models Download these five files and place them in their matching directories: Download minimax_h3_ref2va_pruned_int8_convrot.safetensors (19.5 GB) and put it in ComfyUI > models > diffusion_models. Download qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors (14.6 GB) and put it in ComfyUI > models > text_encoders. Download minimax_h3_video_vae_fp16.safetensors (4.85 GB) and put it in ComfyUI > models > vae. Download minimax_h3_audio_vae_fp32.safetensors (577 MB) and put it in ComfyUI > models > vae. Download minimax_h3_ref2v_turbo_4step_v0.1_comfyui_bf16.safetensors (1.82 GB) and put it in ComfyUI > models > loras. If you already completed the MiniMax H3 text-to-video or Image-to-Video tutorials, you already have the text encoder and audio VAE. You can also reuse minimax_h3_video_vae_int8_convrot.safetensors (2.62 GB) instead of downloading the fp16 video VAE; just select minimax_h3_video_vae_int8_convrot.safetensors in the top Load VAE node (#119). Two important checks before moving on: Make sure you download the ref2va diffusion model and ref2v LoRA above, not the fl2va files used for standard text-to-video and first-frame animation. Keep both VAEs connected to the MiniMax H3 Reference to Video node (#136) as well as the decode nodes. If vae or audio_vae is disconnected on the conditioning node, your references will only condition the text encoder and will lose latent identity lock. Optional style shortcuts: Download any of the 10 style embeddings (minimaxh3_*.safetensors) and put them in ComfyUI > models > embeddings. Step 2: Load the workflow Download the workflow JSON file below and drag it into the ComfyUI window. Download Unlike the subgraph-based text-to-video and image-to-video templates, this workflow lays out all 27 nodes directly on the canvas across six groups: Models, User Inputs, Conditioning, Switch Model and Settings, Sampling, and Decoding and create video. Step 3: Install missing nodes If you see nodes with red borders after loading the workflow, you may need to install missing nodes in ComfyUI Manager or update your ComfyUI build. Click Manager > Install Missing Custom Nodes. Install any nodes that are listed as missing (and update ComfyUI if prompted). Restart ComfyUI and refresh your browser. Step 4: Upload the reference images (or video and audio) In the User Inputs group, upload your reference images to the two Load Image nodes: The left Load Image node (#137) connects to ref_image_0 on MiniMax H3 Reference to Video, which corresponds to in your prompt. The right Load Image node (#139) connects to ref_image_1, which corresponds to in your prompt. You can download and use these two 16:9 test images to reproduce the cyber-ronin and mechanical kitsune demo: Reference Image 1 (ref_image_0 / ): Cyber-ronin character (). Reference Image 2 (ref_image_1 / ): Mechanical kitsune guardian (). Notice that the MiniMax H3 Reference to Video node automatically spawns a new empty slot (ref_image_2, up to ref_image_8) each time you connect an image. How to connect a reference video or voice clip (optional): Reference video (): Do not drag a standard Load Video node’s VIDEO output directly to ref_video_0, because ref_video_0 expects decoded IMAGE frames. Instead, connect Load Video to the built-in Get Video Components node, then wire images to ref_video_0 (and optionally audio to ref_video_audio_0). Standalone reference audio (): Add a built-in Load Audio node and connect its AUDIO output to ref_audio_0 for voice cloning or lip-sync. Step 5: Set the resolution and duration In the User Inputs group, configure your output canvas and clip length: On the Resolution Selector (Size) node (#115), set aspect_ratio to 16:9 (Widescreen). Set megapixels to 0.4 (864x480) for fast drafting, or 0.98 (1344x768) for full 768p quality. Leave multiple at 32. Set Float (Duration) (#132) to 5 seconds. The collapsed Math Expression node right below it automatically snaps 5 seconds at 24fps to 124 frames (about 5.17 seconds) to match H3’s 17-frame block grid ($17k + 5$). Step 6: Revise the prompt The full demo prompt is already loaded inside the Input Text (Prompt) node (#138). When writing your own reference-to-video prompt, follow these structure rules and tips: Declare your reference tags up front: Open with a look and style header (Live-action cinematic sci-fi fantasy film look...) and state which connected tags to use (Use and as character references). Tags are numbered 1-based per modality in the order they are connected. Direct timecoded shots with visual anchors: Start each cut on a new line with a timestamp ([Shot 1] At 00:00.000..., [Shot 2] At 00:02.500...). Always repeat 2 to 3 key visual traits from your reference image (the young Japanese woman from (S1) in the crimson-and-black neo-samurai jacket with the glowing cyan hair clip) so the 32B text encoder locks onto the right visual tokens. Wrap spoken dialogue in tags: Assign a speaker ID ((S1)), put voice descriptions outside the tag, and keep only [Language] and the spoken words inside: Looking straight into the lens, the calm, fierce young woman (S1) says: [English] Stand down, guardian. We are on the same side tonight. Separate sound effects from music at the end: Close your prompt with overall_soundscape: for physical Foley and ambient noise (pouring rain, katana unsheathing, crackling plasma, beast roar) and non_diegetic_music: for the background score (or non_diegetic_music: N/A for no music). Use the 6-part schema for 3+ references or video/voice swaps: When combining character photos with a reference video () or voice clip (), structure your prompt into six blocks (subject_definitions:, summary:, retention_analysis:, detailed_description:, overall_soundscape:, and non_diegetic_music:) so you can explicitly set each asset to fully_preserved, attribute_transfer, reference (voice cloning), or fully_copy (exact lip-sync). Step 7: Generate the video Click the Run button to start generation. The first run loads roughly 37 GB of model weights from disk, so allow a minute or two before sampling starts. When complete, the finished MP4 with stereo audio plays inside the Save Video node (#92) on the right and is saved to ComfyUI > output > video. Top row: the two input reference images ( and ). Bottom row: the generated shots from the 5-second video clip. Settings SettingValue in the workflowEffectref_image_0 / ref_image_1ref_cyber_ronin.jpg / ref_mecha_kitsune.jpgMapped to and in the prompt.ref_image_sizematchDownscales reference images to match output resolution (max keeps up to 2048px short edge).Aspect ratio16:9 (Widescreen)Set on the Resolution Selector (Size) node.Megapixels0.4 (864x480)Safe up to 0.98 (1344x768, native 768p).Multiple32Locked to H3’s latent patch grid.Float (Duration)5 secondsSnaps up to 124 frames at 24fps ($17k + 5$ grid).Boolean (Enable Lightning LoRA)false (20 steps)Set to true to switch to the 4-step Turbo LoRA.Sampler / schedulerres_multistep / betabeta or normal locks identity better than simple on reference prompts.Denoise1.0Keep at 1.0 (references are injected as conditioning tokens).noise_seed757358688076805Change on RandomNoise (#129) for a new take. Tips Avoid the last-frame morph bug: Do not write or inside the detailed_description: section unless you want the shot to freeze or morph into a static copy of that photo at the end of the clip. Bind to inside subject_definitions: and retention_analysis:, and refer only to inside detailed_description:. Always include visual anchors in subject_definitions: Writing a bare line like is the woman in often causes the model to hallucinate a generic face. Add 3 to 4 distinctive visual traits (such as hair style, cyber-clip color, and jacket details) and repeat them the first time appears in detailed_description:. Fix the 10x reference-video slowdown: The ref_image_size: match setting only downscales reference images; it keeps reference videos (ref_video_0) at up to 1344x768 even when you are drafting at 0.4 MP. When doing video character replacement, insert an ImageScaleToTotalPixels node (megapixels: 0.4, resolution_steps: 32) between Get Video Components and ref_video_0 to cut render times by 5x to 8x. Know the numbering order: , , and each use their own 1-based counter in the order slots are connected. However, reference video soundtracks (ref_video_audio_0..2) are always numbered before standalone audio clips (ref_audio_0..2). If you connect both ref_video_audio_0 and ref_audio_0, the video’s soundtrack is and your standalone audio is . When to use ref_image_size: max: Keep ref_image_size on match while testing seeds and camera moves. Switch it to max (up to 2048px short edge) only for final renders when a reference image has small facial details, emblems, or readable text that look soft under match. Try the beta or normal scheduler: For prompts that combine 3 or more references, changing scheduler on BasicScheduler (#124) from simple to beta or normal at 20 to 25 steps improves facial identity lock across camera cuts. Draft faster with the 4-step Turbo LoRA: Flip Boolean (Enable Lightning LoRA) (#146) to true to engage the 4-step ref2v Turbo LoRA for quick layout checks. If faces drift at 4 steps, change Int (Lightning LoRA) (#144) from 4 to 6 or 8 steps and lower strength_model on LoraLoaderModelOnly (#145) to 0.8. Never select 1.0 Megapixels at 16:9: Native 768p at 16:9 is 1344x768 (0.98 MP). Setting megapixels to 1.0 outputs 1376x768, which exceeds H3’s training area cap and degrades anatomy. Generate at 0.98 MP and use an AI upscaler afterward if you need 1080p or 4K. What’s next MiniMax H3 text-to-video MiniMax H3 image-to-video Wan 2.2 image-to-video Wan 2.2 first and last frame video Hunyuan image-to-video LTX Video Video-to-video AI upscalers The post Reference-to-Video (ref2va) with Sound in ComfyUI (Workflow Included) – MiniMax H3 Reference-to-Video first appeared on Stable Diffusion Art.