ControlNet Video-to-Video with Sound in ComfyUI (Workflow Included) – MiniMax H3 Fun ControlNet Union
Text prompts cannot lock choreography. If you try to describe a spinning dance or specific hand movements with text alone, the model will improvise something completely different on every seed, and you will waste hours chasing a lucky roll. With the MiniMax H3 Fun ControlNet Union workflow in…
Annons
Annons
Text prompts cannot lock choreography. If you try to describe a spinning dance or specific hand movements with text alone, the model will improvise something completely different on every seed, and you will waste hours chasing a lucky roll. With the MiniMax H3 Fun ControlNet Union workflow in ComfyUI, a single ControlNet model patch lets you copy exact body motion from any driving video while generating 24fps video and synchronized stereo audio at the same time. Left: original driving video frame. Right: generated MiniMax H3 Fun ControlNet output frame. Driven by a live-action dance clip using the built-in SDPose whole-body pose extractor, with crackling fire, rustling silk robes, ocean waves, and an ancient Greek lyre score. In this tutorial, you will learn how to: Set up the MiniMax H3 Fun ControlNet Union workflow and built-in SDPose whole-body tracker in ComfyUI. Transfer character choreography from a reference video while generating stereo sound effects, dialogue, and music. Switch the same Union patch between Pose, Depth, Canny edges, and masked video inpainting. Combine ControlNet motion transfer with character identity lock. Software needed We will use ComfyUI, a free AI image and video generator. You can run it locally on Windows or Mac, or run it in the cloud on Google Colab if you want a quick setup. Think Diffusion provides an online ComfyUI service if you don’t have a capable local GPU. (20% extra credit for SDA readers.) More example outputs Can one driving video create totally different scenes? Because the SDPose skeleton only locks the body movement and leaves the background and audio open to the prompt, I tested the exact same meadow dance video with two very different prompts. Example 1: Futuristic Reykjavik skybridge Same driving dance clip, prompted as an Icelandic woman in an iridescent cybernetic suit on a neon-lit glass skybridge in futuristic Reykjavik with synthwave music and electric plasma sound effects. Example 2: Rainy Kyoto street illusionist Same driving dance clip, prompted as a cool, eccentric Japanese street illusionist in an indigo haori coat and round tinted glasses spinning through a lantern-lit Kyoto alley with shamisen and taiko beats. What is MiniMax H3 Fun ControlNet Union? MiniMax-H3-Fun-Controlnet-Union is an all-in-one structural control adapter trained by Alibaba PAI for the open-weights MiniMax H3 video-and-audio model. One file handles every mode. Instead of making you download separate multi-gigabyte checkpoints for pose, depth, and edge detection, this single lightweight model patch injects structural skip connections directly into the transformer blocks during sampling. I have tested several local video-to-video workflows in ComfyUI over the past few months, and this is currently my go-to setup for motion transfer. Why does it work so well? One Union checkpoint for all control modes: You can feed Pose skeletons, Depth maps, Canny edges, HED soft boundaries, or MLSD lines into the exact same node without changing a single mode dropdown, and the optional MiniMax-H3-Fun-Controlnet-Union-2.0 upgrade adds Scribble, Layout, and Grayscale control too. Built-in SDPose whole-body tracking: The workflow includes ComfyUI’s native SDPose Video Pose Extractor [Multi-Person] subgraph, which pairs an RT-DETRv4 person detector with a 133-keypoint pose model so you get clean hand, finger, face, and foot tracking without hunting for fragile third-party custom nodes. Mask-based video inpainting: Connecting a black-and-white mask and source_video to Apply MiniMax H3 Fun ControlNet widens the control stream to 49 channels, letting you repaint only the masked region of a clip while keeping the rest of the video untouched. Stereo audio plus character reference lock: Because the control patch sits on top of MiniMax H3 reference-to-video (ref2va) and zeroes out its control skips on audio tokens, your driving video locks the physical movement while the model generates clean stereo soundscapes, music, and lip-synced speech. MiniMax H3 Fun ControlNet Union Step-by-Step Step 0: Update ComfyUI Update ComfyUI before you start. Both Apply MiniMax H3 Fun ControlNet and the built-in SDPose subgraph require ComfyUI 0.35.0 or newer, and you can update in a few clicks using ComfyUI Manager. Click the Manager button on the top toolbar. Select Update ComfyUI. Restart ComfyUI. Step 1: Download the models Download the required model files and place them in their matching folders: MiniMax H3 base models (skip any you already have from earlier MiniMax H3 tutorials): Download minimax_h3_ref2va_pruned_int8_convrot.safetensors (19.5 GB) and put it in ComfyUI > models > diffusion_models. Download qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors (14.6 GB) and put it in ComfyUI > models > text_encoders. Download minimax_h3_video_vae_fp16.safetensors (4.85 GB) and put it in ComfyUI > models > vae (or reuse minimax_h3_video_vae_int8_convrot.safetensors from the MiniMax H3 text-to-video guide). Download minimax_h3_audio_vae_fp32.safetensors (577 MB) and put it in ComfyUI > models > vae. Download minimax_h3_ref2v_turbo_4step_v0.1_comfyui_bf16.safetensors (1.82 GB) and put it in ComfyUI > models > loras. New models for Fun ControlNet Union and SDPose whole-body tracking: Download minimax_h3_fun_controlnet_union_pruned_int8_convrot.safetensors (2.14 GB) and put it in ComfyUI > models > model_patches. Download sdpose_wholebody_fp16.safetensors (1.78 GB) and put it in ComfyUI > models > checkpoints. Download rt_detr_v4-x-hgnet_fp16.safetensors (118 MB) and put it in ComfyUI > models > diffusion_models. Critical note on model_patches and pruned weights: Do not mix unpruned and pruned weights. Always pair minimax_h3_ref2va_pruned_int8_convrot.safetensors with the _pruned_ Fun ControlNet patch from Comfy-Org/MiniMax-H3 above, because loading the unpruned 6.8 GB file from alibaba-pai against a pruned base model triggers an immediate adaln width 2688 does not match the base model's timestep embedding width 8 error. Optional 2.0 upgrade: Download minimax_h3_fun_controlnet_union_2.0_pruned_int8_convrot.safetensors (4.22 GB) into ComfyUI > models > model_patches and select it in Load Model Patch if you want 10-layer control injection (twice as dense as v1) and cleaner video inpainting. Step 2: Load the workflow Download the workflow JSON file below. Drag and drop the file into your ComfyUI window to load the full graph with the Greek priestess prompt pre-configured. Download Step 3: Install missing nodes This workflow uses built-in nodes. However, if you see any red node borders after loading the workflow, you may need to install missing nodes in ComfyUI Manager or update your ComfyUI build before running it. Click Manager > Install Missing Custom Nodes. Install any nodes that are listed as missing. Restart ComfyUI and refresh your browser. Step 4: Upload your control video Upload your driving video in the User Inputs group at the top left using the Load Video (Control Video) node: Download the sample driving video (1344×768, 24fps, 5.17 seconds) and upload it to Load Video (Control Video) if you want to test the exact same driving clip I used. Leave start_time and duration at 0 on Video Slice to use the full video, or enter a custom start time and duration in seconds to trim a longer clip. SDPose Video Pose Extractor [Multi-Person] detects up to 2 people per frame and outputs a 133-keypoint skeleton sequence into Preview Image, which connects directly to control_video on Apply MiniMax H3 Fun ControlNet. Left: frame from the input driving video. Right: whole-body 133-keypoint pose map extracted by the built-in SDPose node. Step 5: Set the resolution and duration In the Video Settings group on the left: On Resolution Selector (Size), set aspect_ratio to match your driving video (16:9 (Widescreen)). Set megapixels to 0.4 (864x480) for fast test runs, or 0.98 (1344x768) for full 768p quality. Leave multiple at 32. Set Float (Duration) to 5 seconds (124 frames at 24fps). Leave Boolean (Enable Lightning LoRA) set to false (20 steps) for maximum quality, or flip it to true (4 steps) for fast previews. Why I deleted the second Boolean (Enable Lightning LoRA) node: The stock ComfyUI template includes two Boolean (Enable Lightning LoRA) nodes in the Video Settings group, and the bottom one has no output wires connected at all. That bottom switch does nothing! I removed the dead duplicate in our workflow file so the single switch cleanly controls both the LoRA loader and the step count. Step 6: Revise the prompt The prompt still matters a lot. While the Fun ControlNet locks the physical pose on every frame, the 32B text encoder decides the character’s face, outfit, environment, lighting, and stereo audio track, and the full Greek priestess prompt is already loaded inside Input Text (Prompt). Follow these rules when writing your own prompt: Start with a Scene: look and character description: Describe the camera style, lighting, background setting, and your character’s outfit and facial features (Cinematic mythic fantasy film look... An attractive young Greek priestess with braided dark curls, a golden laurel crown, and a flowing white-and-gold silk chiton...). Match timecoded beats ([0s-2s], [2s-4s], [4s-5s]) to the driving video: Watch your control clip and describe the visual action during each phase of the movement, such as arching back at [0s-2s], spinning with wide arm sweeps and glowing golden fire at [2s-4s], and stopping in a poised stance at [4s-5s]. Write overall_soundscape: and non_diegetic_music: at the bottom: Audio tokens stay completely untouched. Because the ControlNet patch zeroes out its skip projections on the audio branch, you must list physical Foley and ambience in overall_soundscape: (rustling silk robes, crackling fire, ocean waves) and musical score in non_diegetic_music: (ancient Greek lyre, strings, and female choir). Step 7: Generate the video Click the Run button to run the workflow. First runs take a minute. ComfyUI runs the RT-DETRv4 person detector and SDPose keypoint extractor across all 124 frames first, then loads the MiniMax H3 transformer and Fun ControlNet patch to sample the video and audio together, saving the finished MP4 to ComfyUI > output > video. Settings SettingEffectModelPatchLoaderLoads the 5-block Fun ControlNet-Union patch (or select the 10-block 2.0 patch).Apply Fun ControlNetControls strength, start_percent, and end_percent across sampling steps.Load VideoSelects the source driving video clip.SDPose SubgraphExtracts 133-keypoint body, hand, face, and foot skeletons per frame.Aspect ratio / MegapixelsSets output size (0.4 MP for 480p drafts, 0.98 MP for native 768p).Float (Duration)Sets video length in seconds and snaps up to H3’s $17k + 5$ frame grid at 24fps.Boolean (Enable Lightning LoRA)Switches between false (20 steps) and true (4-step ref2v Turbo LoRA).Sampler / schedulerUses res_multistep / simple (CFG = 1.0 via BasicGuider). Tips Combine Pose ControlNet with character identity lock: Can you lock a specific character’s face while copying a dance video? Yes, because this workflow uses MiniMax H3 Reference to Video as its conditioning node, so you can simply drop in a Load Image node, wire it to ref_image_0, and write The woman from in your prompt to keep her exact face and outfit! Switch to Depth, Canny edges, or Video Inpainting without changing checkpoints: There is no mode dropdown on Apply MiniMax H3 Fun ControlNet because the Union checkpoint detects the control type automatically from whatever image batch you plug into control_video: Canny or Depth: Bypass the SDPose node, wire Get Video Components into ComfyUI’s built-in Canny or Depth Anything V3 node, and connect its image output to control_video. Video Inpainting: Connect your original video frames to source_video and a black-and-white mask batch (1 = repaint, 0 = keep) to mask on Apply MiniMax H3 Fun ControlNet, leaving control_video empty or keeping SDPose connected to guide motion inside the masked region. Avoid frozen frames at the end of shorter clips: If your driving video is shorter than Float (Duration), MiniMax H3 repeats the final frame of the control video for the rest of the timeline, which makes your character freeze in place like a statue. Select the bypassed Select and Ctrl-B to enable group (Get Video Components + Get Image Size), press Ctrl + B, and wire batch_size into length on MiniMax H3 Reference to Video to match your clip’s exact frame count automatically. Lower strength and end_percent when body proportions differ: Leaving strength and end_percent at 1.0 forces your character to match the exact shoulder width, head size, and limb lengths of the actor in the driving video. When transferring human movement onto taller characters, heavy armor, or stylized proportions, I get much better anatomy by lowering strength to 0.80, lowering end_percent to 0.80, and setting draw_face to false inside the SDPose subgraph. Set max_detections to 1 for solo performers: The SDPose subgraph defaults to max_detections: 2. In outdoor clips with trees, statues, or reflections, I noticed RT-DETRv4 occasionally picks up a false second person in the background and draws a stray skeleton, so set max_detections to 1 whenever your video has a single dancer. Lip-sync or clone the driving video’s original audio: By default, the workflow ignores the driving video’s audio track and generates brand-new stereo sound from your text prompt. If you want your character to lip-sync to speech or music from the driving video, wire the audio output of SDPose into ref_audio_0 () on MiniMax H3 Reference to Video. Never select 1.0 Megapixels at 16:9: Native 768p widescreen is 1344x768 (0.98 MP), and setting megapixels to 1.0 outputs 1376x768, which crosses the model’s pixel limit and causes deformed limbs. Render at 0.4 MP or 0.98 MP and run the video through an AI upscaler afterward if you need 1080p or 4K. What’s next MiniMax H3 multiframe reference MiniMax H3 reference-to-video MiniMax H3 image-to-video MiniMax H3 text-to-video ControlNet complete guide Video-to-video AI upscalers The post ControlNet Video-to-Video with Sound in ComfyUI (Workflow Included) – MiniMax H3 Fun ControlNet Union first appeared on Stable Diffusion Art.