Kandinsky 6.0 in ComfyUI: Video with Synchronized Audio & Lip-Sync (Workflows Included)
Generating AI video used to mean accepting dead silence. You had to produce a silent clip first, guess the timing in a separate foley tool, and stitch everything together in a video editor. The new Kandinsky 6.0 Video changes that dynamic completely. It is an open-weights foundation model that…
Annons
Annons
Generating AI video used to mean accepting dead silence. You had to produce a silent clip first, guess the timing in a separate foley tool, and stitch everything together in a video editor. The new Kandinsky 6.0 Video changes that dynamic completely. It is an open-weights foundation model that generates 5-second video clips alongside a synchronized 44.1 kHz audio track in a single pass. If your character speaks, their lips move in sync with the exact words in your prompt. Unmute for sound: The woman walks through an enchanted forest and clearly speaks dialogue with synchronized 44.1 kHz ambient sound. In this tutorial, you will learn how to: Set up and run Kandinsky 6.0 locally in ComfyUI. Generate synchronized soundscapes directly from text prompts (text-to-video). Animate a reference portrait into a talking video with realistic lip movement. Use speech tags like ... to control exact character dialogue and lip-sync. Run the lightweight 3B Lite Distill model in 10 steps on consumer graphics cards. Software needed I will use ComfyUI in this tutorial. It is a free, modular AI generator that you can run on your computer on Windows or Mac, or online through Google Colab. Think Diffusion provides a cloud-hosted ComfyUI environment if your GPU lacks the VRAM to load video diffusion models locally. (SDA readers get 20% extra credit.) Output examples Example 1: Character Dialogue & Enchanted Soundscape Music + dialogue. Unmute for sound. Prompt: Video Caption: A beautiful woman with long flowing hair in an elegant white dress walks along a mossy path in an enchanted golden forest. Glowing mushrooms line the path, soft mist rises from the ground, warm cinematic sunbeams filtering through the autumn canopy. She gracefully spins around, looks toward the camera, and clearly says Welcome to the enchanted forest.. Audio Caption: Loudly, a clear, warm female voice speaks in English. Ethereal mystical forest ambiance, gentle breeze rustling through leaves, delicate crystal chimes. Example 2: Character Dialogue with Lip-Sync Put your dialogue inside ... tags to make the character talk. Unmute for sound. Prompt: Video Caption: A male news anchor sitting at a broadcast news desk speaking into the camera says We interrupt this program with breaking news.. Through the studio window behind him, giant flying dragons and futuristic aircraft fire laser blasts, causing massive cinematic explosions. Audio Caption: Clear male broadcast voice in English, dramatic emergency news sirens, futuristic laser blasts, loud rumbling explosions. Example 3: Text-to-Video Prehistoric Soundscape Environmental sound matching motion! Unmute for sound. Prompt: Video Caption: A massive Tyrannosaurus Rex roars in a lush prehistoric rainforest near a cascading waterfall. Mist rises from the river, ancient fern trees sway in the wind, cinematic lighting. Audio Caption: Deep guttural dinosaur roar, heavy thudding footsteps on wet earth, rushing waterfall ambiance, distant jungle birds and insects. What is Kandinsky 6.0 Video? Kandinsky 6.0 (released by Kandinsky Lab in October 2026) is a multimodal foundation diffusion models designed to jointly generate video and audio. Unlike most video/audio models, Kandinsky 6.0 relies on a CrossDiT architecture that process visual frames and audio streams in separate streams. Cross-attention layers constantly share information between the two streams during sampling. Key highlights: True audio-visual synchronization: Audio is not generated after the fact. Visual actions and sonic events diffuse together step by step. Accurate speech lip-sync: Wrapping text inside ... tags forces the visual mouth geometry to align with the acoustic waveform that the vocoder produces. Lite (3B) and Pro (29B) models: The flagship 29B Pro model offers film-grade quality. Folks like us can use the 3B Lite model to run on consumer hardware with 12 GB to 24 GB of VRAM. 10-step PiFlow distillation: Distilled checkpoints further speed up the generation. Kandinsky 6.0 Step-by-Step Let’s walk through the video generation using the fast 10-step Lite Distill pipeline. Step 0: Update ComfyUI Before loading the workflow, make sure that your ComfyUI installation is up to date: Click the Manager button on the main toolbar. Click Update ComfyUI. Restart ComfyUI completely. Step 1: Install the custom nodes Kandinsky 6.0 requires dedicated nodes to handle its dual-stream latents and audio decoding: Open Manager > Custom Nodes Manager. Search for Kandinsky T8 (or kandinsky6) and click Install. Restart ComfyUI. (Note: The kandinsky-t8 extension provides single-file INT8 ConvRot support and automatically reads extra model paths from secondary drives). Step 2: Download the models Download the required model files and place them in their respective directories under ComfyUI > models: Diffusion Model (diffusion_models): Download k6_lite_distill_int8_convrot.safetensors (3.77 GB) and save it in ComfyUI > models > diffusion_models. (If you prefer unquantized weights, download Kandinsky6_Lite_distill_5s_bf16.safetensors (6.38 GB) instead). Text Encoders (text_encoders): Download qwen_2.5_vl_7b.safetensors (16.58 GB) or the lower-VRAM qwen_2.5_vl_7b_fp8_scaled.safetensors (9.38 GB). Download clip_l.safetensors (246 MB). Video VAE (vae): Download hunyuan_video_vae_bf16.safetensors (493 MB) and place it in ComfyUI > models > vae. Audio VAE & Vocoder (audio_vae): Download v1-44.pth (1.22 GB) and place it directly inside ComfyUI > models > audio_vae. Create a subfolder named bigvgan_vocoder under audio_vae. Download bigvgan_generator.pt (490 MB) and config.json (1.5 KB), placing both files inside ComfyUI > models > audio_vae > bigvgan_vocoder. Step 3: Load the workflow Download the workflow JSON file below that matches your generation goal, then drag and drop it onto the ComfyUI canvas: Download text-to-video workflow Download image-to-video workflow If any node displays red borders, open Manager > Install Missing Custom Nodes, install the listed pack, and restart ComfyUI. Step 4: Configure the input and models Check that your loader nodes point to the files that you downloaded: In UNETLoader, select k6_lite_distill_int8_convrot.safetensors with weight_dtype: default. In DualCLIPLoader, select qwen_2.5_vl_7b.safetensors for clip_name1, clip_l.safetensors for clip_name2, and set type to kandinsky5. In VAELoader, select hunyuan_video_vae_bf16.safetensors. In Kandinsky 6 Audio VAE Loader, ensure that tod_vae is set to v1-44.pth, bigvgan_dir is bigvgan_vocoder, and scaling_factor is set to 0.417. (For Image-to-Video): Load your starting portrait into the Load Image node. The included ImageScale node automatically scales the picture to match the generation canvas dimensions (864×480). Reference Input Portrait Step 5: Write your video and audio prompts Inside the Kandinsky 6 Text Encode node, you will find two distinct text boxes with separate video and audio captions. video_caption: Describe the visual scene, subject appearance, camera movement, and chronological action over 5 seconds. Dialogue in video_caption: If your character speaks, write their exact words wrapped in and at the moment they speak. For example: “A beautiful woman with long flowing hair in an elegant white dress walks along a mossy path in an enchanted golden forest. Glowing mushrooms line the path, soft mist rises from the ground, warm cinematic sunbeams filtering through the autumn canopy. She gracefully spins around, looks toward the camera, and clearly says Welcome to the enchanted forest..” audio_caption: Describe the voice tone and ambient environment, but do not repeat the spoken words here. For example: “Loudly, a clear, warm female voice speaks in English. Ethereal mystical forest ambiance, gentle breeze rustling through leaves, delicate crystal chimes.” Step 6: Generate the video Click Queue Prompt (or press Ctrl + Enter). The Kandinsky 6 Sampler node will run 10 steps using the PiFlow policy rollout. In the Image-to-Video workflow, the Kandinsky 6 Remove I2VA Reference node strips the reference frame, VAEDecode decodes the 121 video frames, and Kandinsky 6 Audio VAE Decode builds the synchronized 44.1 kHz audio track. The final Save Video node outputs an MP4 file containing both the visual animation and the synchronized audio track in your ComfyUI output folder. Settings SettingValue (Lite Distill)Notes & EffectsSteps10Distilled PiFlow models are calibrated for exactly 10 steps. Higher step counts do not improve fidelity.CFG1.0Mandatory requirement: PiFlow distilled models will raise an error if CFG is not 1.0.Denoise1.0Must remain at 1.0 for the full trajectory rollout.Audio Scaling Factor0.417Required for distilled checkpoints. (Non-distilled 50-step base models use 0.5302).Resolution864 × 480 (or 480 × 864)Width and height must be divisible by 16. Total pixel area should stay near 414,720 pixels for stable coherence.Length (Frames)121Must follow 4*n + 1 (e.g. 49, 81, 97, 121). 121 frames at 24 fps yields roughly 5.04 seconds.FPS24.0Both the empty latent node and the Create Video node must match at 24 fps to avoid audio drift. Tips Keep audio words out of the video prompt: Never include acoustic descriptions like hum, footsteps, ringing, or guitar music in the main video_caption box. Words that describe sound belong strictly inside audio_caption. The node automatically formats them into the required token block. Prevent phantom mouth movement: If a person is present on screen but should stay silent, add a short sentence in video_caption stating that they remain silent with their lips closed (for example: “He remains silent with lips closed.”). Omit human voices from audio_caption. This simple trick stops the model from hallucinating random mouth twitches. Skip the 19.3 GB Beautifier LLM: The official templates include a Kandinsky6BeautifyPrompt node that loads Qwen3.5-9B to expand short prompts. If your VRAM is limited, you do not need it! Writing detailed descriptions directly into Kandinsky 6 Text Encode saves nearly 20 GB of disk and memory swapping. Never bypass Kandinsky6RemoveReferenceLatent on I2VA: In Image-to-Video mode, the model attaches your starting image as a reference frame at the very tail of the video latent tensor. If you skip the removal node before decoding, the final few frames of your clip will jump back to the frozen starting photograph. Scale up to Full HD with the VSR pass: Once you are happy with your 480p preview, you can chain the output into Kandinsky6 SR Upscale with the 2.25x scale selected. This raises the output to crisp 1080p (1920×1080) while preserving your synchronized audio track. What’s next Wan 2.2 Text to Video Wan 2.2 Image to Video MiniMax H3 Image to Video MiniMax H3 Reference to Video MiniMax H3 Fun ControlNet Union ControlNet in ComfyUI AI Upscalers Guide The post Kandinsky 6.0 in ComfyUI: Video with Synchronized Audio & Lip-Sync (Workflows Included) first appeared on Stable Diffusion Art.