Agentic conversational video intelligence built on AWS
With video intelligence powered by agentic AI, you can ask natural language questions about uploaded videos and get answers within seconds. Organizations across media, security, insurance, and professional services are generating more video than their teams can review. Meeting recordings accumulate…
With video intelligence powered by agentic AI, you can ask natural language questions about uploaded videos and get answers within seconds. Organizations across media, security, insurance, and professional services are generating more video than their teams can review. Meeting recordings accumulate in shared drives, and security cameras capture weeks of unreviewed footage. Field inspection videos sit in object storage long after the initial review. The information inside these videos is often valuable: a design decision discussed three weeks ago, the exact moment a person arrived at a door, or the sequence of events leading to a vehicle collision. But accessing it has traditionally required watching hours of content manually. The alternative, building custom machine learning (ML) pipelines for each specific question type, demands significant development effort. Each new use case meant new development work: A transcription pipeline for meeting queries. A computer vision pipeline for visual search. A face-matching integration. In this post, we walk through the architecture and key patterns for building a video intelligence solution that accepts natural language questions and returns answers from video content. The solution uses an agentic architecture that decides at runtime which AWS services to invoke. For previously analyzed content, responses return in under a second. Initial analysis of new videos takes 5–10 minutes depending on length and services required. The complete implementation is available in the companion GitHub repository. Rather than pre-building a fixed pipeline for each question type, we use the Strands Agents SDK to create a single AI agent that orchestrates Amazon Bedrock, Amazon Rekognition, and Amazon Transcribe based on what the user asks. A major media and entertainment company adopted this approach during an AWS Professional Services engagement. With this solution, their consultants can query recorded discovery session content, extracting design decisions, action items, and stakeholder positions. The result: a reduction in manual review time of approximately 80 percent across a backlog of more than 200 multi-hour recordings, based on the customer’s internal before-and-after comparison of analyst hours per recording (not independently verified). Solution overview The solution is an AI agent that accepts video files and makes their content instantly queryable through natural conversation. A user can upload a 90-minute meeting recording and ask “What decisions were made in this meeting?” or “Did anyone mention the budget timeline?” The agent determines whether to invoke transcription, visual analysis, or both, then synthesizes the results into a coherent answer. The same system handles security footage queries (“Did this person appear?”), content analysis (“Summarize the first 30 minutes”), and investigative questions (“Which vehicle changed lanes before the collision?”). No separate processing pipelines are required for each use case. The following screenshot shows the interface that provides a chat panel for natural language queries and a sidebar for file uploads and analysis mode selection. Figure 1: The video intelligence chat interface The key insight is that the pipeline is determined at runtime. The agent calls Amazon Transcribe for spoken-content questions, turns to Amazon Rekognition for face matching, and reuses cached results for follow-up questions about previously processed content. The model handles the routing, not application code. Prerequisites To follow along with the implementation in this post, you need: An AWS account with access to Amazon Bedrock (Anthropic Claude Sonnet enabled) and Amazon Simple Storage Service (Amazon S3). For document processing, either Amazon Bedrock Data Automation (BDA) or Amazon Rekognition and Amazon Transcribe is required. See Supported models by AWS Region in Amazon Bedrock. Python 3.11 or later with the Strands Agents SDK installed (pip install strands-agents strands-agents-tools). AWS Command Line Interface (AWS CLI) configured with AWS Identity and Access Management (IAM) permissions for the services listed earlier. Basic familiarity with AI agent concepts such as tool use and reasoning loops. Architecture The system consists of an agent orchestrator connected to multiple AWS AI services, with Amazon S3 providing storage for uploaded videos and cached analysis outputs. The agent orchestrator is the reasoning engine. It’s built with the Strands Agents SDK and powered by Amazon Bedrock, using Claude Sonnet or another large language model (LLM) that supports tool use. It receives natural language queries from users and determines which tools to invoke based on the question, sequences multiple service calls when needed, and synthesizes the results into conversational responses. The agent maintains conversation history, so follow-up questions build on prior analysis without reprocessing. Figure 2: Solution architecture Amazon Rekognition provides visual analysis, including detecting objects, scenes, activities, and faces in video frames. The agent invokes Amazon Rekognition when the user’s question concerns something visible in the video. Amazon Transcribe converts spoken audio to text with automatic language detection across more than 100 languages (see Amazon Transcribe supported languages) and speaker diarization. The agent uses Transcribe when the question relates to spoken content. Amazon Bedrock Data Automation (BDA) offers an alternative analysis path that combines video summary, chapter detection, and full transcription in a single API call. This is useful when the user wants comprehensive analysis in one step, or when Amazon Rekognition or Transcribe aren’t available. All uploaded videos and analysis outputs are stored in Amazon S3 with per-user prefixes for multi-tenant isolation. These three services are the starting set, not a fixed one. Because the agent selects tools from their descriptions rather than from hard-coded workflow logic, the same architecture accepts additional services as tools. We return to this point in Extending beyond video. For production deployments, we recommend adding Amazon Bedrock Guardrails to enforce content filtering and grounding checks on agent responses, particularly for face-matching and surveillance use cases where responsible-AI controls are essential. How agentic orchestration works In a conventional video analysis application, the developer defines a fixed processing pipeline: upload the video, run transcription, perform visual analysis, present results. This approach processes every video through the same steps regardless of the specific query, and users wait for the full pipeline to complete before asking questions. The agentic approach inverts this model. With minimal pre-processing limited to uploading video files to an S3 bucket, the agent reasons about each question independently and calls only the services needed to answer it. When a user submits a query, the agent first parses the intent: the user wants a transcript summary, a visual search, or a face match? Then it checks whether relevant analysis has already been performed and cached. If not, it selects the appropriate tools, executes them (potentially in sequence when one tool’s output feeds another), and combines the results into a natural language answer. In our testing with 60-minute videos, the first question about a video typically takes 5–10 minutes (while transcription or visual analysis runs). Subsequent questions about the same content return in under a second because the agent reuses cached results. Actual times vary based on video length, resolution, and the AWS services invoked. Configuring the agent The following code shows the complete agent setup. We define the model provider, a system prompt that guides the agent’s reasoning behavior, and the set of available tools. With Strands, the entire orchestration logic (deciding which tools to call, in what order, and how to combine their outputs) is handled by the LLM rather than application code. We show two representative tool implementations (search_faces_in_video and analyze_with_bda). The remaining tools, including transcribe_video and analyze_video_visuals, follow the same pattern and are available in the GitHub repository. from strands import Agent from strands.models.bedrock import BedrockModel from tools import ( transcribe_video, analyze_video_visuals, search_faces_in_video, analyze_reference_image, analyze_with_bda, upload_video ) model = BedrockModel( model_id="us.anthropic.claude-sonnet-4-5-20250929-v1:0", max_tokens=4096 ) SYSTEM_PROMPT = """ You are a video intelligence assistant. For each user query: 1. Determine whether it requires spoken content analysis, visual content analysis, or both 2. Check if prior analysis results are already cached 3. Invoke the appropriate tools 4. Synthesize results into a clear answer with timestamps """ The production system prompt spans approximately 250 source lines. The following abbreviated example illustrates three representative policies (cache reuse, service fallback, and multi-modal orchestration) rather than reproducing the prompt verbatim: # --- Cache management (excerpt) --- CACHE_GUIDANCE = """ Before invoking any analysis tool, check the cache: - Call get_cached_result(video_id, analysis_type) first - If cached results exist and are < 24 hours old, use them - If the user says "re-analyze" or "fresh analysis", bypass cache - After any new analysis, store results with cache_result() # --- Tool fallback behavior --- If a tool call fails or returns low-confidence results: - Transcribe failure: suggest BDA as fallback - Rekognition low confidence (Source: AWS Machine Learning — Published — Category: Tools