How to Build Your Own Muse/Dot: Under the Hood of Three Open-Source Architectures
Why securing AI agents requires strict permissions, sandboxing, and safeguards that go beyond the model itself.OpenAI’s Dots introduces an appealing idea: give an agent an ongoing responsibility, connect it to the tools it needs, and let it continue working between conversations. That shifts the…
Annons
Annons
Why securing AI agents requires strict permissions, sandboxing, and safeguards that go beyond the model itself.OpenAI’s Dots introduces an appealing idea: give an agent an ongoing responsibility, connect it to the tools it needs, and let it continue working between conversations. That shifts the product experience from asking individual questions to delegating ongoing work.I have been seeing open-source versions of Dots popping up across my feed, and it got me curious. As a product builder, I wanted to look past the demos and understand how these agents actually work.I picked three projects that caught my attention:Composio-community’s Open DotCopilotKit’s OpenDotsAnil Matcha’s Open-DotsThis comparison comes from reading their documentation and source code. It is an architecture review rather than a hands-on benchmark or security audit. These projects are evolving quickly, so the distinctions describe the implementations inspected.An agent product has several separate parts:An interface where you give it instructions and review its work.A model that decides what to do.A runtime that executes actions and returns their results.Storage for conversations, preferences, and ongoing work.Permissions that control what the agent can access or change.Background infrastructure that keeps work moving when you leave.A familiar interface can hide very different choices underneath. To compare these projects, I followed how a task moves through each one.Composio-community’s Open Dot: A repeating agent-and-tool loopThis project takes a desktop-first approach. Its desktop application runs a local controller that coordinates conversations, saved context, tools, and recurring work.A message, schedule, or connected-app event starts a run. The application loads the relevant conversation, saved memories, and instructions, then asks the model what to do next.If the model needs a tool, the runtime checks the applicable rules and an allowed action can run immediately; Any action that needs approval pauses for the user. The result/denial is returned to the model so it can decide whether to continue or finish.Here is that workflow, step by step:Open Dot Architecture used by Composio-communityThe loop matters because a useful task often requires several decisions. For example, an agent summarizing repository issues may retrieve the issues, inspect what it receives, request more detail, and then prepare its answer. The runtime keeps that process moving and bounds how long it can continue.Composio supplies the connected-app layer. Browser automation and execution tools provide other ways to act when a task needs website interaction or code.Its memory mechanism is also concrete: saved memories, skills, rules, and routines become part of the model’s instructions. Remembering a preference means supplying it as context on a later run.CopilotKit’s OpenDots: an agent working inside a shared workspaceThis implementation emphasizes the application around the agent: conversations, pages, spaces, document editing, and review of proposed changes.A message or scheduled task starts a run. The agent receives conversation and workspace context, then asks the model what to do. If a tool is needed, its availability depends on the enabled capabilities. A proposed page change can pause for review; the user’s decision and tool results feed back into the conversation.Here is the workflow, including the page-review branch:Open Dots Architecture used by CopilotKitThe interface receives streamed events that describe what the agent is doing. That lets the user see text and tool activity as work happens. Underneath, the supplied agent uses a bounded tool loop built with TanStack AI.The workspace is the important product choice. The agent and user can work on persistent pages and documents instead of only exchanging answers in a chat window. Review makes a proposed change visible before it is accepted.Optional computer access runs through OpenBot, a separate service that manages persistent Docker computers for individual dots. Browser profiles and files can survive across interactions. This separates the application controller from the environment where computer actions happen.The storage boundary deserves attention: pages and application state use local SQLite, while conversations in the supplied template require CopilotKit Intelligence. Hosting the web application yourself does not automatically mean conversation history stays on your machine.My interpretation is that this project emphasizes a collaborative agent workspace: documents, review, persistent computer access, and an interface that shows the agent’s work.Anil Matcha’s Open-Dots: a command-driven action flowThis project provides a web interface backed by a Python API. Conversations and configuration are stored locally, and credentials have a separate encrypted storage mechanism.The chat flow first looks for an explicit supported command: a workspace operation, a supported connector command, or search. If it finds one, it validates the action, checks approval requirements, and executes it. The action result becomes context for the model’s response. Ordinary chat skips that action path.Here is the command-and-approval flow:Open Dots Architecture used by Anil MatchaThe action gateway is a useful architectural boundary. It centralizes registered actions, approval requirements, execution handling, and audit events. A builder has an explicit place to control what an operation can do.The important limitation is in the inspected chat implementation: it does not have a repeated model-driven tool-selection loop. The model does not currently choose a tool, inspect its result, choose another tool, and continue. The command is selected before the model generates its response.Computer operations also have separate backend endpoints; their presence does not automatically make them tools the chat model can select. I see this project as a Python foundation that could be extended with orchestration above its existing action gateway.What actually differs between the three?Composio-community: a broad desktop assistant with a custom repeating tool loop, connected apps, saved context, and recurring tasks.CopilotKit: a collaborative workspace with a bounded agent loop, document review, and optional persistent computers.Anil Matcha: a Python action gateway and command-driven chat flow that could be extended into a more autonomous agent.These are different architectural starting points. The right starting point depends on whether you need breadth of integrations, a rich workspace, or a backend I want to extend myself.Where does your data actually go?Running these projects yourself does not automatically keep your data on your machine. Each implementation has different external dependencies:Composio-community’s Open Dot: When the agent executes connected-app actions through hosted Composio, the action requests and results pass through Composio. Conversation context and tool results supplied to a hosted model also leave your machine.CopilotKit’s OpenDots: The supplied template uses CopilotKit Intelligence for conversation infrastructure, so conversation history is handled by that service. Local pages and settings do not make the entire application local-only. Model requests also go to the configured model provider.Anil Matcha’s Open-Dots: Conversation storage is local, but requests still go to the configured model endpoint. Enabling hosted Composio integrations, search, or remote computer services introduces additional external data paths.Connecting an account does not necessarily upload its entire contents. However, when an agent retrieves information to complete a task, that information can pass through the integration service and become part of the model’s context. The amount exposed depends on the task, granted permissions, and implementation.How I’m planning to build my own personal Open DotReading these implementations has given me a strong starting point for my own build. This weekend, I plan to begin developing my personal Dot, which I am calling Orbit, around a few tasks I would like to delegate. I’m planning four everyday workflows:GitHub briefings: summarize open issues and pull requests, flag items that need attention, and prepare a daily update.Inbox summaries and cleanup: summarize unread emails and conversations that need action, highlight important messages, and suggest newsletters or other low-value mail to archive or move to trash after my review.Calendar briefings: show what is coming up today, flag overlapping meetings, and surface relevant context for preparation.Follow-up tracking: identify unanswered messages and commitments in selected conversations, then prepare reminders or draft follow-ups for approval.These are the planned use cases. I’ll build and share them in stages so readers can see what is working and what is still on the roadmap.My architecture will use Python and LangGraph to coordinate the work. Each run will load the task’s instructions and saved context, ask the model what to do next, check any proposed action against its permissions, and execute the allowed tools. Tool results will return to the model so it can continue or finish. Actions that change something externally will pause for my approval.I’m drawing inspiration from all three projects:Composio-community: the repeating loop where the model chooses an action, receives its result, and decides what comes next.CopilotKit: making the agent’s work visible and reviewable, with a workspace for its outputs.Anil Matcha: a central action gateway that checks and controls tool execution.I’ll keep conversation state locally and use direct integrations with GitHub, my email provider, and my calendar provider where practical. Each workflow will receive only the access it needs. Inbox cleanup will show a proposed batch before anything is archived or moved to trash; follow-up messages will remain drafts until I approve sending them.If I use a hosted model, the selected conversation context, email content, calendar details, and tool results sent to it will still leave my machine. Local storage alone won’t make the entire system local-only. A good alternative is to use open-weight models through Ollama.I’ll build the shared workflow, local state, and permission checks, then start connecting these use cases. My first working targets are GitHub and inbox summaries, with calendar briefings and follow-up tracking on the same roadmap. I’ll share progress as each flow becomes usable, and add cleanup and sending actions after the review process works.I’ll build it in public and share the architecture, working code, and lessons along the way.This story is published under the Generative AI publication. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories. Let’s shape the future of AI together!How to Build Your Own Muse/Dot: Under the Hood of Three Open-Source Architectures was originally published in Generative AI on Medium, where people are continuing the conversation by highlighting and responding to this story.