Two Types of AI Agents Everyone needs to understand
Understanding how AI agents interact with tools, software, and your computer.Not every AI agent can perform the same tasks.At first glance, AI agents seem quite similar. However, in the background, they all work in different ways. And this is exactly where it is decided what the AI agent can and…
Understanding how AI agents interact with tools, software, and your computer.Not every AI agent can perform the same tasks.At first glance, AI agents seem quite similar. However, in the background, they all work in different ways. And this is exactly where it is decided what the AI agent can and cannot do.To use AI agents effectively, we should therefore know two fundamental types:Textual AI AgentsVisual AI AgentsA textual AI agent interacts with other programs through machine-readable interfaces or by writing its own program code.A visual AI agent, on the other hand, uses programs similarly to a human. It looks at the screen, recognises buttons and input fields, and clicks through the application by moving the mouse.In this blog post, we will take a closer look at both types of AI agents, how they work using real examples, as well as where their respective strengths and weaknesses lie.What actually is an AI agent?But let’s first begin with the question of what an AI agent actually is. Put simply, you can think of an AI agent as an evolution of a classic AI chatbot.A chatbot primarily answers questions or creates content. An AI agent can additionally work on tasks independently. For this purpose, the AI agent has access to various tools and information sources to solve the given task for us.An AI agent receives a goal, analyses the task, plans the necessary steps, and then tries to achieve this goal as independently as possible.An example would be:An AI agent could, for example, search for train connections, compare hotels, select suitable activities, and create a travel plan from them.The technological foundation of many of today’s chatbots and AI agents is so-called Large Language Models, or LLMs for short. These models can process natural language and generate text themselves.In simple terms, an LLM works like this:It processes the previous text, which is divided into smaller pieces of text called tokens, and calculates the most likely next token.The model repeats this process again and again and, in this way, generates a complete answer step by step. The model learned during its training which token is most likely to come next from very large amounts of data.One dataset on which a large number of models are based is Common Crawl. This dataset is 10 petabytes in size, and just one crawl contains more than 2.1 billion websites.Just to get a feeling for how large the amounts of data are on which these models were trained.What is a text-based agent?The AI agents we probably encounter most frequently at the moment are text-based. These AI agents work exclusively by generating text and using tools that can be called textually.For this reason, these agents mainly work for writing text, code, and terminal commands.#One of the first scientific papers to present this idea was Toolformer back in 2023.In their paper, the authors describe how they taught a Large Language Model to access external programs through machine interfaces and call them independently, such as a calculator, a search engine, a translation system, or a calendar.It becomes more difficult with applications that can only be operated through a graphical user interface.Any application that does not offer a machine-readable interface cannot be operated directly by these agents, such as Instagram, LinkedIn, or other social media sites.These sites and many other sites offer only limited machine-readable interfaces. The same applies to old enterprise software. Such applications cannot be automated with classic text-based AI agents.This once again highlights how important machine-readable interfaces will be in the future for websites and programs if you want to automate processes using AI.An example of such a text-based AI agent is the basic form of ChatGPT, which has changed from a chatbot into an AI agent. Here, we can quite easily add new tools to the AI agent, with which it can, for example, read emails or edit our Notion database.Here, I have connected a Notion database containing all of my recipes to my ChatGPT AI agent.This connection was implemented through an MCP server offered by Notion. An MCP server (Model Context Protocol) is a standardised interface through which AI agents can access external tools, data, and services and work with them.An MCP can be regarded as a universal adapter for AI agents, without which AI providers would have to develop every tool themselves. Through such an MCP server, the AI agent now has the tools to read and edit individual pages in Notion.The tools with which changes can be made are defined here.Add a recipe to the databaseRead a recipe from the databaseIf you want to know in more detail how this works, then take a look at this blog post in general.What Do We Eat Today? How I Turned Claude into My Personal Kitchen ChefImage-based AI agentsNow, not all applications offer MCP servers or any other machine-readable interface, as in this Notion example. And this is exactly where the biggest weakness of these text-based AI agents lies.Many programs were originally developed only for humans and are not prepared to be automated by AI.We open them and see: buttons, menus, images, tables, input fields, and other elements. We intuitively know that we have to click here. A classic text-based AI agent does not see this user interface in the same way.So if there is no machine-readable access to the program, it often cannot continue working. This usually includes older enterprise programs, applications, and data that want to protect their information.These applications provide only rudimentary machine-readable interfaces for AI agents and, therefore, cannot be automated by them either. The authors of the paper “From Pixels to UI Actions” recognised this problem early on at the end of 2023.Instead of predefined tools for controlling different programs, such as email, Notion and Office, image-based AI agents use only a handful of tools for moving the mouse pointer and entering input through the keyboard.These image-based AI agents, therefore, use our computers much more as we humans do. What these scientists began in 2023, OpenAI has taken to a new level with its new model GPT-6-Astra in order to solve real problems and tasks.For this, they proceed as follows. They take a screenshot of the current screen. They analyse what they see based on the task we give them and then move their mouse pointer to the correct position.Then they take another screenshot, and the three steps begin again from the start.An outlook on what is possible in this way is shown by OpenAI’s official website, ranging from designing circuit boards, filling out forms, creating Excel spreadsheets, and creating entire games in Unity.With this, the AI agent is no longer limited to the terminal window and machine-readable interfaces, but can move freely across the entire computer and in all programs.However, alongside great potential, these freedoms for the AI agent also come with major dangers.If we do not monitor what the agent is doing, it can access any folders on our computer, and all screenshots that are taken will very likely be used for training the models.So we need to think very carefully about how we use this feature. One idea from my side would be a second AI user on the computer on which no personal data is stored.This is cumbersome, however, but it offers the highest level of privacy without having to give up this feature. But feel free to write your opinion about this in the comments.In a first experiment, I tried out GPT-6 Computer use capabilities. I wanted to add a Friends link on top of all my Medium articles in the first line of each blog post.Having written around 100 articles in the last few years, this is a lot of work, so I thought GPT-6 could do this manual work for me.Add the following sentence below the title image of each of my Medium blog posts. The word here should contain a Friends Link.Not a Medium member? Keep reading for free by clicking here.And what should I say? It worked excellently…Okay, but only for 12 blog posts. After that, my usage limit for the entire week on the standard paid plan was exceeded, highlighting the biggest drawback of computer use.It has high resource requirements due to the vision LLM, which can burn through your token limit within minutes.Let’s see if we will see at least some improvements here in the next months and years to automate more manual work in our lives.Direct comparisonNow that we have seen how text-based and image-based AI agents work in the background, let’s compare both types directly to each other in a table.This table highlights that there is no superior type of AI agent. Both types of AI agents have their advantages and disadvantages. It is up to us to select the right agent for the task.Do we want to optimise for cost and speed? Choose the text-based agent.Do we want to optimise for flexibility and freedom? The image agent is the best choice.What experience have you already had with using GPT-6 image-based computer use feature?SourcesCommon Crawl https://commoncrawl.org/Toolformer: Language Models Can Teach Themselves to Use Tools https://arxiv.org/abs/2302.04761From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces https://arxiv.org/abs/2306.00245?utm_source=chatgpt.comThis story is published under the Generative AI publication. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories. Let’s shape the future of AI together!Two Types of AI Agents Everyone needs to understand was originally published in Generative AI on Medium, where people are continuing the conversation by highlighting and responding to this story.Source: Generative AI Pub — Published — Category: Image AI