We now support VLMs in smolagents!
Back to Articles We just gave sight to smolagents Published January 24, 2025 Update on GitHub Upvote 113 +107 Aymeric Roucher m-ric Follow merve merve Follow Albert Villanova del Moral albertvillanova Follow TL;DR Table of Contents Overview How we gave sight to smolagents How to create a Web...
Back to Articles We just gave sight to smolagents Published January 24, 2025 Update on GitHub Upvote 113 +107 Aymeric Roucher m-ric Follow merve merve Follow Albert Villanova del Moral albertvillanova Follow TL;DR Table of Contents Overview How we gave sight to smolagents How to create a Web browsing agent with vision Running the agent Next Steps You hypocrite, first take the log out of your own eye, and then you will see clearly to take the speck out of your brother's eye. Matthew 7, 3-5 TL;DR We have added vision support to smolagents, which unlocks the use of vision language models in agentic pipelines natively. Table of Contents Overview How we gave sight to smolagents How to create a Web browsing agent with vision Next Steps Overview In the agentic world, many capabilities are hidden behind a vision wall. A common example is web browsing: web pages feature rich visual content that you never fully recover by simply extracting their text, be it the relative position of objects, messages transmitted through color, specific icons… In this case, vision is a real superpower for agents. So we just added this capability to our smolagents! Teaser of what this gives: an agentic browser that navigates the web in complete autonomy! Here's an example of what it looks like: How we gave sight to smolagents 🤔 How do we want to pass images to agents? Passing an image can be done in two ways: You can have images directly available to the agent at start. This is often the case for Document AI. Sometimes, images need to be added dynamically. A good example is when a web browser just performed an action, and needs to see the impact on its viewports. 1. Pass images once at agent start For the case where we want to pass images at once, we added the possibility to pass a list of images to the agent in the run method: agent.run("Describe these images:", images=[image_1, image_2]) . These image inputs are then stored in the task_images attribute of TaskStep along with the prompt of the task that you'd like to accomplish. When running the agent, they will be passed to the model. This comes in handy with cases like taking actions based on long PDFs that include visual elements. 2. Pass images at each step ⇒ use a callback How to dynamically add images into the agent’s memory? To find out, we first need to understand how our agents work. All agents in smolagents are based on the singular MultiStepAgent class, which is an abstraction of the ReAct framework. On a basic level, this class performs actions on a cycle of following steps, where existing variables and knowledge are incorporated into the agent logs as follows: Initialization: the system prompt is stored in a SystemPromptStep, and the user query is logged into a TaskStep. ReAct Loop (While): Use agent.write_inner_memory_from_logs() to write the agent logs into a list of LLM-readable chat messages. Send these messages to a Model object to get its completion. Parse the completion to get the action (a JSON blob for ToolCallingAgent, a code snippet for CodeAgent). Execute the action and logs result into memory (an ActionStep). At the end of each step, run all callback functions defined in agent.step_callbacks. ⇒ This is where we added support to images: make a callback that logs images into memory! The figure below details this process: As you can see, for use cases where images are dynamically retrieved (e.g. web browser agent), we support adding images to the model’s ActionStep, in attribute step_log.observation_images. This can be done via a callback, which will be run at the end of each step. Let's demonstrate how to make such a callback, and using it to build a web browser agent.👇👇 How to create a Web browsing agent with vision We’re going to use helium. It provides browser automations based on selenium: this will be an easier way for our agent to manipulate webpages. pip install "smolagents[all]" helium selenium python-dotenv The agent itself can use helium directly, so no need for specific tools: it can directly use helium to perform actions, such as click("top 10") to click the button named "top 10" visible on the page. We still have to make some tools to help the agent navigate the web: a tool to go back to the previous page, and another tool to close pop-ups, because these are quite hard to grab for helium since they don’t have any text on their close buttons. from io import BytesIO from time import sleep import helium from dotenv import load_dotenv from PIL import Image from selenium import webdriver from selenium.common.exceptions import ElementNotInteractableException, TimeoutException from selenium.webdriver.common.by import By from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.support.ui import WebDriverWait from smolagents import CodeAgent, LiteLLMModel, OpenAIServerModel, TransformersModel, tool from smolagents.agents import ActionStep load_dotenv() import os @tool def search_item_ctrl_f(text: str, nth_result: int = 1) -> str: """ Searches for text on the current page via Ctrl + F and jumps to the nth occurrence. Args: text: The text to search for nth_result: Which occurrence to jump to (default: 1) """ elements = driver.find_elements(By.XPATH, f"//*[contains(text(), '{text}')]") if nth_result > len(elements): raise Exception(f"Match n°{nth_result} not found (only {len(elements)} matches found)") result = f"Found {len(elements)} matches for '{text}'." elem = elements[nth_result - 1] driver.execute_script("arguments[0].scrollIntoView(true);", elem) result += f"Focused on element {nth_result} of {len(elements)}" return result @tool def go_back() -> None: """Goes back to previous page.""" driver.back() @tool def close_popups() -> str: """ Closes any visible modal or pop-up on the page. Use this to dismiss pop-up windows! This does not work on cookie consent banners. """ # Common selectors for modal close buttons and overlay elements modal_selectors = [ "button[class*='close']", "[class*='modal']", "[class*='modal'] button", "[class*='CloseButton']", "[aria-label*='close']", ".modal-close", ".close-modal", ".modal .close", ".modal-backdrop", ".modal-overlay", "[class*='overlay']" ] wait = WebDriverWait(driver, timeout=0.5) for selector in modal_selectors: try: elements = wait.until( EC.presence_of_all_elements_located((By.CSS_SELECTOR, selector)) ) for element in elements: if element.is_displayed(): try: # Try clicking with JavaScript as it's more reliable driver.execute_script("arguments[0].click();", element) except ElementNotInteractableException: # If JavaScript click fails, try regular click element.click() except TimeoutException: continue except Exception as e: print(f"Error handling selector {selector}: {str(e)}") continue return "Modals closed" For now, the agent has no visual input. So let us demonstrate how to dynamically feed it images in its step logs by using a callback. We make a callback save_screenshot that will be run at the end of each step. def save_screenshot(step_log: ActionStep, agent: CodeAgent) -> None: sleep(1.0) # Let JavaScript animations happen before taking the screenshot driver = helium.get_driver() current_step = step_log.step_number if driver is not None: for step_logs in agent.logs: # Remove previous screenshots from logs for lean processing if isinstance(step_log, ActionStep) and step_log.step_number str: """ accept any visible cookie consent banners. """ wait = WebDriverWait(driver, timeout=0.5) elements = wait.until(EC.presence_of_all_elements_located((By.ID, "onetrust-accept-btn-handler"))) elements[0].click() agent = CodeAgent( tools=[go_back, close_popups, search_item_ctrl_f, close_cookie_popup], model=model, additional_authorized_imports=["helium"], step_callbacks=[save_screenshot], max_steps=20, verbosity_level=2,) See translation 🔥 1 1 + Reply christianweyer Feb 8, 2025 • edited Feb 8, 2025 Hm, what happened to the Agentic Web Browser sample? @m-ric https://github.com/huggingface/smolagents/blob/main/examples/vlm_web_browser.py See translation 1 reply · m-ric Article author Jun 2, 2025 We integrated it in the main repo here: https://github.com/huggingface/smolagents/blob/main/src/smolagents/vision_web_browser.pyThus it was removed from examples/ See translation 🚀 1 1 + kdb6df Feb 27, 2025 I'm getting this deprecation warning "The 'logs' attribute is deprecated and will soon be removed. Please use 'self.memory.steps' instead." What action should be taken? See translation Reply EditPreview Upload images, audio, and videos by dragging in the text input, pasting, or clicking here. Tap or paste here to upload images Comment · Sign up or log in to comment Upvote 113 +101Source: Hugging Face — Published — Category: Models