Understanding How PaliGemma 2 ScalesIn Bronzino’s An Allegory with Venus and Cupid, a smiling boy on the right holds roses or rose petals. When I asked PaliGemma 2 what he was holding, the 3B model at 224×224 answered “grapes.” The 448×448 checkpoint answered “flower.”At first, this looked like a…
Annons
Annons
Understanding How PaliGemma 2 ScalesIn Bronzino’s An Allegory with Venus and Cupid, a smiling boy on the right holds roses or rose petals. When I asked PaliGemma 2 what he was holding, the 3B model at 224×224 answered “grapes.” The 448×448 checkpoint answered “flower.”At first, this looked like a straightforward victory for higher resolution: perhaps the larger image preserved a detail that disappeared at 224 pixels.But a control experiment complicated that explanation. I reduced the painting to 224×224, enlarged it back to 448×448, and submitted the degraded image to the 448 checkpoint.The missing visual information could not have been restored by enlarging the image, yet the model still answered “flower.”The example raised a broader question about multimodal scaling. When a model improves, what deserves the credit: more visual detail, a longer sequence of visual tokens, additional high-resolution training, or a larger language model?PaliGemma 2 changes several of these factors together, so the answer depends both on the task’s bottleneck and on which variables the comparison actually isolates.The example raised a broader question about multimodal scaling. When a model improves, what deserves the credit: more visual detail, a longer sequence of visual tokens, additional high-resolution training, or a larger language model?PaliGemma 2 changes several of these factors together, so the answer depends both on the task’s bottleneck and on which variables the comparison actually isolates.How Does an Image Become an Answer?Before diving into the paper, let us first understand how a vision–language model turns an image and a question into an answer. A VLM such as PaliGemma 2 is built around a language model, but adds a vision encoder that allows visual information to enter the language model.The vision encoder converts an image into high-dimensional numerical representations called embeddings. These representations are then projected into a form that the language model can process together with the text prompt.In PaliGemma 2, this process can be divided into six steps:The input image is resized to the resolution expected by the model.The image is divided into small patches.The SigLIP vision encoder converts the patches into contextual embeddings.A projection layer maps these embeddings into Gemma 2’s embedding space.The projected visual tokens are combined with the tokens of the text prompt.Gemma 2 generates the answer one token at a time.The number of visual tokens depends on the input resolution. Because SigLIP uses 14×14-pixel patches, a 224×224 image produces 256 visual tokens, while a 448×448 image produces 1,024. Higher resolution therefore gives the language model a longer and potentially more detailed visual sequence.This explains how visual information reaches the language model, but it raises another question: How does the vision encoder learn representations that correspond to meaningful objects, actions, and concepts? PaliGemma 2 uses SigLIP as its vision encoder.During SigLIP’s pretraining, an image encoder and a text encoder independently transform images and captions into numerical representations.The model learns to assign a high matching probability to related image–text pairs and a low probability to unrelated pairs. This contrastive training encourages visual representations to align with concepts that can be expressed in language.SigLIP determines what visual evidence is made available, but Gemma 2 must still interpret that evidence and turn it into an answer. This distinction helps explain PaliGemma 2’s two main scaling choices: increasing resolution changes the visual sequence supplied to the model, while increasing model size changes the capacity available to interpret that sequence.What Does the Paper Investigate?Now that we have an idea of how PaliGemma 2 can process visual information, we can try to understand the main questions of the paper. The paper does not propose an entirely new architecture.Instead, PaliGemma 2 builds on the architecture and multistage training approach of the original PaliGemma, while replacing its language model with models from the Gemma 2 family. The ‘transfer’ in the title of the paper means starting from a generally pretrained checkpoint and then fine-tuning it for a particular downstream task.PaliGemma 2 is therefore not expected to perform every task perfectly without adaptation; the paper studies how useful the same starting point becomes after task-specific training. This training process can roughly be divided into three stages:Stage1: The SigLIP vision encoder and Gemma 2 language model are combined and pretrained at 224×224 resolution on a mixture of vision-language tasks.Stage2: The model continues training at higher image resolutions, 448×448 and 896×896, allowing it to process finer visual details.Stage3: The pretrained models are transferred to specific downstream tasks through fine-tuning.By training different model sizes at different image resolutions and transferring them to a wide range of tasks, the authors can investigate several important questions:What happens to performance when image resolution increases?What happens when the PaliGemma2 variant becomes larger?To answer these questions, three model variants (3B, 10B, and 28B) were investigated at two resolutions (224×224 and 448×448) on more than 30 academic benchmarks.Scaling Visual Evidence: What Resolution Changes?The Bronzino example is useful because it makes the ambiguity visible, but a single painting cannot tell us whether the pattern is systematic. To investigate it more carefully, I tested PaliGemma 2 on twelve Renaissance paintings using sixty questions about details that could become difficult to identify after resizing.The set contained four famous, four moderately familiar, and four relatively obscure paintings.(Famous group: The Birth of Venus — Sandro Botticelli; The Last Supper — Leonardo da Vinci; The Wedding Feast at Cana — Paolo Veronese; An Allegory with Venus and Cupid — Agnolo Bronzino, Moderately familiar: Virgin and Child with an Angel — Sandro Botticelli; The Garden of Paradise — Workshop of Hieronymus Bosch; The Baptism of Christ — Piero della Francesca; The Feast of the Gods — Giovanni Bellini and Titian, Relatively obscure: The Adventures of Ulysses — Apollonio di Giovanni; Allegory — Piero di Cosimo; Claudia Quinta — Neroccio de’ Landi; Orpheus — Venetian, 16th century).This division was not intended as a formal measure of memorization, but it reduced the risk of drawing every conclusion from works that frequently appear online. Each painting received five questions. Across the complete set, 24 questions concerned distant details, 10 concerned hand-held attributes, 2 concerned inscriptions, 12 concerned secondary figures, and 12 contained false premises about objects that were not present.The paper evaluates PaliGemma 2 primarily through task-specific fine-tuning. My experiment instead uses the later 3B Mix checkpoints, which were fine-tuned on a mixture of vision-language tasks and can be prompted directly. It should therefore be read as a small behavioral study of the released Mix models, not as a replication of the paper’s transfer benchmarks.I evaluated every image-question pair under three conditions:Reducing an image to 224×224 removes fine visual information, and enlarging it cannot restore that information. At the same time, using the 448 checkpoint keeps its model weights and 1,024-token visual sequence unchanged.This produced 180 answers in total. Before grading, the answers were randomized, and their condition labels were hidden. Correctness was scored as 0 for an incorrect answer, 1 for a partly correct answer, and 2 for a correct answer. I also recorded whether an answer hallucinated visual evidence and whether it rejected a false premise.At first sight, the result appears to support higher resolution. The native 448 condition achieved the highest mean score, produced more fully correct answers, and hallucinated less often than the 224 condition. But the control comparison makes the result less straightforward.Native and degraded 448 received the same 0–2 correctness score on 50 of the 60 questions, although their wording was not necessarily identical. Among the remaining ten questions, each condition performed better five times. However, the native condition’s wins were larger: it improved by two points on three questions and by one point on two, whereas each degraded-condition win was a one-point difference.This produced total scores of 59 for native 448 and 56 for degraded 448, corresponding to means of 0.983 and 0.933. False-premise rejection was also identical: both conditions rejected five of twelve misleading questions. The native 448 input was slightly better on average, but it did not produce a consistent advantage over the degraded input.The 224 condition obtained 43 total correctness points, compared with 56 for degraded 448 and 59 for native 448. In other words, 13 of the native 448 condition’s 16-point advantage over 224, approximately 81%, remained after the image had been reduced to 224×224 and its fine detail could no longer be recovered.Native and degraded 448 received the same correctness score on 50 of the 60 painting questions. This raised an alternative explanation: perhaps many of the selected details were already visible at 224×224, in which case the degradation procedure would have little opportunity to affect the answer. Before interpreting the control, I therefore needed to verify that it could detect a difference when the task genuinely depended on fine visual information.I created a small calibration set containing ten synthetic museum labels. Each label contained five lines of text, including invented names, room numbers, and catalog codes, rendered at sizes between 10 and 20 pixels. Because the complete reference transcription was known, the outputs could be evaluated using character error rate rather than subjective grading.I tested the same three conditions used in the painting experiment: native 224, native 448, and an image reduced to 224 before being enlarged back to 448. All conditions received the same prompt: “Read text.”Native 448 produced the lowest mean character error rate and exactly transcribed seven of the ten labels. Native 224 exactly transcribed only one. Degraded 448 exactly transcribed six labels, but failed catastrophically on two others, producing a substantially higher mean error rate.The exact-match comparison reveals a second result. Degraded 448 contained no more recoverable pixel information than native 224, yet it exactly transcribed six labels rather than one. On an 11-pixel label, for example, native 224 rendered “TOMAS VELIN” as “TOBACCO KEEN,” while degraded 448 transcribed all five lines correctly.As in the painting experiment, much of the 448 configuration’s advantage therefore survived without additional recoverable input detail. That advantage could come from checkpoint training, model weights, the denser visual-token grid, or an interaction between them.In the paired native-versus-degraded comparison, native 448 achieved a lower error rate on four labels, degraded 448 performed better on one, and five tied. The small-text test therefore established two points at once: the degradation procedure could remove useful visual information, but the 448 configuration could also outperform native 224 without receiving more recoverable detail.The 50 ties in the painting experiment may mean that many questions were already answerable at 224×224, although the test does not prove that small text and painting details are affected in the same way.This calibration is not a general OCR benchmark. It contains only ten synthetic labels in one typeface; its mean error rate was distorted by two catastrophic degraded-input failures, and the degraded condition was sensitive to prompt wording. Even so, the two experiments point in the same direction.In the small-text calibration, native detail helped on some labels, but degraded 448 still exactly transcribed six labels compared with one for native 224. In the Renaissance-painting experiment, most of the 448 checkpoint’s improvement likewise survived after native fine detail was removed.Taken together, these results show that “the 448 model performed better” is not equivalent to “the extra image detail caused the improvement.” The remaining advantage belongs to the combined 448 configuration: these experiments cannot separate checkpoint training, model weights, the larger visual-token sequence, or interactions among them.Scaling the Language Model: Does Bigger Mean Better?Resolution is only one way to scale a vision–language model. Once visual evidence has been converted into visual tokens, the language model must interpret those tokens, combine information across the image, relate it to the prompt, and generate an answer.The paper compares PaliGemma2 variants with approximately 3B, 10B, and 28B parameters. Larger models tend to be more useful when the relevant evidence is already visible but must be combined or interpreted. This includes tasks involving counting, spatial relationships, multi-step reasoning, and multilingual understanding.However, the improvement is not uniform. Moving from 3B to 10B often produces a clearer gain than moving from 10B to 28B, and some tasks receive little benefit from additional language-model capacity.Hugging Face published a comparison in which the 3B and 10B models receive the same dense cocktail menu at 448×448 with the prompt “Read text.”This example did not convince me that 10B is better. Both models recover a surprising amount of text, but both merge lines and substitute or invent words. The 10B output still contains errors such as “SUN-HISE” and “fleur de bureau.” Without a reference transcription and a metric such as character error rate, there is no clear winner.The menu example should not be interpreted as contradicting the paper’s aggregate results. The benchmarks provide broader evidence that larger language models help on tasks requiring reasoning and interpretation. The menu instead illustrates the boundary of that advantage: when the main bottleneck is recovering exact visual evidence, additional language-model capacity may not solve the problem.This distinction also matters computationally. According to the paper’s estimates, the 3B/448 configuration requires 4.6 times the relative training cost per example of the 3B/224 baseline, while the 10B/448 configuration requires 18.3 times.The larger model therefore uses almost four times the relative training cost per example of 3B/448 while receiving the same number of visual tokens. Whether that additional cost is worthwhile depends on whether the task is limited by interpretation rather than by visual fidelity.What Did Scaling Cost in Practice?The paper reports estimated relative training cost per example, but I did not train these models. For my experiment, the more relevant question is how much the released checkpoints cost to run.I therefore used the timing information recorded during the 180 inference runs. All three conditions were evaluated on an Apple M4 Max with 36 GB of unified memory using PyTorch MPS and float16 precision.The timer included model preprocessing, generation, and decoding, but not image loading or the creation of the degraded image. Each condition contained the same sixty image–question pairs and used greedy decoding with a maximum of 48 generated tokens.Moving from 224×224 to the native 448 configuration made inference approximately 2.7 times slower per answer. This increase is smaller than the fourfold increase in visual-token count, from 256 to 1,024, because inference also contains fixed costs and because runtime does not scale perfectly linearly with sequence length.The degraded condition provides the most revealing comparison. Although its fine visual information had been removed, it took almost exactly as long to process as the native 448 image: median times were 0.648 and 0.664 seconds.The computational cost follows the size of the representation rather than the amount of useful information contained in it. Enlarging a degraded image still creates 1,024 visual tokens, and the model must process all of them even when they do not contain additional recoverable detail.This produces an important contrast with the accuracy results. Native 224 obtained 43 total correctness points, degraded 448 obtained 56, and native 448 obtained 59. Most of the accuracy improvement came from moving to the 448 configuration, while restoring native high-resolution detail added only three more points. Yet degraded and native 448 incurred almost the same inference cost.In this experiment, it cost approximately the full 448 inference cost even when the input did not contain native 448-level detail. Higher resolution was computationally more expensive, but the value of that cost depended on whether the additional visual tokens carried information the model could actually use.What Convinced Me and What Did Not?What convinced me most was not a single benchmark result, but the consistency of the paper’s broader pattern: scaling helps only when it addresses the task’s actual bottleneck.Higher resolution matters most when information is visually small or densely packed, while a larger language model is more useful when the visible evidence must be interpreted, combined, or expressed in a complex answer. PaliGemma 2 therefore presents scaling as a choice between different resources, rather than a guarantee that every larger model will perform better.Our small-text experiment supported the first part of this argument: when characters were genuinely difficult to see, preserving visual detail mattered. However, the Renaissance-painting experiment made me less convinced by comparisons that treat resolution as an isolated variable. The 448 checkpoint generally outperformed the 224 checkpoint, yet native 448 and an image reduced to 224 and then enlarged back to 448 received the same score on 50 of 60 questions.In the remaining ten, each condition performed better five times. This suggests that additional image detail alone was not the main explanation for the difference. However, the experiment cannot determine whether the remaining advantage came from training history, checkpoint weights, the larger visual-token grid, or an interaction between them.The paper’s design cannot fully separate these effects because its higher-resolution models also receive additional training, with greater emphasis on tasks that benefit from resolution. Our experiment does not prove that pixels are unimportant: most of our questions may already have been answerable at 224×224. It does show that “the 448 model performed better” is not equivalent to “the extra image detail caused the improvement.”My main takeaway is therefore conditional: more pixels help when the missing evidence is visual, and more language-model capacity helps when the challenge is interpretation. But checkpoint, training data, token count, model size, and input detail change together easily. A convincing scaling claim must identify which of them actually produced the improvement.ReferencesSteiner, A., et al. (2024). PaliGemma 2: A Family of Versatile VLMs for Transfer. arXiv:2412.03555.Beyer, L., et al. (2024). PaliGemma: A Versatile 3B VLM for Transfer. arXiv:2407.07726.Zhai, X., Mustafa, B., Kolesnikov, A., & Beyer, L. (2023). Sigmoid Loss for Language-Image Pre-Training. ICCV 2023.Gemma Team. (2024). Gemma 2: Improving Open Language Models at a Practical Size. arXiv:2408.00118.Dosovitskiy, A., et al. (2020). An Image Is Worth 16×16 Words: Transformers for Image Recognition at Scale. ICLR 2021.PaliGemma documentation — Hugging Face TransformersPaliGemma 2 3B Mix, 224×224 model cardPaliGemma 2 3B Mix, 448×448 model cardPaliGemma 2 10B Mix, 448×448 model cardPaliGemma 2 28B Mix, 224×224 model cardPaliGemma 2 28B Mix, 448×448 model cardHugging Face. PaliGemma 2 Mix: New Instruction Vision Language Models by Google.Hugging Face. Welcome PaliGemma 2: New Vision-Language Models by Google.https://github.com/azizayan/paligemma2-experimentThis story is published under the Generative AI publication. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories. Let’s shape the future of AI together!Seeing More or Reasoning Better? was originally published in Generative AI on Medium, where people are continuing the conversation by highlighting and responding to this story.