How I Built a Python Tool to Recommend the Right Chunking Strategy for RAG
Using document structure, rule-based scoring, and an optional LLM decision to choose between hierarchical and recursive chunking.RAG has become a popular approach for building AI applications that answer questions using documents, knowledge bases, and enterprise data.But before a RAG system can…
Annons
Annons
Using document structure, rule-based scoring, and an optional LLM decision to choose between hierarchical and recursive chunking.RAG has become a popular approach for building AI applications that answer questions using documents, knowledge bases, and enterprise data.But before a RAG system can retrieve relevant information, documents need to be prepared and divided into manageable pieces, commonly called chunks.This introduces an important question:How do we decide which chunking strategy is appropriate for a document?A document containing multiple chapters, nested headings, and numbered sections has a different structure from a short text file or a form filled with fields and values. Applying the same chunking strategy to every document may not always be the best approach.That question inspired me to build a Python-based tool that analyzes document structure and recommends a suitable chunking method: hierarchical chunking or recursive chunking.In this article, I’ll explain the problem, the approach, the scoring system, and how I designed the code as a standalone project.Understanding the two chunking strategiesBefore discussing the implementation, let’s understand the two methods.1. Recursive chunkingRecursive chunking divides text into smaller sections using a sequence of separators or boundaries. Depending on the implementation, it may try paragraph breaks first, followed by sentences, spaces, or smaller units.It is a useful general-purpose approach when a document does not contain a strong, reliable hierarchy.For example, consider a document containing a few paragraphs of general information. Recursive chunking can divide the text into manageable pieces without requiring a complex heading structure.2. Hierarchical chunkingHierarchical chunking takes the document’s organization into account, such as chapters, sections, subsections, and nested headings.Preserving this hierarchy can help a retrieval system maintain relationships between broader topics and their more specific sections.The important point is that neither strategy is universally better. The appropriate choice depends on the document and the retrieval system’s requirements.The challenge: documents have different structuresDocuments come in many forms:Technical manuals with chapters and subsectionsPDFs containing bookmarks and tables of contentsMarkdown documents with multiple heading levelsWord documents with formatted headingsForms containing labels and valuesShort text files with little or no hierarchyA simple approach would be to choose a chunking method based only on the file extension. However, two PDF files can have completely different structures. Instead, I wanted to examine the content and its structural signals before recommending a method.My approach: a rule-based scoring systemThe core of the project is a scoring engine that looks for structural characteristics in a document.Instead of immediately asking an LLM to classify every document, the tool first applies deterministic rules. Each rule contributes positive or negative points to an overall score.The score then determines whether a recommendation can be made directly or whether an additional decision is needed.Structural signalsThe implementation examines several features.1. PDF bookmarksPDF bookmarks can reveal the document’s navigation structure. A bookmark outline with multiple levels may indicate that the document has a meaningful hierarchy.2. Table of contentsA printed table of contents can provide evidence of organized sections and subsections.3. Markdown headingsMarkdown headings use prefixes such as #, ##, and ###. Multiple heading levels can indicate a hierarchical structure.4. Font-based headingsFor PDFs, the tool examines font sizes and bold formatting to identify possible headings. It also attempts to filter out repeated headers and footers.5. Numbered sections and clausesPatterns such as 1.1 Overview, 1.2 Configuration, and nested numbered clauses can provide additional evidence of document hierarchy.6. Heading distributionThe tool considers how many headings appear, how widely they are distributed, and the approximate token distance between sections.7. Prose-heavy and form-heavy contentDocuments dominated by running prose or form fields may have less useful section hierarchy for this particular decision. The scoring rules can reduce their scores accordingly.8. Document lengthVery short documents receive a negative score because a complex hierarchical strategy may not be justified by their limited content.These signals are combined rather than treated as independent proof. For example, a document can contain many headings but still be a form or table with little meaningful section structure.How the scoring worksThe project uses configurable thresholds defined in constants.py.A low score leads to recursive chunking. When the score falls between the thresholds, the standalone implementation can call an optional decision callback. If no callback is supplied, it uses recursive chunking as the fallback. This design keeps the rule-based part predictable while allowing an LLM integration to be added when needed.Where an LLM fits into the processRule-based logic is useful, but document structure can be ambiguous. For example, a document may have several numbered headings without forming a meaningful hierarchy. Another document might contain a mixture of prose, sections, and nested clauses.An LLM can provide another opinion when the rules do not produce a decisive result. In the standalone version, I separated this responsibility from the document-analysis logic by introducing an optional callback.Conceptually, the flow looks like this:Read the document.Extract structural information.Calculate the rule score.Return a recommendation if the score crosses a threshold.Otherwise, call the optional decision function.If the decision function is unavailable or fails, fall back to recursive chunking.The callback can be connected to a model provider of your choice. The standalone project does not include a working provider-specific LLM integration by default. This separation also means the rule-based functionality does not require an API key.Designing the Python projectI organized the implementation into three main Python files.utils.pyThis file contains the document-reading functions, structural analysis, scoring logic, explanations, and the main choose_chunking_method() function.It supports PDF, DOCX, Markdown, and TXT files. JPG, JPEG, and PNG extensions are also recognized, but image files currently default to recursive chunking without OCR.For PDFs, the tool extracts text lines and, where available, font metadata and bookmarks. The number of pages analyzed is configurable, with a default limit of 30 pages.constants.pyThis file contains the configurable values used by the scoring engine, including:Hierarchical and recursive thresholdsMaximum PDF pagesHeading size and boldness thresholdsMinimum heading-style occurrencesSupported file extensionsRegular expressions for headings, clauses, and form fieldsKeeping these values in one place makes it easier to adjust the rules without searching through the entire implementation.models.pyThis file defines the data structures used by the analysis.The Line model stores extracted text, page number, font size, boldness ratio, token count, and token offset.The Doc model stores document metadata, extracted lines, and bookmarks. It also provides a property for calculating the total token count.The ChunkingMethodDecision model represents a structured decision containing the selected chunking method and a confidence value.Separating these models from the processing logic helps keep the code organized and easier to maintain.Making the code standaloneOne of the important steps in preparing the project for GitHub was removing dependencies on private modules from the original application.The original implementation relied on application-specific utilities for token counting, LLM construction, provider configuration, and prompt definitions. Those dependencies are useful inside a larger application, but they prevent the code from running independently when the surrounding modules are unavailable.For the standalone version, I replaced the private token-counting dependency with a lightweight token-count approximation and exposed LLM decision-making through an optional callback. This makes the project easier to explore and extend without requiring the original application.There are trade-offs, however. The approximate token count may differ from the tokenizer used by a particular embedding model, and the standalone version does not automatically connect to OpenAI, Gemini, or Anthropic.Running the projectAfter installing the dependencies, the tool can be executed from the command line.For a Markdown document:python utils.py path/to/document.mdTo analyze a PDF with a custom page limit:python utils.py path/to/document.pdf --max-pages 10To save the result as JSON:python utils.py path/to/document.pdf --out result.jsonThe output contains the recommended chunking method and an explanation of the rules that contributed to the decision.For example, the result might look like this:{ "path/to/document.md": { "chunking_method": "recursive", "reason": "Rule score ..." }}The exact score and explanation depend on the document being analyzed.Limitations and future improvementsThe project is a starting point rather than a complete RAG pipeline. There are several areas I would like to improve:1. Better token countingThe current standalone version uses an approximate tokenizer. Integrating the tokenizer used by the target embedding model would make section-size estimates more representative.2. OCR supportImage files currently default to recursive chunking. Adding OCR would allow the system to extract and analyze text from scanned documents and images.A higher structural score is a heuristic, not proof that a strategy will perform better. Empirical evaluation is necessary to validate the approach.Final ThoughtsBuilding AutoChunk-AI helped me think about document preprocessing as a decision-making problem rather than a one-size-fits-all step in a RAG pipeline.The key idea is simple: inspect the document first, apply explicit rules when the evidence is strong, and use an additional model-based decision when the evidence is ambiguous.Just as importantly, separating document parsing, scoring rules, data models, and optional model integration makes the implementation easier to understand, maintain, and extend. This project is still evolving, but it provides a foundation for exploring how document-aware preprocessing can improve the way we prepare data for RAG systems.You can explore the project, review the implementation, and follow its development on GitHub:🔗 GitHub Repository: https://github.com/NarmadhaGanesan/AutoChunk-AIIf you’re building RAG applications, I’d love to hear how you approach chunking different types of documents.What signals have you found most useful when deciding how to chunk a document? Do you rely on document structure, semantic similarity, token limits, rule-based heuristics, or a combination of these approaches?Let’s share ideas and learn from each other’s experiences!This story is published under the Generative AI publication. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories. Let’s shape the future of AI together!How I Built a Python Tool to Recommend the Right Chunking Strategy for RAG was originally published in Generative AI on Medium, where people are continuing the conversation by highlighting and responding to this story.