latent.rag.chunking

Native markdown chunker — replaces langchain-text-splitters.

Classes

MarkdownChunker

MarkdownChunker(max_chunk_size: int = 1000, chunk_overlap: int = 100)

Split markdown text on headers, tracking hierarchical section paths.

Parameters

max_chunk_size: Maximum characters per chunk. Sections longer than this are split on paragraph boundaries. Must be > 0. chunk_overlap: Number of characters to overlap between consecutive sub-chunks produced when a section exceeds max_chunk_size. Must be >= 0 and < max_chunk_size.

Methods

MarkdownChunker.chunk

chunk(text: str, source: str = '') -> list[DocumentChunk]

Split text into DocumentChunk objects.