harshaA fixed chunk size is convenient. Documents are rarely written in fixed units of meaning.
A heading introduces a section. A table depends on its column labels. An exception modifies a rule in the previous paragraph. A footnote may change the interpretation of a number. Splitting these relationships can make a perfectly extracted document difficult to retrieve correctly.
Preserve useful boundaries
For an illustrative handbook, I would prefer a passage that carries its section heading and the qualification attached to its rule. The goal is not to make every chunk large. It is to keep the context needed to interpret a retrieved passage.
For a spreadsheet-derived document, a row without column names is often an incomplete fact. For an invoice, a value without its label is ambiguous. For a multi-page table, the page boundary may be less meaningful than the table boundary.
Chunking should therefore start with document structure where that structure is available.
Treat extraction as a separate source of error
OCR fallback can make scanned material searchable, but extraction errors can distort both retrieval and answers. An identifier with one wrong character may defeat exact search. A reordered table may turn the right values into the wrong relationships.
I would retain extraction method and document version as metadata, then test representative scanned and digital documents separately. A chunking change should not be credited with solving a problem that actually came from better extraction.
Likewise, more overlap may recover missing context while increasing redundancy. It is a tradeoff to measure, not an automatic improvement.
Review the retrieved passage as a reader
A useful manual test is to hide the original document and read only the retrieved chunk. Can a person understand what it claims? Can they identify its subject, scope, and qualifications? Can they follow a citation back to the surrounding material?
That test catches problems a token-count report cannot.
I think of chunking as preparing evidence for a reader who may arrive out of order. The system is responsible for giving that reader enough context to interpret what was found. Good boundaries make retrieval more useful before a model writes a single sentence.