Document Loaders and Text Splitters

How documents are loaded into a standard format and split into chunks, the popular loaders and splitters, two worked examples and the splitter parameters.

12Retrieval Augmented Generation (RAG) in LangChain 15Embeddings and Semantic Search 13LangChain with Vector Databases

14.1

What are Document Loaders?

Document Loaders help load data from different sources into a standard format (Document objects).
They support file types like PDF, TXT, DOCX, CSV, HTML, JSON, web pages, databases, etc.
Each loaded document contains page_content (text) and metadata (source, page number, etc.).
Loaders make it easy to integrate your private data with LLM applications.

14.2

What are Text Splitters?

Text Splitters divide large documents into smaller chunks.
LLMs have token limits, so splitting helps process large documents efficiently.
Different splitters are available based on the use case (character, token, semantic, recursive).
Proper chunking improves retrieval accuracy and reduces irrelevant context.
You can also set chunk size, overlap and custom splitting rules.

14.3

Document Loading and Splitting Flow

Data SourcesPDF, TXT / DOCX, web, databases

Document LoaderLoad into LangChain

Document Objectspage_content + metadata

Text SplitterSplit into chunks

Document ChunksChunk 1 ... Chunk N

14.4

Loader	Supported Source	Use Case
PyPDFLoader	PDF files	Load PDF documents
TextLoader	Text files (.txt)	Simple text documents
Docx2txtLoader	Word documents (.docx)	Load Word files
CSVLoader	CSV files	Structured data from CSV
WebBaseLoader	Web pages (URLs)	Load data from websites
DirectoryLoader	Multiple files in a folder	Batch load documents
JSONLoader	JSON files	Load structured JSON data
UnstructuredLoader	Various formats (PDF, HTML, etc.)	Advanced loader using unstructured

Popular Text Splitters in LangChain

Splitter	Description	Use Case
CharacterTextSplitter	Split by character count	Simple and fast
RecursiveCharacterTextSplitter	Split recursively by characters, sentences and other separators	Most commonly used
TokenTextSplitter	Split by tokens (e.g., using tiktoken)	When you want token-based chunks
SentenceTransformersTokenTextSplitter	Split using sentence transformers	Semantic chunking
MarkdownHeaderTextSplitter	Split based on markdown headers	Documents with headers (e.g., docs)
SemanticChunker	Split using semantic similarity	Better context preservation

14.6

Example 1: Load PDF and Split Using RecursiveCharacterTextSplitter

python

from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter

# 1. Load PDF document
loader = PyPDFLoader("data/company_policy.pdf")
docs = loader.load()
print(f"Total pages: {len(docs)}")

# 2. Split documents into chunks
splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    separators=["\n\n", "\n", " ", ""]
)
splits = splitter.split_documents(docs)
print(f"Total chunks: {len(splits)}")

# 3. Check a chunk
print(docs[0].page_content[:200])
print(splits[0].page_content)
print(splits[0].metadata)

14.7

Example 2: Load Web Pages and Split

python

from langchain_community.document_loaders import WebBaseLoader
from langchain_text_splitters import CharacterTextSplitter

# 1. Load web pages
loader = WebBaseLoader(["https://example.com/docs"])
docs = loader.load()
print(f"Loaded {len(docs)} documents")

# 2. Split into chunks
splitter = CharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50
)
splits = splitter.split_documents(docs)
print(f"Total chunks: {len(splits)}")

# 3. View chunk
print(splits[0].page_content)
print(splits[0].metadata)

14.8

Useful Parameters for Text Splitters

Parameter	Description	Example
chunk_size	Size of each chunk (in characters/tokens)	1000
chunk_overlap	Overlap between chunks	200
separators	List of separators to split text	["\n\n", "\n", ""]
length_function	Function to calculate length (len or tiktoken)	len
keep_separator	Keep the separator in the chunk	True / False

14.9

Key Takeaways

Document loaders help load data from multiple sources.
Text splitters divide large documents into smaller chunks.
Use the right loader based on your data source.
RecursiveCharacterTextSplitter is the most commonly used.
Proper chunk size and overlap improve retrieval accuracy.
Split documents before creating embeddings.
Support for PDF, Word, web, CSV, JSON and more.
Essential for building RAG applications and knowledge systems.