If you read what a vector database does, this is the part where you build one. This n8n workflow takes raw text files, transcripts, articles, notes, anything text-based, sitting in cloud storage, splits them into meaningful chunks, and loads them into a vector database for semantic search or retrieval-augmented generation (RAG): the pattern of pulling real, relevant content into an AI's context before it generates a response.

It's written to be rebuilt with any file source, any embedding provider, and any vector database. The specific tools in the reference build, Google Drive, OpenAI, Pinecone, Claude, are examples, not requirements.

What the pipeline does, in one paragraph

It finds unprocessed text files in a folder, downloads and cleans each one, breaks the text into chunks small enough to stay semantically coherent, optionally tags each chunk with metadata using an LLM, converts each chunk into a vector embedding, and writes the vectors to a vector database, with a progress checkpoint at every stage so a failure partway through never leaves duplicate or missing data behind.

Architecture at a glance

Stage 1: Setup & discovery
Trigger → resolve vector DB endpoint → list source files → loop (one file at a time)

Stage 2: Prepare the text
Download file → clean & normalize text → categorize content → assign target namespace/collection

Stage 3: Chunk the text
Split cleaned text into semantically bounded chunks

Stage 4: Resume check (optional but recommended)
Look up prior progress for this file → skip completed chunks → keep only what's left

Stage 5: Enrich & embed (batched)
Loop chunks in batches → optionally tag each chunk with an LLM → generate embeddings for the batch → save checkpoint

Stage 6: Write & verify
Upsert vectors to the database → re-read progress to confirm the count matches → mark the file complete → move or tag it as processed → advance to the next file

What you'll need

  • A trigger: Manual for testing, Schedule or Webhook for production. Nothing downstream depends on which one you pick.
  • A file source: a Google Drive folder, an S3 bucket, a Dropbox folder, or a local directory, anything that can list and download files.
  • An embedding provider: any embeddings API with a batch-input endpoint, like OpenAI, Cohere, Voyage, or a self-hosted model.
  • A vector database: Pinecone, Qdrant, Weaviate, pgvector, Milvus, or similar.
  • Somewhere to persist progress: an n8n Data Table, a database table, or even a flat file, anything that survives outside a single workflow run.

Stage 1: Trigger and set up the destination

The reference build uses a Manual Trigger for on-demand runs while testing. For production, swap in a Schedule Trigger that polls a folder every few minutes, or a Webhook that fires when a file lands. The rest of the workflow doesn't care which one you use, it just needs to hand off to the next node.

Some vector databases, Pinecone included, separate the control plane, where you manage index configuration, from the data plane, the actual host you write vectors to. If your database works this way, the first real step is a lookup call (an HTTP Request node) that returns the write endpoint you'll use later. Databases with a single flat API, like a self-hosted pgvector instance, Qdrant, or Weaviate, can skip this node entirely and hardcode the endpoint.

Next, list your source files with whatever native node your storage provider has. Point it at an "inbox" folder so the workflow only ever sees unprocessed files, and pull back just the fields you need (id, name, type) to keep the payload light.

Then loop over those files one at a time using a Split In Batches node set to a batch size of 1. This is a deliberate choice: process one file fully, chunk it, embed it, write it, verify it, before moving to the next. That isolates failures to a single file instead of corrupting a whole batch, and it's what makes the resumability pattern in Stage 4 possible.

Stage 2: Download and normalize the text

If your source stores rich formats, Google Docs, Word docs, PDFs, convert to plain text at download time rather than trying to parse formatting later. Chunking logic is much simpler against clean plain text.

Raw transcripts and documents carry a lot of noise that doesn't belong in an embedding: timestamps like [00:12:34], audio or video tags like [Music] or [Applause], inconsistent line breaks, and doubled-up whitespace. Strip all of it in a Code node before chunking. This matters for two reasons: noise inflates your word and token counts, throwing off chunk-size targeting, and it can leak into the embedding itself, making two chunks about different things look artificially similar to the vector database's similarity search.

This is also the natural place to categorize the content, by filename keywords, folder, or a quick LLM classification, and decide where it should land in the database: a namespace, a collection, or a metadata tag. Categorizing here lets your chunking logic branch by content type later, which matters because a tutorial transcript and a list of social posts don't chunk the same way.

Stage 3: Chunk the text (the step that matters most)

This is a Code node, and it's the heart of the whole pipeline. The reference build branches its strategy by content type:

  • Structured, narrative content (tutorials, interviews, long-form transcripts): split on sentence boundaries, then group sentences into a chunk until either a natural topic or function shift is detected (greeting, hook, instruction, example, call-to-action, and so on, caught via keyword or regex markers) or the chunk hits a target word count. This keeps each chunk centered on one idea instead of cutting mid-thought.
  • Short, self-contained content (social posts, FAQ entries, short notes): split on the natural unit boundary that already exists in the source, a blank line or a post separator, rather than forcing a word-count split.
  • Everything else, the fallback: a sliding sentence window. Keep adding sentences until you hit a target word count, then start the next chunk a sentence or two before the previous one ended. That overlap means no sentence's context is fully lost at a chunk boundary.

See the chunking section below for why this approach, and chunk size in general, matters so much for retrieval quality.

Stage 4: Check for resumable progress

Before doing any expensive work, LLM calls, embeddings, API writes, check whether this file has partial progress from a previous run, stored in a lightweight persistent store: an n8n Data Table, a database table, or even a simple key-value store. If it does, skip the chunks already written and only process what's left. This single pattern turns a fragile "run the whole file or start over" job into one that survives API timeouts, rate limits, and crashes without creating duplicate vectors or silently dropping content.

Stage 5: Batch, enrich, and embed

Batch the chunks with a Split In Batches node, 25 at a time is a reasonable default. Don't call your embedding API once per chunk, and don't send an entire file's worth of chunks in one request. Too many small calls burns time and hits rate limits; one giant call risks timeouts and makes a single failure expensive to retry. Tune the batch size against your embedding provider's rate limits and your average chunk size.

Optionally classify each chunk with an LLM. The reference build sends each chunk to a fast, cheap model (Claude Haiku) with a prompt asking it to return structured JSON describing tone, topic, audience, and stylistic patterns. This isn't required to build vectors, but it turns your database's metadata from just the raw text into something you can filter and facet on later, like "only search chunks tagged tutorial and beginner," which meaningfully improves retrieval precision when you've got mixed content types. Skip this node if you don't need faceted filtering.

Generate embeddings with an HTTP Request to your embedding provider, or a native embeddings node. Send the batch of chunk texts as a single array rather than one request per chunk. Many providers, OpenAI included, accept batch input natively, and it's dramatically cheaper in request overhead.

Build the vector payload in a Code node: assemble what your database expects to receive. That's a unique ID per chunk (something like {fileId}-chunk-{index}, deterministic IDs mean re-running a batch overwrites rather than duplicates), the embedding values, and a metadata object holding the chunk text itself plus anything from earlier stages you want to filter on later, like source document, category, chunk index, or LLM-derived tags.

Save a checkpoint after each batch is embedded and ready, recording how many chunks are done before moving to the write step. This is what Stage 4's resume check reads from on a future run.

Stage 6: Write to the vector database and verify

Upsert the batch to your database's write endpoint. Upsert (update-or-insert) matters here because IDs are deterministic, so a retried batch overwrites the same vectors instead of creating duplicates.

Loop until all batches for the file are done, then run a final verification: re-read the progress checkpoint and confirm the completed-chunk count matches the total chunk count for the file. Only mark the file complete if they match. This is your last line of defense against silently under-writing a file because of a swallowed error mid-loop.

Once verified, move or tag the source file as processed, a "Processed" folder, a status tag, a database flag, so the next run of the workflow never picks it up again.

Why chunking matters (and why smaller isn't automatically better)

The core problem chunking solves: embedding models compress text into a single fixed-length vector, and that vector is meant to represent one coherent idea. Feed it an entire 5,000-word transcript, and you get one of two bad outcomes: the model truncates the input and only "sees" the beginning, or it embeds the whole thing into one vector that averages a dozen different topics together. Either way, when someone later searches for one specific idea from that document, the giant vector is a poor match for it, the signal you want gets diluted by everything else stuffed into the same chunk. This is often called semantic dilution.

So smaller chunks generally retrieve better: a chunk that's about one thing produces an embedding that's a sharp match for a query about that one thing. But smaller isn't free. Push chunk size too small, a single sentence, a fragment, and you lose context. "It broke because of that" is a chunk with no idea what "it" refers to. A search might correctly retrieve that fragment and hand a downstream LLM a sentence it can't use.

The practical target is a range, not an extreme: chunks big enough to hold a complete thought, a paragraph-ish unit, roughly 100 to 300 words is a common range, with boundaries chosen at sentence and topic breakpoints rather than a blind character count that might slice a sentence in half. Two refinements make retrieval meaningfully better:

  • Semantic-boundary chunking over fixed-size chunking. Splitting at detected topic or function shifts, as in Stage 3, rather than every 500 characters keeps each chunk thematically coherent, which is what drives retrieval accuracy. Chunk size is a proxy for topic coherence, not the goal itself.
  • Overlap between chunks. Carrying the last sentence or two of one chunk into the start of the next means a query that lands right on a chunk boundary still gets full context, instead of retrieving a chunk that opens mid-thought.

The downstream effect compounds in RAG specifically: every irrelevant word in a retrieved chunk is a word an LLM has to read, and pay for, when generating an answer, and a word that can distract the model from the relevant sentence sitting right next to it. Tight, well-bounded chunks keep retrieval precise and keep the context you hand to a downstream LLM clean.

Adapting this to a different stack

Nothing about this pattern is tied to Google Drive, OpenAI, Pinecone, or Claude specifically:

  • File source: any node that can list and download files, Drive, S3, Dropbox, SharePoint, or a local-file trigger.
  • Embedding provider: any embeddings API (OpenAI, Cohere, Voyage, a self-hosted model), the only requirement is a batch-input endpoint and knowing the vector dimension your database expects.
  • Vector database: Pinecone, Qdrant, Weaviate, pgvector, Milvus, or others, check whether it needs a separate endpoint-resolution step and confirm its expected upsert payload shape, since field names vary.
  • Chunk classification model: optional entirely. If you use one, pick the cheapest, fastest model that reliably returns structured JSON, since it runs once per chunk.
  • Progress store: an n8n Data Table, a Postgres or SQLite table, or even a flat file, anything that persists outside a single workflow execution.

Common pitfalls

  • Chunking on character count alone. It's the easiest thing to implement and the worst for retrieval quality, it will regularly slice a sentence, and sometimes a word, in half.
  • No overlap between chunks. Saves a small amount of redundant storage at the cost of losing context right at chunk boundaries, usually not worth it.
  • Skipping the resume and checkpoint pattern because it's extra nodes. The first time a rate limit or timeout kills a run halfway through a 40-chunk file, you'll either have duplicate vectors or a silently incomplete document in your database. The checkpoint pattern costs a few nodes and prevents both.
  • Embedding one giant call per file instead of batching chunks. Slower, more expensive, and a single failure forces you to redo the entire file instead of one batch.
  • Not verifying the write. An upsert call returning HTTP 200 doesn't guarantee every vector in the batch was written correctly. A final count-check against the database, or its progress ledger, is cheap insurance.

That's the whole pipeline: files in, embeddings out, with a checkpoint at every stage so nothing gets duplicated or dropped. If you haven't read what a vector database is and why this is worth building in the first place, that's the companion piece. If n8n itself is new to you, start here before this one, and if a Claude-powered node in the middle of a workflow is unfamiliar, this build walks through one end to end.

If you're setting this up for an organization and want a second set of eyes on the build, that's worth a conversation.