Index a large PDF for RAG with the AIVAX CLI: OCR page by page, then import

The first time I pointed the AIVAX CLI at a 142-page IRS guide, it wrote 659 chunks and an error file listing 63 pages it could not process. OCR had worked on every one of those pages. The second time, on a 336-page scanned book, one request was still running after 50 minutes. Both documents ended up fully searchable, each after changing one setting, and most of this post is about which setting and why.

The documents are public, so you can repeat the runs. The IRS Publication 17 is born-digital, 2.95 MB, with dense tax tables at the back. Mining Without Timber is a 1911 book scanned by the Internet Archive: 25.8 MB, every page an image. The goal for both was the same. Split the document into pieces small enough to retrieve, keep the page each piece came from, and put them in a collection an agent can search. A chat upload does not give you that; even Claude's limits stop visual analysis above 100 pages and suggest dividing bigger files yourself.

The pipeline is two commands

npm install -g @aivax/cli
aivax auth login

aivax ocr document irs-p17.pdf --format jsonl -o p17.jsonl
aivax rag import -c <COLLECTION_ID> -f p17.jsonl

aivax ocr document opens the PDF on your machine and splits it into one standalone PDF per page. It packs pages into requests of up to --chunk-size megabytes (5 by default), sends --concurrency requests at a time (4 by default) to the Fetch and OCR API, and, with --format jsonl, passes each page's Markdown through the segmentation endpoint. The output is one JSON line per chunk:

{"docid":"doc_irs-p17.pdf:pg27chk78e7c496","__ref":"rf_irs-p17.pdf:pg27","text":"...","__meta":{"file_name":"irs-p17.pdf","file_hash_sha1":"a043071538a5483ed971f74c5eebe3295f1535fe","file_hash_crc32":"464e2935","page":27,"pages":142,"page_chunk":1,"extracted_at":"2026-11-12T20:58:45.738Z"}}

aivax rag import streams that file to the collection over one WebSocket; the server commits every 300 lines and reports progress. Without --format jsonl, ocr document writes a single Markdown file with an HTML comment before each page, which is the easier format for reading the OCR next to the original before you index anything.

Splitting locally means a failure is scoped to the pages of one request, and the CLI writes those page numbers and the error to <output>.errors.txt and exits with code 1. The per-item limit on Fetch is 50 MB, so a whole 25 MB book would be accepted as one item; it would also succeed or fail as one item. Before splitting, the CLI tries to recompress large embedded images (to JPEG at quality 75, longest side 2,400 px) and strips each page's resource dictionary down to what that page uses. Even so, the standalone pages add up to more than the original: the 2.95 MB tax guide became four requests of up to 5 MB, holding 23 to 40 pages each.

The tax guide failed on its tables

On the first run, all 63 failed pages carried the same message: Segmentation failed: An external service is temporarily unavailable. A second run failed on the same 63. They were exactly two of the four requests, pages 80–119 and 120–142, which pointed at the segmentation step rather than at the pages one by one.

So I sent each page's Markdown to the segmentation endpoint on its own. Called one after another, eight pages failed every time, and all eight were pages of the 2025 tax table (114 to 120, and 122). I do not know what the segmenter objects to; what those pages share is that they are almost entirely numbers in pipe-delimited rows, and cutting one of them in half made the error go away. Called in parallel, fifteen other pages also returned 503 and then passed when retried alone, which is a separate, transient problem. The endpoint returns one error for the whole request when any of its documents fails, and that is why one bad page took 39 good ones down with it.

The fix is a request small enough to hold one page:

aivax ocr document irs-p17.pdf --format jsonl --chunk-size 0.1 -o p17.jsonl

At 0.1 MB, each of the 142 pages became its own request. That run took 1 minute 30 seconds, wrote 1,020 chunks from 134 pages, and named the eight tax-table pages as the only failures. For those eight I took the text from the plain Markdown run (aivax ocr document irs-p17.pdf -o p17.md, which skips segmentation and had all 142 pages) and wrote each page as one JSONL line with the same __ref and page metadata. The cost of that 0.1 MB run, as the CLI reported it, was 142 OCR processing units, covered by my plan's daily allowance, plus $0.024 for segmentation. Embedding the chunks is billed separately at the collection embedding rate.

The tax table was then in the collection, though not usefully. This search:

aivax rag search -c <COLLECTION_ID> -q "tax for taxable income of 15,000 married filing jointly" \
  --filter 'metadata.page >= 110' --include metadata

returned pages 125, 113 and 124, at scores between 0.52 and 0.63. The row I wanted is on page 115. A page of tax table is one embedding standing for a few hundred numbers, and similarity search cannot tell one bracket from the next. If people will ask about specific rows, I would parse the table into one document per row with my own script. Indexing it as pages is enough only when the table needs to be findable as a whole.

The scanned book was slow, not broken

The 1911 book never hit a segmentation error. With the defaults, the CLI packed its 336 pages into six requests. A born-digital page converts from its text layer almost at once; a scanned page goes through OCR, and Fetch works through the items of one request in order. Four of the five requests that finished, each about 60 pages, took between 12 and 15 minutes. The sixth, pages 1 to 75, was still running when I stopped it at 50 minutes. The CLI gives a long request 20 minutes and then retries it, so the log cannot tell me whether that request was slow or being retried.

Recompression did not help either. The log said Recompressed 0 images: the Internet Archive encodes its scans as JPEG 2000 and JBIG2, and the CLI re-encodes only JPEG and Flate images, so the 25.8 MB went up as it was.

Smaller requests, more of them in flight:

aivax ocr document mining.pdf --format jsonl --chunk-size 1 --concurrency 8 -o mining.jsonl

With 1 MB requests the book became 28 requests, most of them 11 to 13 pages, eight at a time. The whole book took 13 minutes 53 seconds and exited with code 0: 336 of 336 pages transcribed, 739 chunks, and 13 pages with no text at all, all among the first 20 pages and the last 7. The CLI counts pages without text separately from failures and writes no line for them. Importing the 739 lines took 32 seconds.

That run reported 8,847 OCR processing units, against 142 for the whole tax guide: roughly 26 per scanned page against one per digital page. That ratio is what I would budget with. My plan's allowance covered it; at the Fetch rates for uncovered pages it comes to about $1.33 on Free, $0.44 on Pro and $0.27 on Max. Segmentation added $0.018.

For a scanned PDF I would now start at --chunk-size 1. I chose --concurrency 8 without hitting a rate limit, so I cannot tell you where your account's ceiling is; the CLI makes up to five attempts per request, retrying 429 and 5xx responses four times with exponential backoff, before it marks a request's pages as failed.

A re-run skips what is already indexed

The docid the CLI writes is doc_<file name>:pg<page>chk<crc32 of the chunk text>, so two runs over the same file produce the same ID for every chunk whose text did not change. On the tax guide, the 659 chunks of the first and second runs matched one for one, and so did the pages that succeeded in both the 5 MB and the 0.1 MB runs.

The WebSocket import matches lines by docId: a new ID is created, the same ID with the same text is skipped, and the same text with different __meta updates only the metadata, without embedding again. Importing the 1,020-line file a second time returned "updated": 0, "skipped": 1020. Importing the 659-line file from the first run on top of it returned "updated": 659, because extracted_at differs between runs; all of those were metadata updates.

That is what makes the retry loop cheap: run, read the errors file, run again with a smaller --chunk-size or patch the stubborn pages, import again. Chunks already in the collection are not embedded a second time. If the collection should hold exactly the current file and nothing else, aivax rag import --sync deletes documents whose IDs are absent from the uploaded JSONL. It asks for confirmation first, and I would not use it on a collection that holds more than one source.

Every chunk carries page, pages and file_hash_sha1 in its metadata. The first result for "standard deduction for married filing jointly" came back as page 94 with that number attached, which is what an agent needs to cite its source, and a metadata filter can limit a search to a page range or to one version of the file.

When I would not use this path

For a PDF of a few dozen pages, the splitting and retry handling buy little. PDF to Markdown for RAG does the same job with two API calls and a short script, and every request stays in your code.

For a document with many unrelated topics, where you would rather have AIVAX write focused documents than keep the source wording, aivax rag import -f file.pdf without JSONL sends the file to Media Injector instead. That path generates new text, its daily file count depends on the plan, and processing fails for a document above 20 MB, so the 25.8 MB book would have needed splitting there too.

For PDFs that must not leave your infrastructure, run a local parser and use only aivax rag import. The JSONL format does not care what produced it.

Whatever produced the chunks, read some of them next to the original before you build on them; the ingestion checklist lists what to look for. On the tax guide, OCR kept tables as Markdown tables but merged multi-row column headers into one run-on line. On the 1911 book the prose came through readable, and the running header "MINING WITHOUT TIMBER" sits inside 139 of the 739 chunks; I would strip it before importing.

Note: every number above comes from runs of @aivax/cli 1.1 against the live AIVAX API on November 12, 2026, on a Max-plan account, using a temporary collection deleted afterward. OCR time and processing units vary from one document to another.