The RAG generator.

Point it at a folder, an upload or a Google Drive folder. Watch the engine compile it. Then search the result and export it. A local, single-user workbench that turns "we have a shared drive full of documents" into a retrievable corpus you can interrogate before you trust it.

What it is

Everything the generator does is available through the engine API: submit a batch to /v1/jobs, poll it, read /v1/jobs/{id}/documents/{doc}/chunks, query /v1/search. What the generator adds is the loop around them, so the whole thing is something you can watch and judge rather than a sequence of curl commands whose output you have to hold in your head.

It is deliberately small and deliberately local: one browser tab, one user, no accounts, no database of its own beyond a JSON file recording which documents came from which folder.

Figure: the workbench on the Search tab. Every hit carries the file, page and section it came from.

How the pieces fit

The generator is a client. It ships no parser, no models and no index; it calls an engine you already run. That means three things have to be alive, and the second one catches people out:

• api accepts uploads and answers status. In the standard topology it runs NAUTRIS_WORKERS: "0", so it does no parsing itself.
• worker does the actual extraction. An engine with no worker looks perfectly healthy and accepts your files, then every job sits at queued forever.
• a vector index, enabled by setting NAUTRIS_RAG_DB on both. Without it documents still compile, but /v1/search answers 409 and nothing is retrievable.

Diagram: the generator only ever talks to your engine, and the engine only ever talks to itself.

Run it with Compose

One file brings up the engine and the generator together. The generator image is private to your organization; the engine image is public, and the license is the gate.

# docker-compose.yml - engine + RAG generator, one command.
# Pinned versions, not :latest, so a redeploy cannot change what you tested.
services:
  postgres:
    image: postgres:16-alpine
    restart: unless-stopped
    environment:
      POSTGRES_USER: nautris
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?see .env}
      POSTGRES_DB: nautris
    volumes: ["pgdata:/var/lib/postgresql/data"]
    # api and worker wait for this, so it must report readiness honestly.
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U nautris -d nautris"]
      interval: 10s
      timeout: 5s
      retries: 5

  api:
    image: registry.nautris.com/nautris:${ENGINE_TAG:-0.9.20}
    restart: unless-stopped
    ports: ["8000:8000"]          # /ui and the API. Drop to expose nothing.
    environment:
      NAUTRIS_LICENSE: ${NAUTRIS_LICENSE:?see .env}
      NAUTRIS_DATABASE_URL: postgresql://nautris:${POSTGRES_PASSWORD}@postgres:5432/nautris
      NAUTRIS_RAG_DB: /data/lancedb   # without this, search answers 409
      NAUTRIS_WORKERS: "0"            # the api enqueues...
      NAUTRIS_BACKGROUND: "1"         # ...the workers process
      NAUTRIS_API_KEYS: "rag=${ENGINE_KEY:?see .env}"
    volumes: ["engine-data:/data"]
    read_only: true                 # writes go to the volume and /tmp only
    tmpfs: [/tmp]
    depends_on:
      postgres: { condition: service_healthy }

  # Scale the parsing tier:  docker compose up -d --scale worker=4
  worker:
    image: registry.nautris.com/nautris:${ENGINE_TAG:-0.9.20}
    restart: unless-stopped
    command: ["python", "-m", "nautris.worker"]
    environment:
      NAUTRIS_LICENSE: ${NAUTRIS_LICENSE:?see .env}
      NAUTRIS_DATABASE_URL: postgresql://nautris:${POSTGRES_PASSWORD}@postgres:5432/nautris
      NAUTRIS_RAG_DB: /data/lancedb   # must match the api
    volumes: ["engine-data:/data"]
    # Workers parse untrusted bytes: no privilege escalation, hard caps. A
    # single heavy document then kills only its own worker (exit 137) and
    # dead-letters, instead of taking the host down.
    security_opt: ["no-new-privileges:true"]
    mem_limit: ${WORKER_MEM_LIMIT:-8g}
    pids_limit: 256
    depends_on:
      postgres: { condition: service_healthy }

  rag:
    image: registry.nautris.com/nautris-rag-generator:${RAG_TAG:-0.5.2}
    restart: unless-stopped
    ports: ["3100:3100"]
    environment:
      # Fetched by the generator's SERVER, not your browser, so it must resolve
      # INSIDE this network. localhost would be the container itself.
      NAUTRIS_URL: http://api:8000
      RAG_DATA_DIR: /data
    volumes:
      - rag-data:/data              # projects, settings, uploaded workspaces
      # The folder you want to compile. Read-only is enough: the generator
      # never writes to your documents. Scan it as /docs in the UI.
      - ${DOCS_DIR:?see .env}:/docs:ro
    depends_on:
      api: { condition: service_started }
      # Without a worker every job stays queued forever, so depend on it.
      worker: { condition: service_started }

volumes:
  pgdata:        # Postgres data. Losing it loses job history.
  engine-data:   # extracted trees, the vector index, cached models
  rag-data:      # the generator's own project store and settings

# .env - sits next to docker-compose.yml. Compose reads it automatically,
# so it survives a new terminal and a reboot.

# ---- required: nothing starts without these --------------------------------

# Your signed license token. Free and instant at https://get.nautris.com
# The engine refuses to start without one.
NAUTRIS_LICENSE=nl1....

# The folder to compile, as an ABSOLUTE path on the host. Mounted read-only at
# /docs, so you scan "/docs" in the generator, never the host path.
DOCS_DIR=/srv/documents

# Postgres password. You invent it. Used twice: it sets the database password
# AND is interpolated into the engine's connection string, so one value keeps
# all three services consistent.
# Applied only when the pgdata volume is FIRST created; changing it later needs
# `docker compose down -v` or the api can no longer authenticate.
# Avoid @ : / # or generate it: openssl rand -hex 24
POSTGRES_PASSWORD=change-me

# The engine API key. You invent it. Registered as NAUTRIS_API_KEYS="rag=<this>"
# and pasted into the generator's Settings page.
ENGINE_KEY=change-me

# ---- optional: sensible defaults, override only if you mean it -------------

# Image versions. Pinned rather than :latest so a redeploy cannot change what
# you tested. These are the current releases; the minimum that works is engine
# 0.9.13 (the first with DELETE /v1/documents/{id}, which the generator needs to
# retire superseded documents on a re-scan) and generator 0.4.0.
ENGINE_TAG=0.9.20
RAG_TAG=0.5.2

# Per-worker memory ceiling. The full pipeline peaks past 4 GB on large or
# OCR-heavy PDFs. Must be <= the memory available to Docker.
WORKER_MEM_LIMIT=8g

Every variable, explained

Which folders it can see

This is the single most common source of confusion, and it has a one-sentence rule: the generator scans its own filesystem. It does not upload from your browser and it cannot reach your engine's disk.

So in a container it sees exactly what you mount. Mount read-only; it never writes to your documents:

    volumes:
      - /srv/documents:/docs:ro     # then scan  /docs
      - /home/you/reports:/reports:ro  # then scan  /reports

If you would rather type any path on your machine with no mounting at all, run the generator directly instead of in a container, and point it at an engine that publishes a port:

npm install && npm run start     # http://localhost:3000
# engine URL: http://localhost:8000  (add  ports: ["8000:8000"]  to the api service)

The four steps

1. Connect

Enter the engine URL and an API key. The generator reports back the plan, days remaining, page quota position, queue depth, whether a vector index exists, and whether folder linking is included in your license. Everything that changes what you can do next is stated here rather than discovered as a failure three steps later.

The key is used server-side and kept in session storage only, so it disappears when you close the tab.

2. Documents

Pick a source, then read the report. Whichever source you use, the scan types every file by content, using the same magic-byte routing the engine applies at intake. A PDF named .txt is reported as a PDF, because that is what the engine will make of it.

Spreadsheets and CSV become tables, not prose. .xlsx and, from engine 0.9.14, .csv are compiled into table blocks, so the row and column relationship survives into the chunks. That matters more than it sounds: a price list flattened into a wall of numbers retrieves perfectly well and answers wrongly, because the figure ends up next to the wrong line item. CSV detection is deliberately conservative, since the format has no signature: prose that happens to contain regular commas is left as text rather than risking a mangled table.

Nothing reaches the engine yet, so this is the cheap moment to discover that half the selection is archives. Files are set aside with a reason:

Hidden folders, caches and dependency directories are skipped, and symlinks are never followed.

3. Compile

Files go up in batches of 200, the engine's own limit, and each document is reported as it settles: pages, blocks, chunks, how long it took, any flags raised for review, and the engine's exact reason for a failure. Stopping mid-run keeps everything already compiled, and re-running is safe because documents are keyed by content rather than by filename.

4. The RAG

Four tabs. Search opens first, because "is this any good?" is the question you actually arrive with. Chunks lists every indexed chunk with its provenance and a redaction toggle. Overview shows the shape of the corpus, review flags, and the documents that failed. Export takes it elsewhere.

Where documents come from

Three sources, one contract: each produces the same scan, so everything after it (compile, search, export) is identical. A source is a way of getting files onto the machine, never a separate flow.

Uploading folders and archives

Drag a folder and its subfolders come with it, structure preserved. You can also use the folder picker, drop one or more .zip archives, or hand over loose files. Uploads are sent in bounded batches with a progress bar, so a large folder is predictable rather than one enormous request.

Archives are extracted on the server, and treated as hostile input, because an archive is the classic way to attack an extractor:

• Entry paths are sanitized, so a ../../ entry cannot write outside the workspace.
• Symbolic-link and encrypted entries are skipped and reported, never silently dropped.
• Entry-count and total-size budgets bound any single extraction.
• A zip inside a zip is extracted, but only one level deeper: a bomb chain stops there, and anything left unextracted is reported and shows in the scan.

The extraction report appears above the scan results, so the numbers before and after always reconcile.

Settings

The gear icon in the header, or /settings. Stored server-side in RAG_DATA_DIR/settings.json, so choices hold across browsers and restarts rather than living in one browser's storage.

Data is worth knowing precisely, because deletion means different things per source. Deleting an uploaded or Drive project removes its stored files and frees the disk. Deleting a server-path project removes only the generator's record: your folder is never touched. Workspaces orphaned by a cancelled upload are listed separately and can be swept, so disk cannot leak invisibly.

Google Drive

Read-only import of a Drive folder tree. Google does not permit shipping shared OAuth credentials inside self-hosted software, so the connection uses your own OAuth client. It is a one-time setup of about five minutes:

1. In Google Cloud Console, create or pick a project.
2. APIs & Services > Library: enable the Google Drive API.
3. OAuth consent screen: External, and add yourself as a test user.
4. Credentials > Create credentials > OAuth client ID, type Web application. Register the redirect URI shown in Settings > Integrations.
5. Paste the client id and secret into Settings, save, then Connect Google Drive.

Once connected, the Drive source browses My Drive with a breadcrumb trail and previews what an import will do with each file before you start it. Importing takes the whole tree, subfolders included.

Google-native documents cannot be downloaded, only exported, so they are exported to the formats the engine reads natively:

Types with no useful export (Forms, Sites) are skipped with a reason, and Google caps exports at roughly 10 MB, which is reported as a per-file skip rather than a failed import. Everything else downloads as-is.

The client secret and the OAuth tokens are write-only through the generator's API: they live in its data directory and are never sent to the browser. Disconnecting revokes the grant at Google and clears them locally.

Validating retrieval

The point of the Search tab is to let you disbelieve the corpus productively. Ask something you already know the answer to. If the right passage comes back with the right page, the corpus is sound. If it does not, two things usually explain why.

The mode matters. Semantic search finds meaning and tolerates different wording; keyword (BM25) finds exact terms, names, codes and identifiers that an embedding will happily miss; hybrid fuses both and is the best default. A term that returns nothing under one mode is often the top hit under another.

Provenance tells you where a hit came from. Every result carries its source file, page, section breadcrumb, the block ids it was built from and, for formats that have geometry, the bounding boxes. When retrieval surfaces something odd, that trail usually shows whether the problem is the query, the chunking, or the document itself.

Re-scanning a folder

Scan a folder you have already compiled and the generator diffs it instead of rebuilding the RAG. Only new and edited files are sent, and the documents they replace are removed from the index, so the RAG matches the folder as it is now:

Re-sync: 2 new, 3 changed, 412 unchanged (skipped), 1 removed
Retired 4 superseded document(s).

The second half of that is the part worth understanding. The engine keys documents by content, so an edited file becomes a different document rather than a new version of the old one. Compiling it again indexes the new text but leaves the previous version searchable, and queries return both. The generator therefore un-indexes the superseded document as well, and it does so only after the replacement is compiled: if a run is interrupted, having both versions is recoverable by re-syncing, whereas deleting first and then failing to compile would lose the content outright.

Change detection is deliberately cheap. A file whose size and modification time both match what was compiled is never re-read. Anything else is hashed, and the bytes decide, so a file that was merely touched (a git checkout, an rsync -a) is correctly recognised as unchanged instead of being recompiled.

• Needs engine 0.9.13+, which added single-document deletion. Against an older engine the compile still succeeds and the superseded documents simply stay searchable until the next re-sync.
• Projects compiled before generator 0.4.0 carry no per-file identity, so they cannot be diffed. They recompile once, say so in the log, and are incremental from then on.
• This works on every plan and on every source, including uploads and Drive. Linked folders below are the paid alternative where the engine does the delta itself.

Linked folders

By default the generator uploads, and a re-scan is already incremental (see re-scanning a folder). If your license includes the sources feature, it can instead register the folder with the engine and let the engine pull it, so the engine itself tracks what changed:

new: 1   changed: 1   removed: 1   unchanged: 11

Two constraints, both surfaced in the interface rather than left to fail:

• The feature is part of trial, pro and enterprise. On personal and starter the engine answers 402 and the option is hidden. Uploading works on every plan.
• The path is resolved by the engine, not the generator. If the engine runs in a container, the folder must be mounted into it at the same path.

Exporting the RAG

A RAG you cannot move is a RAG you do not own. Everything is regenerated from the engine's stored trees at the moment you ask, using the same deterministic chunker that filled the index, so the same project always exports the same bytes.

The bundle is self-describing

From generator 0.5.0 the zip carries two files whose only job is to make the other three usable by someone who has never seen nautris. A bare chunks.jsonl forces the recipient to reverse-engineer the payload before they can act on it, which is how good corpora end up unused.

manifest.json declares a spec version, so a consumer can refuse a bundle it does not understand rather than guess:

{
  "bundle": "nautris-rag-bundle/v1",
  "kind": "rag-ready-corpus",
  "files": { "chunks.jsonl": "One chunk per line. The retrievable unit...", ... },
  "chunk_schema": {
    "provenance.section_path": "Heading trail, e.g. ['Contrat','5. Maintenance']...",
    "provenance.pages": "Source page numbers this chunk covers. Cite these.",
    ...
  },
  "embeddings": { "included": false, "reason": "Chunks ship as text so you choose your own model..." }
}

README.md is generated per project and opens with the distinction that matters: this is a RAG-ready corpus, not a running RAG. Your documents are parsed, structured, chunked and carry the provenance needed to cite them, which is the expensive half. You supply the embedding model, the store and the retrieval, and nothing in the bundle needs nautris at runtime.

Using the corpus in your application

The Integrate tab answers the question the Export tab leaves open. It gives complete, copy-ready code for four destinations, pre-filled with that project's own document and chunk counts, because a generic example gets read while one carrying your numbers gets pasted and run.

Every snippet carries source_file, pages and section_path through to the destination. Those fields cost nothing to store and cannot be rebuilt afterwards, and they are the difference between an answer that cites "contrat.pdf, section 5.2, page 3" and one that says "according to my documents".

Troubleshooting