Backup and Disaster Recovery for an AI Server

A typical AI server backup is small, because most of the disk is not yours. Take an office server holding three models of about 5 GB, 20 GB and 40 GB, a vector index of 200,000 chunks, a chat interface with a year of history, and a folder of prompts and config files. The models add up to 65 GB. The vectors, at 1,024 dimensions stored as 4-byte floats, come to 200,000 x 1,024 x 4 = about 820 MB before index overhead. Prompts and configs are a few megabytes. Close to 99% of the bytes (65 GB of about 66 GB) are public files that can be downloaded again, and the small remainder that cannot be replaced is what the backup plan exists to protect.

Sort everything into three piles

Irreplaceable, back up every day:

• Configuration: runtime settings, service unit files, reverse proxy config, environment files, firewall rules.
• Prompts and system messages, Modelfiles, templates and the version manifest that pins each model.
• Fine-tuned weights and LoRA adapters. These came from your data and your training run; nobody else has a copy.
• Application databases: chat history, user accounts, uploaded documents, feedback ratings, audit logs.
• The test set used to evaluate models, with its expected answers.
• Secrets: API keys, signing keys, TLS certificates. Store these in the backup but encrypted and access-controlled separately.

Rebuildable, but backing up is cheaper than rebuilding:

• The vector index. It can be regenerated from the source documents, but only if the documents, the chunker settings and the exact embedding model are all still available, and only at the cost of re-embedding everything.

Re-downloadable, record instead of copying:

• Base model weights, provided the version is pinned and the source still hosts it.
• The operating system, GPU drivers, container images and the inference runtime, provided their versions are written down.

The third pile is where plans quietly fail. "Re-downloadable" means the exact same file can be fetched again. A tag like :latest or a Hugging Face main branch can point at different weights next month, so a restore that pulls by name may bring back a different model than the one you tested. Recording a digest or full commit hash for every model, as covered in how to pin and swap model versions safely, turns the third pile from a hope into a procedure.

What to copy, and how

Configs and prompts

Keep them in a git repository and push it to a second machine. Anything changed by hand on the server and never committed is what gets lost.

Databases

Back up a database with its own tool, not by copying its files while it runs. The PostgreSQL documentation describes pg_dump output as "internally consistent", a snapshot of the database at the moment the dump began, and notes it does not block normal reads and writes. If the vector index lives in Postgres through pgvector, the same dump carries the vectors with no extra step.

Two restore details catch people out. A plain pg_dump does not include roles, so the users that own objects must exist before the restore, and pg_dumpall --globals-only captures them. And a custom-format dump (-Fc) is restored with pg_restore, not psql.

A separate vector database

A dedicated engine has its own procedure. Qdrant, for example, creates a per-collection snapshot through POST /collections/{name}/snapshots, and its docs state that a snapshot can only be restored to a Qdrant instance on the same minor version or the next one. That makes the engine's version part of the backup record: a snapshot taken today may not load into whatever version you install after a disaster a year from now.

The chat interface

Open WebUI keeps its state in one data directory: an SQLite database (webui.db), an uploads/ folder, a vector_db/ folder and a cache. Its backup guide notes that the admin panel's database download is not the recommended way to back up, and its sample script for SQLite and ChromaDB leaves out uploads/, so check that uploads are covered by some job. The details of stopping the container and keeping the secret key alongside are in running Open WebUI for a team.

Fine-tuned weights

Copy the adapter or merged model files, plus the training data, the training config and the base model's pinned revision. An adapter without the exact base model it was trained against is not usable, so treat the pair as one item.

Model files you might not be able to fetch

For a connected office, copying 65 GB of public weights nightly wastes space, with two exceptions. A site with no internet access cannot re-download anything, and every model must sit on the backup media; this is the normal case for an air-gapped AI installation. And a model the publisher may withdraw or relicense is worth one archived copy, not nightly ones, since its files never change once pinned.

Know where the files live. Ollama's FAQ gives the default locations, /usr/share/ollama/.ollama/models on a Linux install and ~/.ollama/models on macOS, with OLLAMA_MODELS to move them. The Hugging Face cache defaults to ~/.cache/huggingface/hub and places a CACHEDIR.TAG file there, which tells backup tools such as Borg, restic and rsync that the folder is re-downloadable and can be skipped.

Decide how much loss and downtime you accept

Two numbers set the schedule. The recovery point is how much recent work you can lose: a nightly backup means up to a day of chat history and uploads. The recovery time is how long the service can be down before it hurts.

Estimate the recovery time by adding the steps, with your own figures. Re-downloading 65 GB of models over a 100 Mbps connection takes 65 x 8,000 / 100 = 5,200 seconds, about 87 minutes, and longer if the source throttles. Rebuilding the vector index instead of restoring it means embedding every chunk again: at a measured rate of, say, 100 chunks per second on your hardware, 200,000 chunks take 2,000 seconds, about 33 minutes, plus re-reading and re-chunking all the source files. Restoring an 820 MB snapshot from local disk takes minutes. These numbers decide whether a local copy of the models and a nightly index snapshot are worth the disk.

Store copies in at least two places, one of them off the machine and ideally out of the building. The common 3-2-1 rule, three copies on two kinds of media with one off-site, fits. For sensitive data, encrypt the off-site copy, because a backup of chat history and indexed documents holds the same confidential content as the server.

Restore in dependency order

A restore fails when a step needs something that is not back yet. A workable order:

1. Hardware, OS and GPU driver, at the recorded versions. Confirm the GPU is visible before going further.
2. Container runtime and inference server, at the pinned versions. A newer runtime may reject an older model format or load it with different defaults.
3. Model files. Restore from the local copy or download by pinned digest or commit hash, then verify. The hf cache verify command checks cached Hugging Face files against the Hub's checksums, and Ollama's /api/tags endpoint returns each installed model's SHA256 digest to compare with the one recorded.
4. Fine-tuned adapters, placed next to the exact base model they belong to.
5. Databases: roles first, then data, then the vector index or snapshot. Check that the vector engine version accepts the snapshot.
6. Configuration and secrets from the git repository and secure store: prompts, Modelfiles, proxy, certificates.
7. Applications and the chat interface, pointed at the restored services.
8. Smoke test: sign in as an ordinary user, ask a question whose answer comes from a known document, and run the model test set to confirm answers match what the old server gave.

A spare-hardware plan

A failed GPU or power supply can stop service, and a replacement GPU can take days or weeks to arrive. Three options, from cheapest to most resilient:

Option	What it costs	What happens on failure
No spare	Nothing up front	Down until a replacement ships; possibly weeks
Degraded fallback	A smaller card or an existing workstation	A smaller model serves the same API at lower quality until repair
Cold spare server	A second machine, usually idle	Restore onto it within hours, at full quality

The degraded fallback is often the sensible middle. If applications call a model alias rather than a model name, a workstation with a smaller card can serve a smaller model under the same alias, and staff keep working with slower, weaker answers instead of none. Decide in advance which tasks pause in degraded mode, such as extraction jobs tuned for the large model. A cloud API as fallback is acceptable only for data already allowed to leave the building.

Test the restore, on schedule

A backup that has never been restored is a guess. Once a quarter, and after any major change, restore onto a spare machine or a virtual machine with no access to production:

• Time each step and compare the total with the recovery time you promised.
• Run the full model test set against the restored server and compare scores with production. A drop points to a wrong model revision, a missing adapter or a changed prompt.
• Ask questions that should hit recently added documents. If they fail, the index restore is older than the database restore, which the checks for keeping a RAG index in sync will catch on the next run, but which users would notice first.

The decision rule is short: if a restore test has not passed in the last 90 days, treat the server as having no backup until one does.

Frequently asked questions

Do embedding models need the same pinning as the chat model?

Yes, and arguably more. Vectors from one embedding model cannot be searched with queries embedded by another, so a restore that fetches a different embedding version returns poor results without any error. Record the embedding model's exact revision alongside the index snapshot.

How long should old backups be kept?

Long enough to recover from a problem noticed late, such as a corrupted index or a bad prompt change, which argues for weeks of daily copies plus a few monthly ones. Retention also sets how long deleted chats and documents survive, so align it with the organization's data retention policy.

Should GPU driver installers be kept in the backup?

For an air-gapped site, yes, along with the runtime packages and container images at their pinned versions. For a connected site, recording the exact versions is usually enough, though keeping the installers avoids a surprise if an old version is withdrawn from the vendor's download page.