Skip to main content

Troubleshooting

Problem: VCAL_KEY_APP not set or no API keys loaded​

Authentication is enabled but no app/admin keys were configured.

Fix: configure keys directly or through files:

VCAL_AUTH_REQUIRED=1
VCAL_KEYS_APP=app_key_1,app_key_2
VCAL_KEYS_ADMIN=admin_key_1

or:

VCAL_KEYS_APP_FILE=/opt/vcal-server/app.keys
VCAL_KEYS_ADMIN_FILE=/opt/vcal-server/admin.keys

Restart the server after changing keys.

Problem: /readyz returns 401 Unauthorized​

When VCAL_AUTH_REQUIRED=1, /readyz requires a valid key.

curl -i   -H "X-VCAL-Key: <app_key>"   http://127.0.0.1:8084/readyz

/healthz remains public.

Problem: License file exists on the host but server says no license was provided​

The path configured in VCAL_LICENSE_PATH or VCAL_LICENSE_FILE must exist inside the container.

Check the effective environment:

docker inspect vcal-server --format '{{range .Config.Env}}{{println .}}{{end}}' | grep VCAL_LICENSE

Avoid conflicting values such as:

VCAL_LICENSE_FILE=/etc/vcal/annual-license.json
VCAL_LICENSE_PATH=/var/lib/vcal/license.json

Mount the real host file into the same container path you configure:

-v /etc/vcal/annual-license.json:/etc/vcal/annual-license.json:ro -e VCAL_LICENSE_PATH=/etc/vcal/annual-license.json

Problem: Snapshot not found​

This is normal on first startup with an empty data directory:

WARN Snapshot not found at '/var/lib/vcal/data/vcal.index'; building fresh index with dims=768

Make sure VCAL_DIMS is set when no snapshot exists.

Problem: Snapshot save fails with an I/O or permission error​

v0.6.1 validates and uses VCAL_DATA_DIR explicitly. Make sure the directory exists and is writable by the runtime user.

For a bind-mounted Docker directory, inspect the container runtime UID/GID and match the host directory ownership:

docker inspect vcal-server --format 'User={{.Config.User}}'
ls -ld ./vcal-data ./vcal-data/data

For images running as UID 10001, for example:

sudo chown -R 10001:10001 ./vcal-data

Do not assume that a directory writable by your host user is writable by the non-root container user.

Problem: missing field format while loading vcal.index​

This usually means the server found an older snapshot file that is not in the expected Envelope v1 format.

Example:

Error: parsing vcal.index as SnapshotV1
Caused by: missing field `format`

Fix for non-critical/local data: archive the old snapshot and start fresh.

mkdir -p /opt/vcal-data/archive-before-v0.6.1
mv /opt/vcal-data/data/vcal.index /opt/vcal-data/archive-before-v0.6.1/
mv /opt/vcal-data/data/answers.json /opt/vcal-data/archive-before-v0.6.1/ 2>/dev/null || true
mv /opt/vcal-data/data/tombstones.json /opt/vcal-data/archive-before-v0.6.1/ 2>/dev/null || true

For important customer data, do not delete files. Archive them and perform a controlled migration or compatibility load.

Problem: Autosave not working​

Make sure autosave is enabled and the data directory is writable:

VCAL_AUTOSAVE_SECS=3600
VCAL_AUTOSAVE_ATOMIC=1

Background autosave logs can look like:

INFO background paired snapshot completed bytes=4929 entries=1

Shutdown final save logs look like:

INFO quiescent paired snapshot completed pause_ms=3 bytes=4929 entries=1
INFO Final snapshots saved successfully (paired).

Problem: Deleted entry appears in logs as entries=1​

The snapshot log may count raw index entries, while tombstones prevent deleted IDs from being returned.

Verify active behavior with search or QA:

curl -s   -H "X-VCAL-Key: <app_key>"   -H "Content-Type: application/json"   -d '{"query":[0.001,0.001],"k":1}'   http://localhost:8084/v1/search

If the response is {"hits":[]}, the deleted item is not active.

Problem: Prometheus shows no metrics​

Check the endpoint:

curl http://localhost:8084/metrics | head

If authentication is enabled and metrics should be public, set:

VCAL_ALLOW_METRICS_WITHOUT_AUTH=1

Problem: High memory usage​

Check and adjust capacity settings:

VCAL_CAP_MAX_BYTES=1073741824
VCAL_CAP_MAX_VECTORS=50000

Do not set VCAL_CAP_MAX_BYTES higher than the available container or VM memory. For example, avoid an 8 GiB cache cap in a container limited to 2 GiB.

Problem: API returns 403 Forbidden​

Possible causes:

  • wrong app/admin key,
  • expired or invalid license,
  • using an app key for an admin operation.

Check license status with an admin key:

curl -H "X-VCAL-Key: <admin_key>"   http://localhost:8084/v1/license/status

Problem: Cache hit ratio remains low​

Possible causes:

  • embedding model mismatch,
  • VCAL_DIMS differs from the vector dimension,
  • TTL is too short,
  • similarity threshold is too high,
  • workload has little repetition.

Monitor:

vcal_cache_hits_total
vcal_cache_misses_total
vcal_search_requests_total

Logs and Diagnostics​

Docker:

docker logs --tail=100 vcal-server

Binary:

RUST_LOG=info ./vcal-server

Verbose debugging:

RUST_LOG=debug RUST_BACKTRACE=full ./vcal-server

Still stuck?​

  1. Check container environment variables with docker inspect.
  2. Check file mounts and permissions with ls -l on the host and inside the container.
  3. Check /metrics for errors and license state.
  4. If the supplied artifact, license, or verification key appears incorrect or expired, contact VCAL with the deployment identifier and artifact version. Do not substitute files from an old public release or unofficial mirror.
  5. Archive old snapshots before starting fresh.
  6. Contact VCAL support with logs, version, Docker command, and data directory listing.