Skip to main content

Troubleshooting

Problem: VCAL_KEY_APP not set or no API keys loaded​

Authentication is enabled but no app/admin keys were configured.

Fix: configure keys directly or through files:

VCAL_AUTH_REQUIRED=1
VCAL_KEYS_APP=app_key_1,app_key_2
VCAL_KEYS_ADMIN=admin_key_1

or:

VCAL_KEYS_APP_FILE=/opt/vcal-server/app.keys
VCAL_KEYS_ADMIN_FILE=/opt/vcal-server/admin.keys

Restart the server after changing keys.

Problem: /readyz returns 401 Unauthorized​

When VCAL_AUTH_REQUIRED=1, /readyz requires a valid key.

curl -i   -H "X-VCAL-Key: <app_key>"   http://127.0.0.1:8080/readyz

/healthz remains public.

Problem: License file exists on the host but server says no license was provided​

The path configured in VCAL_LICENSE_PATH or VCAL_LICENSE_FILE must exist inside the container.

Check the effective environment:

docker inspect vcal-server --format '{{range .Config.Env}}{{println .}}{{end}}' | grep VCAL_LICENSE

Avoid conflicting values such as:

VCAL_LICENSE_FILE=/etc/vcal/annual-license.json
VCAL_LICENSE_PATH=/var/lib/vcal/license.json

Mount the real host file into the same container path you configure:

-v /etc/vcal/annual-license.json:/etc/vcal/annual-license.json:ro -e VCAL_LICENSE_PATH=/etc/vcal/annual-license.json

Problem: Snapshot not found​

This is normal on first startup with an empty data directory:

WARN Snapshot not found at '/var/lib/vcal/data/vcal.index'; building fresh index with dims=768

Make sure VCAL_DIMS is set when no snapshot exists.

Problem: missing field format while loading vcal.index​

This usually means the server found an older snapshot file that is not in the expected Envelope v1 format.

Example:

Error: parsing vcal.index as SnapshotV1
Caused by: missing field `format`

Fix for non-critical/local data: archive the old snapshot and start fresh.

mkdir -p /opt/vcal-data/archive-before-v0.6.0
mv /opt/vcal-data/data/vcal.index /opt/vcal-data/archive-before-v0.6.0/
mv /opt/vcal-data/data/answers.json /opt/vcal-data/archive-before-v0.6.0/ 2>/dev/null || true
mv /opt/vcal-data/data/tombstones.json /opt/vcal-data/archive-before-v0.6.0/ 2>/dev/null || true

For important customer data, do not delete files. Archive them and perform a controlled migration or compatibility load.

Problem: Autosave not working​

Make sure autosave is enabled and the data directory is writable:

VCAL_AUTOSAVE_SECS=3600
VCAL_AUTOSAVE_ATOMIC=1

v0.6.0 background autosave logs look like:

INFO background paired snapshot completed bytes=4929 entries=1

Shutdown final save logs look like:

INFO quiescent paired snapshot completed pause_ms=3 bytes=4929 entries=1
INFO Final snapshots saved successfully (paired).

Problem: Deleted entry appears in logs as entries=1​

The snapshot log may count raw index entries, while tombstones prevent deleted IDs from being returned.

Verify active behavior with search or QA:

curl -s   -H "X-VCAL-Key: <app_key>"   -H "Content-Type: application/json"   -d '{"query":[0.001,0.001],"k":1}'   http://localhost:8080/v1/search

If the response is {"hits":[]}, the deleted item is not active.

Problem: Prometheus shows no metrics​

Check the endpoint:

curl http://localhost:8080/metrics | head

If authentication is enabled and metrics should be public, set:

VCAL_ALLOW_METRICS_WITHOUT_AUTH=1

Problem: High memory usage​

Check and adjust capacity settings:

VCAL_CAP_MAX_BYTES=1073741824
VCAL_CAP_MAX_VECTORS=50000

Do not set VCAL_CAP_MAX_BYTES higher than the available container or VM memory. For example, avoid an 8 GiB cache cap in a container limited to 2 GiB.

Problem: API returns 403 Forbidden​

Possible causes:

  • wrong app/admin key,
  • expired or invalid license,
  • using an app key for an admin operation.

Check license status with an admin key:

curl -H "X-VCAL-Key: <admin_key>"   http://localhost:8080/v1/license/status

Problem: Cache hit ratio remains low​

Possible causes:

  • embedding model mismatch,
  • VCAL_DIMS differs from the vector dimension,
  • TTL is too short,
  • similarity threshold is too high,
  • workload has little repetition.

Monitor:

vcal_cache_hits_total
vcal_cache_misses_total
vcal_search_requests_total

Logs and Diagnostics​

Docker:

docker logs --tail=100 vcal-server

Binary:

RUST_LOG=info ./vcal-server

Verbose debugging:

RUST_LOG=debug RUST_BACKTRACE=full ./vcal-server

Still stuck?​

  1. Check container environment variables with docker inspect.
  2. Check file mounts and permissions with ls -l on the host and inside the container.
  3. Check /metrics for errors and license state.
  4. Archive old snapshots before starting fresh.
  5. Contact support with logs, version, Docker command, and data directory listing.