Skip to content

Operations and Troubleshooting

Day-to-day tasks and diagnosing problems, mostly through the admin panel.

Inspecting a deduplication set

Home › Api › Deduplication sets › <set>

The set detail shows the current lifecycle state, the error field (populated when a run fails, with the traceback truncated), and the log — an append-only JSON audit of every processing run with timestamp, action (encode/deduplicate), the config snapshot used, counts of encodings processed and findings created, and any error.

Related admin pages: Encodings (per-image records with embedding status and cached quality scores), Findings (results), Deduplication set groups (settings, processing lock).

Re-running processing

If a run failed, or a set should be re-processed after changes:

  1. Navigate to Home › Api › Dedup jobs (or the jobs section on the set).
  2. Select the job for the deduplication set.
  3. Use the Queue button to re-queue it.

Alternatively the client system can simply call POST /deduplication_sets/{id}/process/ again — retry is allowed from the failed states.

Celery task execution can be monitored at:

Home › Celery Results › Task results

and, if deployed, in Flower (celery-flower service).

Cleaning up old findings and rejected sets

cleanup_redundant_data deletes data that is no longer needed once it is older than CLEANUP_RETENTION_DAYS (default 60 days, about two months; change it under Constance › Config):

  • findings on approved sets (HOPE has already read them and approved the set)
  • findings, encodings, and the set itself when the set is rejected

Encodings on approved sets are kept. Later runs in the same group compare new images against them.

Only database rows are removed. The image files stay in the shared HOPE storage, which the engine only reads from and never deletes from.

The task is not scheduled automatically. Turn it on from the admin:

  1. Open Home › Periodic Tasks › Periodic tasks › Add.
  2. Set the task to hope_dedup_engine.apps.api.celery_tasks.cleanup_redundant_data.
  3. Choose a schedule (for example, once a day) and enable the task.

Celery beat must be running for the schedule to fire. A one-off run can pass {"retention_days": 60} as keyword arguments to override the Constance value.

Releasing a stuck processing lock

Each group has a processing_locked flag preventing concurrent runs. The worker releases it in a finally block, so it should clear even on failure — but if a worker was killed hard (OOM, node eviction) the flag can remain set, and all process calls for the group will return 409. Check the group in Home › Api › Deduplication set groups and clear the flag if no task is actually running (verify in Task results / Flower first).

Common issues

Symptom Likely cause Fix
All findings have status 404 Wrong FILE_STORAGE_HOPE configuration, or image files missing from storage Verify the storage configuration and that the filenames sent to the API exist at those paths in the HOPE storage.
process always returns 409 "another task is already running" Stale processing lock See releasing a stuck lock.
Encoding fails instantly Missing model weights Check the DEEPFACE_HOME volume is mounted and writable (weights are auto-downloaded on first use).
Quality checks fail unexpectedly / OFIQ errors OFIQ model data missing Check OFIQ_DATA_DIR contents.
Webhooks not arriving at the client notify=false on the set, or client URL unreachable Check the set's notify/notification_url; delivery failures are reported to Sentry.
API token rejected (401/403) Token's user not linked to a System, or missing Can use api permission See creating API credentials.

Sentry

All unhandled processing errors, per-image generic errors, and webhook delivery failures are captured by Sentry when SENTRY_DSN is configured. The set's error field only holds a truncated message — the full traceback is in Sentry.