Skip to content

Project Structure

hope-dedup-engine/
├── src/hope_dedup_engine/
│   ├── apps/
│   │   ├── api/            # the REST API and core domain models
│   │   ├── faces/          # face pipeline: DeepFace/OFIQ services
│   │   ├── security/       # User, System, UserRole (multi-tenancy)
│   │   ├── core/           # base utilities, `demo` and `upgrade` commands
│   │   └── social/         # social-auth integration pieces
│   ├── config/             # settings, env definitions, celery app, urls
│   ├── web/                # non-API pages (index, healthcheck), templates
│   └── utils/              # shared helpers
├── tests/                  # pytest suite (mirrors the app layout)
│   └── extras/demoapp/     # demo compose stack, images, API scripts
├── docs/                   # this documentation (MkDocs)
├── docker/                 # production Dockerfile and entrypoint
└── compose.yml             # local development stack

The api app

The heart of the service:

  • models/deduplication.py — DeduplicationSetGroup, DeduplicationSet (with the State machine and VALID_TRANSITIONS), Encoding, Finding. Read this first; the data model and lifecycle pages describe it.
  • models/jobs.py — DedupJob / MainJob, Celery-backed job records (django-celery-boost).
  • models/auth.py — HDEToken (token bound to user + system).
  • views.py, serializers.py, urls.py, filters.py, pagination.py — the DRF API surface. OpenAPI schema is generated by drf-spectacular from @extend_schema annotations — keep them updated when changing endpoints.
  • deduplication/config.py — DeduplicationSetConfig, the single source of truth for tunable settings. Field metadata (api=True, ofiq_metric=...) drives both the API serializer and the quality checks; adding a new setting here propagates everywhere.
  • deduplication/process.py — the find_duplicates Celery task orchestrating the run, state transitions, notifications, and the audit log.
  • auth.py — token authentication and the api.can_use_api permission.
  • admin/ — admin panel customizations per model.
  • utils/notification.py — webhook delivery.

The faces app

The computer-vision layer, kept separate from the API:

  • services/facial.py — encoding (encode_faces) and vectorized duplicate search (dedupe_all, find_duplicate_pairs).
  • services/quality.py — OFIQ scoring and threshold evaluation.
  • celery_tasks.py — Celery tasks for the faces app.

The config package

  • __init__.py — the CONFIG dict declaring every environment variable (type, defaults, required flag, docs) for django-smart-env. Add new env vars here.
  • settings.py — Django settings, importing modular fragments/ (celery, constance, sentry, spectacular, …).
  • fragments/constance.py — global default settings shown in the admin; keep in sync with DeduplicationSetConfig.

Conventions

  • State changes go through DeduplicationSet.set_state(), which validates transitions against VALID_TRANSITIONS — don't assign state directly.
  • Settings scale: API and admin use the 0–1 range; internally DeepFace/OFIQ comparisons use 0–100. DeduplicationSetConfig handles the conversion — always go through it.
  • System scoping: every queryset in API views filters by request.auth.system. Any new endpoint must do the same.
  • Migrations live in apps/api/migrations/; generate with ./manage.py makemigrations api.