Skip to content

OCR processing flow

OCR is executed across Country Workspace (CW), RabbitMQ, and Hope Documents (HD).

RabbitMQ batches are transport only. Persistent RdpOperationUnit records are the source of truth for OCR completion.

Each eligible Individual has at most one OCR unit.


End-to-end flow

sequenceDiagram
    autonumber

    participant CW as Country Workspace
    participant DB as CW Database
    participant MQ as RabbitMQ
    participant Blob as HOPE blob storage
    participant HD as Hope Documents

    CW->>DB: Claim operation and increment attempt
    CW->>DB: Create or reset OCR units

    alt No eligible OCR documents
        CW->>DB: Operation -> SUCCESS
    else OCR units exist
        DB-->>CW: PENDING units ordered by PK

        Note over CW,DB: Retry preserves SUCCESS units<br/>and resets unresolved units to PENDING

        loop Each non-empty batch
            CW->>Blob: Upload document images
            CW->>CW: Build batch request with blob names
            Note right of CW: Same correlation_id for the attempt<br/>New batch_id for each batch
            CW->>MQ: hd.ocr.request
            MQ->>HD: Deliver request
            HD->>Blob: Read document images
            HD->>HD: Validate and process every document
            HD->>MQ: hcw.ocr.result
        end

        loop Results may arrive in any order
            MQ->>CW: hcw.ocr.result
            CW->>CW: Validate result

            alt Stale attempt / wrong correlation_id
                CW->>CW: Ignore result
            else Current RUNNING attempt
                CW->>DB: Apply entries by individual_id
                CW->>DB: Derive outcome from all units

                alt Any PENDING
                    DB-->>CW: Keep RUNNING
                else Any FAILURE
                    DB-->>CW: Operation -> FAILURE
                else All SUCCESS
                    DB-->>CW: Operation -> SUCCESS
                    CW->>CW: Schedule push evaluation if all operations succeeded
                end
            end
        end
    end

Key rules:

  • batches are independent and results may arrive out of order;
  • CW applies results by individual_id, not batch order;
  • batch_index and batch_total are not used for completion;
  • no eligible OCR documents is a successful no-op;
  • an HD document result with status="error" creates an OCR finding, but its unit still becomes SUCCESS.

Batching and unit state

flowchart TB

    A["Load PENDING units"]
    B["Split into batches<br/>max OCR_BATCH_SIZE"]
    C["Prepare each unit"]
    D{"Document available?"}
    E["Unit -> FAILURE"]
    F["Add document to request"]
    G{"Prepared batch empty?"}
    H["Skip batch"]
    I["Publish hd.ocr.request"]
    P{"Definitely not sent?"}
    Q["Unit -> FAILURE"]
    J["Wait for hcw.ocr.result"]
    K["Apply result by individual_id"]
    L{"Any PENDING?"}
    M{"Any FAILURE?"}
    N["Operation -> FAILURE"]
    O["Operation -> SUCCESS"]

    A --> B --> C --> D
    D -->|No| E --> G
    D -->|Yes| F --> G
    G -->|Yes| H
    G -->|No| I --> P
    P -->|Yes| Q --> L
    P -->|No / unknown| J --> K --> L
    L -->|Yes| J
    L -->|No| M
    M -->|Yes| N
    M -->|No| O

Unit semantics:

PENDING  - result may still arrive
SUCCESS  - processing completed; duplicate results are ignored
FAILURE  - local unit work could not proceed

On retry, SUCCESS units are preserved and unresolved units are reset to PENDING.


Batch identifiers

Field Meaning
correlation_id One ID per OCR attempt, derived from operation.id and attempt.
batch_id New UUID per transport batch.
batch_index Informational batch number.
batch_total Calculated before document preparation.
individual_id Stable request/result identity.

batch_total may be larger than the number of published batches because unavailable documents can make a prepared batch empty. This does not affect completion.


Publish failures

CW publishes through django-streaming. The helper retries the same event once after resetting the streaming engine.

The publish outcome distinguishes:

  • SUCCESS - publishing completed without an observed failure. This is not publisher-confirmed delivery.
  • NOT_SENT - both attempts failed before RabbitMQ publishing started. The affected pending units are marked FAILURE.
  • UNKNOWN - at least one attempt may already have reached RabbitMQ. The affected units remain PENDING so a valid late result can still be applied.

Because publisher confirms are not used, an UNKNOWN outcome cannot determine whether the message was delivered or lost.

If a publish failure leaves PENDING units, the operation remains RUNNING and may still receive results. If no PENDING units remain, the operation fails.

Retrying the same event can also produce duplicate delivery; duplicate results are safe because already-SUCCESS units are ignored.


Manual retry

Manual retry means the operator considers the current attempt stuck.

RUNNING attempt N
    -> mark attempt N FAILURE
    -> claim operation again
    -> attempt N+1
    -> preserve SUCCESS units
    -> reset unresolved units to PENDING
    -> publish with a new correlation_id

Late results from attempt N are ignored because their correlation_id is stale.

If several operations are retryable, FAILURE operations are retried first; a still-RUNNING OCR operation is restarted only when no operation is currently FAILURE.


Document images

Images are not embedded in hd.ocr.request. As for biometric deduplication, CW uploads each document image to the shared HOPE blob container (FILE_STORAGE_HOPE, see Blob Storage) with services.hope_blob.sync_record_blobs and sends only the blob name:

{"individual_id": 12345, "filename": "2024/afghanistan/CP-2024/CW_ind_12345_national_id_image.png", "pattern": "AB123456"}

The upload is scoped to the document image field, so the other image blobs of the Individual are left untouched, and an unchanged image is not uploaded again. A unit whose image references a missing file is marked FAILURE. A storage failure fails the whole attempt, which can then be retried.

Hope Documents must point its own FILE_STORAGE_HOPE at the same container, with read access. A blob it cannot read or decode becomes a document result with status="error".

Hope Documents mapping

Hope Documents builds each result directly from the request batch and preserves the original individual_id:

for item in payload.get("documents") or []:
    outcome = process_document(
        item["filename"],
        item["pattern"],
        individual_id=item["individual_id"],
    )
    documents.append({"individual_id": item["individual_id"], **outcome})

It also preserves:

correlation_id
rdp_id
batch_id
batch_index
batch_total

So the core model is:

transport:  batches and RabbitMQ messages
state:      RdpOperationUnit records
identity:   individual_id
attempt:    correlation_id