How AI checks your passport photo (and why human reviewers miss things)

A 2019 NIST study on face-recognition algorithms found that the variance between two human reviewers grading the same passport photo is around 11% — meaning if you submitted an identical photo to two different consulate clerks on the same day, there’s roughly a one-in-ten chance one would accept it and the other would reject it. The same study found algorithmic checks reduced inter-rater variance to under 1%.

That gap is the entire reason an automated compliance check exists before you submit. A human reviewer is fast but inconsistent. An algorithm is consistent but limited to objective rules. The trick is to use both: let the algorithm clear every measurable rule, and reserve human judgment for the small set of genuinely subjective calls.

This post explains what the in-browser editor actually does to your photo when you upload it — the models, the measurements, the 22 checks — and where AI is genuinely better than a human reviewer, where it’s worse, and what that means for whether you should trust the “passes” verdict before you submit.

What happens when you upload a photo

The editor runs entirely in your browser. Nothing is uploaded to a server, which matters both for privacy (your face never leaves your device) and for speed (no round-trip latency). The pipeline has four stages:

1. Face detection and landmarking — find the face, identify 478 anatomical landmarks on it.
2. Person segmentation — separate the person from the background at pixel level.
3. Region statistics — measure lighting, sharpness, colour, and geometry across face, hair, and background regions.
4. Rule validation — run 22 grouped checks against the document’s spec and return a pass/fail with reasons.

Each stage uses a specific model or technique. Let’s walk through them.

Stage 1: 478-point face mapping

The editor uses Google’s MediaPipe Face Landmarker, a model that identifies 478 specific points on a human face — corners of the eyes, edge of the lips, tip of the nose, jaw line, eyebrow arc, ear position, forehead hairline. The 478-point map is dense enough to measure:

• Eye-line tilt — the angle of the line between the two pupil centres relative to horizontal. The spec says under 5°; the model measures it to within 0.1°.
• Head yaw — left-right rotation of the head, measured from the asymmetry between left and right ear visibility and the position of the nose tip relative to the face midline.
• Head pitch — up-down tilt, measured from the vertical position of the eyes within the face oval.
• Mouth openness — gap between upper and lower lip landmarks. Closed mouths register 0–3px; smiling-with-teeth registers 8px+.
• Eye openness — distance between upper and lower eyelid landmarks. Below a threshold and the eyes are classified as half-closed.
• Iris centring — position of the iris inside the eye opening. Drifting more than 15% off centre triggers a “looking aside” flag.

A human reviewer eyeballs all of these. The algorithm measures them in pixels. That’s the consistency advantage — the algorithm doesn’t have a “good day” or a “bad day”.

What 478 points means in practice

The dense landmarking is what lets the editor catch the subtle stuff that human reviewers miss. A head turned 6° to the right looks straight to the human eye but produces a measurable nose-to-midline offset and ear-visibility asymmetry. The algorithm flags it; the human passes it; the biometric system at the border later fails to match. Better to catch it before you submit.

Stage 2: Person segmentation

Background compliance is the single most common rejection reason (see the 22-reasons post), so the editor uses a dedicated person-segmentation model — specifically a lightweight U-Net variant trained on the COCO-person-segmentation dataset and adapted for portrait framing.

The output is a per-pixel mask: 1 if the pixel belongs to the person, 0 if it belongs to the background. The mask is used for three things:

• Background analysis — collect every pixel marked as background and run colour statistics on it. Mean RGB, standard deviation, hue distribution. A truly plain white background has very low standard deviation (under 5 on each channel) and a near-grey hue. A textured wall, a window edge, or a shadow shows up as elevated standard deviation.
• Background replacement — if the photo fails the background check, the mask is used to composite the person onto a clean white or off-white background.
• Shadow detection — a shadow cast on the background appears as a localised dark region in the background mask. The editor detects it by comparing brightness in the background pixels around the head to the rest of the background.

The segmentation model is good but not perfect — fine hair strands and soft jaw lines are the hard cases. The editor uses an alpha-feathered mask at the boundary to avoid the “cut-out paper doll” look that amateur background-removal produces.

Stage 3: Region statistics

Once the face is landmarked and the person is segmented, the editor computes statistics across specific regions:

• Face brightness: mean luminance of the skin pixels. Compared against the background brightness to detect underexposed faces (face much darker than background) or overexposed faces (face washed out).
• Lighting symmetry: ratio of brightness on the left half of the face to the right half. A symmetric ratio (close to 1.0) means even lighting; a ratio above 1.2 means a strong side-light is shadowing one half of the face.
• Skin smoothness: high-frequency texture content in the cheek and forehead regions. A natural unedited face has measurable texture (pores, micro-shadows). A photo run through a beauty filter has unnaturally low high-frequency content — that’s the signal the filter detector uses.
• Sharpness: Laplacian variance in the face region. A sharp, in-focus face produces a high value; a blurry one (motion blur, out-of-focus, portrait-mode edge blur) produces a low value.
• Eye sharpness specifically: measured separately because eyes must be sharp even when other regions can tolerate slight softness.

Each statistic is a number. Each number maps to a rule. Each rule maps to a check. That’s where the 22 checks come from.

Stage 4: The 22 grouped checks

The editor’s final stage takes the measurements from stages 1–3 and runs them against the target document’s spec (US passport, UK passport, Schengen visa, DV Lottery, and so on — each has its own thresholds). The checks are grouped:

• Background: plain, correct colour, no shadows, no objects.
• Face geometry: head size in frame, head position, eye height, head tilt, head turn.
• Expression: mouth closed, eyes open, looking at camera, neutral expression.
• Lighting: face exposure, lighting symmetry, no harsh shadows.
• Quality: sharpness, no filters, no red-eye, no compression artefacts.
• Accessories: no glasses (unless allowed), no hat, hair off face.
• Technical: correct aspect ratio, correct pixel dimensions, correct file size, correct format.

Each check returns pass, warning, or fail. The composite verdict is: all checks pass = green. Any check fails = red with the specific reason. Warnings (close to a threshold) don’t fail the photo but get flagged so you can decide.

Where AI beats human reviewers

There are five categories where the algorithm is genuinely more reliable than a human:

1. Geometric measurements. Head height in mm, eye line in degrees, crown-to-frame margin in percent. A human eyeballs it; the algorithm measures it.
2. Consistency across photos. A human reviewer who has been on shift for six hours is more lenient than they were at hour one. The algorithm doesn’t have shifts.
3. Subtle filter detection. Beauty filters that smooth skin or slim jaw lines are nearly invisible to a casual eye but the high-frequency content drops to detectable levels.
4. Background tone analysis. A wall that looks white but is actually warm-tinted from incandescent lighting reads as off-spec to the algorithm. A human reviewer will probably miss it under their own fluorescent office light.
5. Compliance with file-format rules. Pixel dimensions, file size, colour depth, JPEG vs PNG — these are mechanical checks and an algorithm runs them in milliseconds.

Where human reviewers beat AI

There are also categories where humans are better, and being honest about them is important:

1. Expression nuance. “Is this a neutral expression or a faint smile?” The mouth-open landmarks measure gap in pixels, but the underlying question — what counts as “natural and neutral” — is subjective. A human reviewer often has the right answer here even when the algorithm flags a borderline case.
2. Unusual faces. People with facial differences, scars, congenital conditions, or significant asymmetry can trip algorithmic checks that a human would correctly accept. The algorithm trained on typical faces sometimes misreads atypical ones.
3. Cultural and religious headwear. Whether a specific religious covering meets the country’s policy is a judgment call. The algorithm can measure forehead-to-chin visibility but not whether the covering qualifies as religious.
4. Context. A reviewer can tell if a slightly imperfect photo is “the best you’re going to get” for, say, an elderly relative who can’t easily retake. The algorithm just sees pass or fail.

What this means for you

The honest summary: if your photo passes the AI check, you have very high confidence the objective rules are met. Geometry, lighting, background, file format, accessories — all measured, all green.

What remains is the small surface area of subjective judgment: the exact line between a neutral expression and a faint smile, the interpretation of a religious headwear case, whether your face is “recent enough” given subtle aging. Those are the questions a human reviewer is genuinely better at.

So the workflow is:

1. Take the photo.
2. Run it through the editor to clear every objective rule.
3. If the editor flags anything subjective (borderline expression, borderline crop), make a judgment call and retake if you can.
4. Submit.

You’re not relying on AI to replace human review. You’re using it to clear the consistent, measurable, automatable failures — which is most of them — so the human reviewer only has to think about the genuinely hard cases.

Try the check before you submit

If you have a photo and you want to see what 22 checks look like running on it, drop it into the in-browser editor. It runs in your browser, takes about two seconds, and tells you specifically which rules pass, which fail, and which are borderline. For a list of every rejection reason and the fix for each one, see the 22-reasons guide. For country-specific specs the editor matches against, browse the country pages.