Keeping an AI Character's Face Consistent — Three Methods, Measured

An image model builds a picture from scratch every time, satisfying whatever the prompt asks for. Ask for "silver hair, hime cut, grey eyes" and you get that. But an unlimited number of faces satisfy those words.

Everything you did not specify is decided by the seed:

• Face shape
• How the eyes are drawn
• Lip weight
• Apparent age

For a one-off illustration that does not matter. For a service where the same character returns in video after video, it is fatal.

We changed exactly one variable and generated 20 images per condition.

Outfit, pose, framing and location were all pinned. Adding variables would make it impossible to tell whether drift came from the seed or from the settings.

First we described the character in tags: silver hair hime cut blunt bangs grey eyes slender expressionless.

The result was twenty different people sharing a haircut.

Eyes and lips drifted most. Seed variance flows into whatever you left unspecified, which means the more tags you write, the more concentrated the drift becomes in what remains.

So we named the drifting attributes explicitly: eye shape, brows, lip colour, face shape, framing.

• Framing held
• The face still drifted

It also backfired. Adding sharp narrow eyes and thin eyebrows pushed the model toward hollow, unsettling faces — worse looking than before we added the tags.

While iterating we also tried to pin the lighting, stacking four tags at once: dim indoor lighting soft shadows low key lighting night.

The output was almost entirely black — faces sunk into a blue-cast frame, and the comparison stopped being possible at all.

The original problem was the face, not the lighting. Touching things outside the actual problem creates new ones.

Next we transferred appearance from a reference image (IP-Adapter). We picked one image from the first batch and pinned it as the reference.

We tested three strengths, 8 images each.

0.7 was the sweet spot.

But holding up in the same scene and outfit proves nothing. What a service actually needs is the same character across changing scenes and clothes. So we kept the reference fixed and varied outfit and location across 8 settings:

• Office (suit)
• Living room (blouse)
• Summer festival (kimono)
• Mansion (evening dress)
• Bedroom (hoodie)
• Poolside (swimwear)
• Lingerie
• Oversized sweater

All 8 read as the same person.

The most reliable way to pin a character is to train on images of them (LoRA).

That requires two things:

1. Assembling a consistent training set
2. Standing up a training environment

Method 2 gave us enough consistency, so we skipped the investment.

For photoreal work, injecting facial embeddings via face detection is powerful. It does not work on illustration. The underlying detector only finds photographic faces; on anime art it finds nothing.

The comparison above only asked whether faces held. In production, speed matters too.

The first image is slow because it includes loading a 6.7GB checkpoint from disk onto the GPU. As long as you keep generating with the same model, you pay that once.

Adding the reference image barely changed per-image time. The extra computation is one image embedding, negligible against sampling itself. Consistency costs essentially nothing.

For a service where one character recurs, this order gets you there fastest.

1. Generate a few dozen with tags alone and see the drift yourself. Skip this and you will pay for training without knowing whether you needed it.
2. Identify what drifts, then try a reference image. Change one thing at a time. We touched lighting at the same time and blacked out the frame; that destroys your ability to attribute cause.
3. Always test multiple strengths. 0.5 and 0.9 are different products. Defaults tend to land on either no effect or too much.
4. Re-measure with varied outfits and locations. Holding in one fixed scene proves nothing.
5. Only then consider training. Characters without a distinctive feature (common hair colour, average build) are meaningfully harder.