One uploaded drawing becomes a face that blinks and speaks. Identity holds because everything outside three marked ellipses is the merchant's own bytes, not because the model was asked nicely.
September 21, 20264 min readSaytu team
A merchant uploads one drawing of their character and gets back an avatar that blinks and talks. (A rigged Live2D model is the other way in, for merchants who already have one.) In between, a model has to produce five more pictures of somebody it has seen once, and every one of them has to be recognisably the same person. Ask an image model for that and it will mostly oblige. Mostly is not a guarantee, and a character whose face shifts every time it closes its mouth is worse than no avatar at all.
The set is smaller and stranger than it sounds. Two eye states, open and closed, and three mouth states, closed, half and open. Their cross product is six images, and that is the whole package.
Nothing in there is an emotion. This is a rig for two behaviours a face has to do continuously, blinking and speaking, and the six cells are the frames those two loops are drawn from. A happy avatar and a sad one would be a different problem with a different answer.
The merchant marks three ellipses on their own drawing: left eye, right eye, mouth. Those ellipses are the model's entire licence.
Each state is generated as a full candidate image, and then the candidate is thrown away everywhere except inside the marks:
result = base outside the mask, candidate inside itWe render the mask from the three ellipses as white on black, feather it, and join it onto the candidate as its alpha channel before compositing over the merchant's original. Their hair, their collar, their line weight and their palette are not regenerated and not compared against anything. They are the same bytes they uploaded.
Identity drift is bounded by construction rather than by how well the model behaved on that particular call. It is the difference between checking the work and removing the opportunity.
We crop to the 1024-pixel canvas in the browser, before anything is uploaded, for the same reason. The generator treats whatever it receives as the whole canvas, so cropping first means the marks, the masks, the style references and the composited output all share one coordinate space. The alternative, sending the original plus a crop rectangle, means applying the same offset in five places and being wrong in one of them.
A generated state inherits its source's background, and a source is usually an opaque rectangle, so an avatar dropped onto a coloured widget arrives inside a white box. Removing that box is not decoration.
The obvious approach is a colour distance threshold: anything close enough to the background colour becomes transparent. It also eats a white shirt, and a pale cheek, and the highlight in an eye.
So we cut with a flood fill from the image border instead. It removes only pixels connected to the edge. A white shirt in the middle of the frame is never reached, however close its colour is to the background it is sitting on. Connectivity is the whole reason to prefer it, and it needs no model call at all, which means it gives the same answer every time.
It works on flat and near-flat backgrounds. Painted scenery is not that, and the studio says so rather than mangling it.
The three ellipses start somewhere, and for a long time they started at fixed defaults that assumed a portrait with the eyes near the middle. On a standing full-body figure those defaults put two eye marks and a mouth mark on the waist and hands.
Guessing from the alpha bounding box does not rescue it either, because plenty of uploads are opaque rectangles with a painted background and the bounding box is the whole image. So we ask for the head's extent rather than infer it: one cheap vision call returns the face as fractions of the image, and because they are fractions, the same plan survives any resolution.
A three-quarter action pose has no front-facing eyes or mouth to mark. The honest options are to redraw the character front-facing or to refuse, and refusing every dynamic pose would refuse a lot of real uploads.
So we allow one redraw, and it is the single operation in this pipeline that returns a new picture rather than compositing into an existing one. The character is described back to the model and drawn again, and whatever comes out is what the six states are then built from. The guarantee that everything outside the marks is the merchant's own bytes does not survive it, because after a redraw none of the bytes are.
The redraw earns its place. Doing it quietly would not. The studio warns before we run it, and the result carries a label saying it was redrawn, for the whole life of the package.
A pipeline is allowed to have one step that breaks its own rule. What it is not allowed to do is break the rule without saying which step did it.
