Skip to content
← All projects

AI SECURITY · PHYSICAL TESTBED

A small U-Net that erases adversarial stickers from stop signs

A printed sticker on a stop sign made our test car's detector miss more than half of its approach frames. I built a small network that paints the sticker out of the camera image, so the same detector gets a second, clean look.

The code is private: ask me for access
Three crops from a real QCar2 camera frame: the stop sign with an adversarial sticker, the U-Net's predicted sticker mask, and the restored sign.
  • 48% → 76% detection

THE GAP

A printed sticker can hide a stop sign from a trained detector.

On our QCar2 testbed, a printed patch on the physical stop sign dropped the car's YOLOv8s detector from reliable detection to missing more than half of the approach frames. My teammates built the attack and a flagger that watches the detector's confidence for signs of trouble.

Flagging an attack does not make the sign visible again. My part was the next step: clean up the flagged frame so the same detector can read the sign.

01

No clean photo

There was no clean capture of the testbed sign, so nothing showed what sits under the sticker.

02

Do no harm

A fix that alters clean signs, or loses a detection the car already had, is worse than none.

03

Pretty is not enough

Picking checkpoints by synthetic image quality gave a model that looked great in simulation and hurt real crops.

WHAT IT DOES

A restorer that only touches the sticker.

The restorer is a 3.35M-parameter U-Net with two heads: one paints a repair, the other predicts a mask of where the sticker is. The repair is blended in only where the mask says sticker, so a clean sign passes through almost unchanged, and the mask area doubles as a tamper signal. It is meant to run only on frames the flagger raises, and it gives the same detector a cleaner picture instead of replacing it.

Changes only the sticker

The output is m·g + (1 − m)·x, so pixels the mask calls clean pass through. Clean signs barely change.

Trained without a clean photo

Synthetic sticker pairs matched to the real sign's color, noise and scale, plus real crops where the loss ignores the sticker.

Tuned for the detector

Fine-tuning pushes up the deployed detector's stop-sign confidence, and checkpoints are picked on real crops with a heavy penalty for collateral change.

Small and fast

3.35M parameters. 1.4 ms per 256 px crop on a workstation GPU in FP16, about 24 ms on a CPU.

HOW IT WORKS

How a flagged frame gets a second look

  1. 1

    Detect

    The deployed YOLOv8s runs on every camera frame as the car approaches the sign.

  2. 2

    Flag

    My teammates' flagger watches the confidence stream and raises frames that look attacked.

  3. 3

    Crop

    The sign is boxed, cropped square at 1.15× the box, and resized to 256 px.

  4. 4

    Restore

    The U-Net predicts a sticker mask and a repair, and changes only masked pixels.

  5. 5

    Paste back

    The restored crop goes back into the full frame, matching how the car detects.

  6. 6

    Re-detect

    The same detector takes a second, clean look, and the car decides.

SYSTEM DESIGN

Where the restorer sits in the loop

The detector, attack and flagger came from my teammates; the restorer, its training data and its evaluation are mine. In testing, a stock COCO model boxes the sign because the patch does not fool it; on the car, that box would come from tracking.

RESULTS

What changed on 25 attacked frames

76%

of attacked frames with the stop sign detected after restoration, up from 48%

7 of 13

missed frames that flipped back to detected

0

frames where restoration took away a detection the car already had

0.13/255

mean pixel change when the input is a clean sign

Scored offline against the deployed detector on 25 full frames from one attacked run, at a 0.25 confidence threshold; the clean-sign check uses synthetic signs, since no clean photo of the testbed sign exists.

WHAT IT TAUGHT ME

Doing less was the fix

The model that worked was the one allowed to do less. My first version regenerated the whole sign and changed pixels outside the sticker by 13.4/255 on average; compositing brought that down to 0.13/255. The six frames it still misses all have the sticker over parts of the sign the detector needs, and a bigger training run, with four times the batch and an extra perceptual loss, did not beat the final checkpoint. The fix is data: a clean photo of the real sign would give the repaint a true target.

BUILT WITH

  • PyTorch
  • Ultralytics YOLOv8
  • OpenCV
  • NumPy
  • scikit-image
  • Quanser QCar2

Like what you see?

Thanks for reading. There's more where this came from.

See all projects