Skip to content
← All projects

VISION-LANGUAGE · BENCHMARK

An honest number for how well a VLM finds cars, people and signs

I measured how well an off-the-shelf vision-language model, Qwen2.5-VL-7B, finds objects in BDD100K driving scenes when you simply ask it to. The answer is not very well, and the point of the project is a number you can trust and reproduce with two Docker commands.

The code is private: ask me for access
Charts of the real results: AP for each of the 10 BDD100K classes, from bus at 0.115 down to train at 0.003, and AP by object size, 0.004 small, 0.05 medium and 0.19 large, for an overall mAP of 0.041.

THE GAP

It is easy to show a VLM drawing boxes, and hard to say how good the boxes are.

Vision-language models are moving into driving stacks for open-vocabulary perception and scene understanding. Ask one for boxes and you get tidy JSON back. Whether those boxes land on the right things takes a full dataset and a standard metric to answer.

I wanted one number on a full public driving dataset, scored the standard way, that anyone with a single 24 GB GPU could reproduce.

01

Wrong yardstick

Caption and question-answer scores say little about whether a box actually lands on the car.

02

Fragile numbers

One prompt clause made the model return nothing on about 58% of busy scenes.

03

Hard to reproduce

Serving versions and image-size settings shift results unless every piece is pinned.

WHAT IT DOES

A benchmark anyone with one GPU can rerun.

Two Docker services split the work. A vLLM server loads Qwen2.5-VL-7B-Instruct and serves an OpenAI-compatible API. A small CPU-only client builds COCO ground truth from the BDD100K labels, sends every validation image, maps the boxes back to the original pixels and scores them with pycocotools. The goal is an honest, repeatable reference point for comparing VLMs with each other and with trained detectors.

Two commands

docker compose up -d vllm, then docker compose run --rm eval. One .env file holds every setting.

Exact box mapping

The client resizes each image the way the server does, so boxes map back to original pixels exactly.

Safe to interrupt

Detection is checkpointed per image, so a crash or restart picks up where it left off.

Keeps partial answers

When a long box list gets cut off, the parser keeps every finished box and drops only the broken tail.

HOW IT WORKS

What one run does

  1. 1

    Preflight

    A check script confirms the GPU, driver, Docker GPU access and disk before anything downloads.

  2. 2

    Serve

    vLLM loads Qwen2.5-VL-7B in bf16 on one 24 GB GPU, pinned to v0.23.0.

  3. 3

    Ground truth

    BDD100K labels, in either layout BDD ships, become COCO ground truth for 10 classes.

  4. 4

    Detect

    Each image goes to the model with one prompt asking for every object as JSON.

  5. 5

    Map

    Boxes are scaled back to original pixels and free-text labels mapped to BDD classes.

  6. 6

    Score

    pycocotools computes COCO mAP overall, by object size and per class.

  7. 7

    Report

    Optionally, sample images with boxes drawn and a Word report with the tables.

SYSTEM DESIGN

Two containers, one HTTP line between them

The GPU work and the scoring live in separate containers that only talk over an OpenAI-compatible API, so the client can run on another machine and the model can be swapped from one config file.

RESULTS

The number, and the shape behind it

0.041

COCO mAP@[.5:.95] on all 10,000 BDD100K val images (0.082 at IoU 0.5)

0.19 vs 0.004

AP on large objects versus small ones

58% → 0.1%

empty answers: about 58% of busy scenes with the old prompt, 0.1% of all images after the fix

~1.4 s

per image at 16 concurrent requests on one RTX A5000

Trained detectors such as YOLO or Faster R-CNN are expected to reach 0.30 to 0.40+ mAP on BDD100K; that range is a reference point, not something this repo measured.

WHAT IT TAUGHT ME

The prompt was the experiment

The prompt mattered more than anything else I touched. An early version told the model it could return an empty list if nothing was there, and it did exactly that on about 58% of busy scenes, which pushed mAP toward zero. Insisting on every visible object brought empty answers down to 0.1%, and now I check the empty rate before I trust any mAP. The result is humbling in a useful way: big, nearby vehicles work, small and distant objects mostly do not, and near-constant confidence scores cap what COCO AP can show. A prompted VLM is a baseline, not a detector.

BUILT WITH

  • Python
  • Qwen2.5-VL-7B
  • vLLM
  • Docker Compose
  • pycocotools
  • OpenAI Python client

Like what you see?

Thanks for reading. There's more where this came from.

See all projects