A call auditor that listens, scores and flags what breaks the rules

Most of a compliance review is listening: what was said, whether anyone got abusive, whether a required phrase was said or a banned one slipped out. It is slow, and the result depends on who is listening.

I wanted to see how much of that first pass a chain of open models could handle, and to turn the rules into something a team writes down once and a system checks every time.

A FastAPI backend transcribes the call with Whisper, runs it through a chain of open models, checks it against rules stored in the database, and saves the result to SQLite. A React and TypeScript front end shows the score, transcript and timelines, with logins, roles, teams, comments and tags so several reviewers can work on the same calls. Alerts, webhooks, emails and scheduled reports carry the findings to people who are not looking at the screen.

A single FastAPI endpoint runs every analysis in sequence on the uploaded file. Each step is wrapped on its own, so one failing model leaves a gap in the report instead of failing the whole request.

Looking back at the code, the feature list got ahead of it in places. The speaker turns are a placeholder: the audio is cut into two-second windows and each window is labeled by its spectral centroid, which is not real diarization. The sentiment model returns LABEL_0 to LABEL_2, but the score's weight table expects NEGATIVE, NEUTRAL and POSITIVE, so sentiment never moves the score. And custom rules are Python expressions run through eval, on routes that need no login, which is fine on a laptop and nowhere else. The next version needs a real diarization model, a tested label map, and rules that cannot run code.