Skip to content
← All projects
Code

GENAI · FULL STACK

A call auditor that listens, scores and flags what breaks the rules

A full-stack app for reviewing recorded calls. Upload a file or record in the browser, and it returns a transcript, sentiment, emotion, toxicity, intent and a 0 to 100 compliance score, then checks your own rules and can send alerts and scheduled reports. Personal data is redacted before anything is stored.

View the code on GitHubWatch it run
An example: the mel spectrogram of a synthesized two-voice support call, its first transcript line, a keyword rule flagging a refund promise, and a 0 to 100 compliance score gauge.

SEE IT RUN

A polite call that broke two rules

This is a real run, recorded from the app running on my laptop. The calls are synthetic: ten scripted support calls for a made-up energy company, voiced with macOS text-to-speech and checked against four rules. Tell the caller the call is recorded, never promise a guaranteed result, make no risk-free promises, and stay civil.

In the call the film follows, a customer was charged twice. The models heard a polite, grateful caller and scored the call 99.9 out of 100. The rules heard the agent say a new plan was “guaranteed to lower your bill every month”, and noticed that no one mentioned the recording. The app marked the call critical and posted both findings to a QA webhook as the analysis finished.

Across the ten calls, the rules flagged three. The score flagged none: nine calls scored 99.9 and one 87.8, partly because of a sentiment label bug, since fixed (see below). On a laptop CPU, each analysis took about twice as long as the call itself.

  • 10 synthetic calls
  • 4:59 of audio
  • 3 flagged by rules
  • every flag sent by webhook
Read the film as text
  1. 01Every support call has rules. Tell the caller the call may be recorded. Never promise a guaranteed result. Stay civil, even when the caller isn't. Who checks every call?
  2. 02billing-duplicate-charge.wav, speaker 1, at 0:24: “It won't. And if you move to our SAVER plan, it's guaranteed to lower your bill every month.” The call's waveform plays, and “guaranteed” turns red: prohibited guarantee language. One word.
  3. 03One word. One broken rule. Policy, rule 2 of 3: never promise a guaranteed result.
  4. 04Today, someone has to listen. The recording plays at 1x. Nothing to skim.
  5. 05Call after call. In real time. Ten calls add up to 4:59 of listening.
  6. 06And the rules live on paper. Every agent, on every call: tell the caller the call may be recorded; never promise a guaranteed result; stay civil, even when the caller isn't. A document can't flag a single call.
  7. 07Why I built it. Abhijeet Solanki, AI engineer and adversarial ML researcher, built this one end to end: models, API and front end. Most of a compliance review is listening. I wanted to see how much of that first pass open models could handle, and turn the rules into something a system checks every time.
  8. 08Voice Auditor. A first pass on every call, with the rules written down. Whisper, open models, FastAPI + React, open source under MIT.
  9. 09Step 1 of 5, upload: drop in a recording, or record one in the browser. Step 2, transcribe: Whisper writes the transcript, line by line. Step 3, score: open models score the tone 99.9 out of 100. It sounds fine. Step 4, check: your rules check what was said, and two match, a critical rule for guarantee language and a warning for the missing recording disclosure. Step 5, review: findings reach the team as the call is scored, and a webhook is delivered to the QA alerts channel with one critical finding.
  10. 10One call, two readings. The models heard a polite call: compliance 99.9, positive, gratitude, toxicity 0.1%, review below 70. The rules heard “guaranteed”: a critical finding, QA alerted by webhook, plus no recording disclosure.
  11. 11Built for review teams. Every call, every rule. Rules you write once: regex, keywords, a toxicity threshold and custom checks. Findings that travel: webhooks, email alerts and weekly reports. Private by design: “my card is [card number] and my email is [email]”, redacted before storage. Every action audited: admin, sign in, allowed; analyst, call analysis, allowed; viewer, delete record, refused; with roles, sign-in and retention. Built for teams: roles, teams, comments and tags. Every call on record: history, compare, and PDF, CSV and JSON exports. This run: 10 calls, 4:59 of audio, 3 flagged by rules, 79 seconds per call on a laptop CPU.
  12. 12Every call reviewed. Every rule checked. Voice Auditor, designed and built by Abhijeet Solanki. abhijeetsolanki.com/portfolio/voice-auditor. Open source, MIT, FastAPI + React.

THE GAP

Checking a call against policy means listening to the whole thing.

Most of a compliance review is listening: what was said, whether anyone got abusive, whether a required phrase was said or a banned one slipped out. It is slow, and the result depends on who is listening.

I wanted to see how much of that first pass a chain of open models could handle, and to turn the rules into something a team writes down once and a system checks every time.

01

Slow to review

Listening happens in real time, so review time grows with every call you record.

02

Rules on paper

A policy written in a document cannot flag anything. Someone has to apply it, call by call.

03

Findings stall

Even when a problem is found, someone still has to write it up and tell the right people.

WHAT IT DOES

A first pass on every call, with the rules written down.

A FastAPI backend transcribes the call with Whisper, runs it through a chain of open models, checks it against rules stored in the database, and saves the result to SQLite. A React and TypeScript front end shows the score, transcript and timelines, with logins, roles, teams, comments and tags so several reviewers can work on the same calls. Alerts, webhooks, emails and scheduled reports carry the findings to people who are not looking at the screen. Personal data is replaced with placeholders straight after transcription, and every sign-in and change lands in an audit log.

Transcript and signals

Whisper transcribes and detects language. Open models score sentiment, emotion and toxicity, classify intent zero-shot, pull topics and summarize.

A 0 to 100 score

Sentiment, emotion and toxicity combine into a 0 to 100 compliance score. Below 70 raises a warning, below 50 a critical alert.

Your own rules

Regex, keyword, sentiment, emotion, toxicity and score-threshold rules live in the database, each with a severity, and can be tested first.

Findings that travel

Alerts can fire webhooks and emails. Daily, weekly or monthly reports go out by email, with PDF, CSV and JSON exports.

Private by design

Card numbers, Social Security numbers, emails, phone numbers and similar details are replaced in the transcript before anything analyzes, stores or sends it. Recordings are deleted as soon as the analysis ends, and optional retention limits remove old analyses and audit entries.

Secure by default

Every route needs a signed-in user with the right role. There are no default passwords, the first sign-in forces a new one, and rate limits and an audit log cover every change.

HOW IT WORKS

What happens to one call

  1. 1

    Upload

    Drop in an audio file or record straight from the browser microphone.

  2. 2

    Transcribe

    Whisper, small model by default, transcribes the call and detects its language. Personal details are replaced with placeholders before anything else sees the text.

  3. 3

    Analyze

    Open models score sentiment, emotion and toxicity, then add intent, topics, summary and action items.

  4. 4

    Explain

    SHAP highlights which words pushed the toxicity score up.

  5. 5

    Check rules

    Your stored rules and the default thresholds run against the transcript and scores.

  6. 6

    Notify

    Alerts trigger any configured webhooks and emails, and the analysis is saved.

  7. 7

    Review

    Reviewers see the score, transcript and timeline, then comment, tag, compare or export.

SYSTEM DESIGN

One endpoint, many small models

A single FastAPI endpoint runs every analysis in sequence on the uploaded file. Each step is wrapped on its own, so one failing model leaves a gap in the report instead of failing the whole request.

RESULTS

What is in the repo

0–100

compliance score per call, with warning and critical alerts below 70 and 50

7

rule types a team can define, from regex and keywords to score thresholds

70

API routes, from analysis to users, teams, webhooks, schedules and the audit log

16

pages in the React and TypeScript front end

This is an application, not a benchmark: the repo has no accuracy measurements, so these are counts from the code.

WHAT IT TAUGHT ME

Where the feature list got ahead of the code

Looking back at the code, the feature list got ahead of it in places. The speaker turns are a placeholder: the audio is cut into two-second windows and each window is labeled by its spectral centroid, which is not real diarization. The sentiment model returned LABEL_0 to LABEL_2 while the score expected NEGATIVE, NEUTRAL and POSITIVE, so sentiment never moved a score or raised an alert. Custom rules were Python expressions run through eval, on routes that needed no login, behind a default admin password. Those parts are fixed: sentiment now counts, rules go through a small evaluator that can't run code, every route needs a signed-in user with the right role, there are no default credentials, and personal data is redacted before anything is stored. The next version still needs a real diarization model, and a set of labeled calls to calibrate the score.

BUILT WITH

  • FastAPI
  • Whisper
  • Transformers
  • Detoxify
  • SHAP
  • scikit-learn
  • SQLAlchemy
  • SQLite
  • APScheduler
  • React
  • TypeScript
  • Tailwind CSS
  • Recharts

Like what you see?

Thanks for reading. There's more where this came from.

See all projectsView the code on GitHub