Transcription with speakers

Goal

Audio → text with timestamps and “Speaker 1, 2…” → rename once and have the name carry over to SRT, VTT and TXT. Measured on 5, 20 and 60 min audios.

Why

Transcribing is already easy; knowing who spoke is the hard part (diarization). They are two different models: Whisper hears words, the diarizer groups chunks of voice by similarity. The pipeline crosses them by time. And an error number is only worth something if you know which answer was the correct one: that is why it is measured with audio whose truth you know.

Repo: IchiSieben/modelos-mac. Requires the manager from step 16 running on the Mac.

Steps

1. Test audio with a known truth

The owner’s personal videos were not used: they have identifiable people. Dialogues are generated in Spanish with synthetic voices and who said what and when is saved:

mac$ python scripts/make_synthetic.py --minutes 60 --voices Mónica,Paulina,kokoro:em_alex,kokoro:ef_dora --name real-60

It leaves ~/modelos/audios/real-60.wav (60.1 min) and real-60.truth.json (each turn with its voice, start, end and text).

kokoro:* are voices from the Kokoro-82M model (Apache-2.0), via mlx-audio. Only for generating tests: its phonemizer drags in a GPL dependency, so it lives in a separate extra (--extra synth), outside the service.

2. The pipeline

audio ──ffmpeg──► WAV 16 kHz mono ──► whisper-large-v3-turbo ──► segmentos con texto
                                  └─► Nemotron-3-Diarization ──► turnos (quién, cuándo)
                         merge: cada segmento toma el hablante con más solape → "Hablante N"

Speakers are numbered in order of appearance. Renaming does not touch the original result: a map {"Hablante 1": "Moderadora"} is saved and applied on export.

3. From the web

Tunnel (ssh -L 8765:127.0.0.1:8765 mac) and in the browser http://127.0.0.1:8765/: Task: Transcription with speakers, upload the file, Queue. The 5 min audio took 19.0 s (model load 3.2 s, footprint 2.9 GB). Click the row, type the names, Rename:

(capture of the real setup: not published; see docs/CAPTURAS.md for the demo-mode ones)

The SRT now comes out with the names:

(capture of the real setup: not published; see docs/CAPTURAS.md for the demo-mode ones)

4. Measuring

mac$ ~/modelos/envs/main/bin/python scripts/bench_speakers.py ~/modelos/audios/real-{05,20,60}.wav

~24 hours of audio per hour on long audios. In the 60 min one, Whisper takes 135 s and diarization only 12 s. Every wrongly assigned segment is listed in bench/speakers-<audio>.review.txt to review by hand.

5. Two diarizers, one by default

Similar error, but Nemotron gets right how many people there are, which is what makes renaming useful, and it is 5.5× faster. Risk: on a 5 min audio it gave 36.6 % error (on another with the same two voices, 3.0 %). If the result looks odd: {"diarizer": "sherpa", "num_speakers": N}.

6. Why Nemotron failed with 36.6 %

Nemotron does not cluster voices at the end, as sherpa does. It processes the audio in chunks and has 8 speaker slots that fill in order of arrival: a speaker cache. What is decided at the start weighs on everything else. The model was loaded once and variants were tried on the same 5 min audio (Mónica + Eddy es-MX, distinct voices):

Three conclusions:

• It is not chance: the same audio always gives the same error.
• The start decides it. What the model hears in the first ~30 s conditions the cache: the second voice ends up in slot 0 and slot 1 never gains confidence. If the audio starts after those 30 s, it separates well. Which exact part of the start causes it was not isolated: the second voice has a short turn there, but cutting the first 15 s (which removes it entirely) still leaves 14.4 %.
• Lowering the threshold is a patch: it improves the error, but does not fix it.

Practical consequence: if the result has almost all the time on a single speaker, distrust it. Repeat with sherpa, which clusters the whole audio at the end and does not depend on the start.

7. With a real audio (56 min)

A real audio from the owner, 56 min, with real people. The audio and the transcripts do not leave the Mac and do not enter any repo: they stay in ~/audios-prueba/salidas/. It is queued from the Mac itself:

mac$ curl -s -F task=speakers -F 'params={"diarizer":"nemotron"}' -F "file=@$HOME/audios-prueba/<audio>.m4a" 127.0.0.1:8765/jobs
mac$ curl -s "127.0.0.1:8765/jobs/<id>/export?fmt=srt" > ~/audios-prueba/salidas/nemotron.srt

* This run shared the Mac with another test; sherpa’s clean time is the one in the row above.

What you can see without knowing the correct answer:

• Whisper performs the same as with synthetic voices: 129–146 s for 56 min.
• sherpa, without being told how many people there are, is useless on real audio: 129 groups.
• The two diarizers do not agree: they disagree on 48 % of the time, measured by taking one as the reference for the other. Of that, ~38 % is who is speaking and ~11 % is that one hears voice where the other does not. Nemotron splits the time almost in halves; sherpa with 3 speakers fixed puts 92 % on one. Given what we know from step 6, neither of the two deserves blind trust.

The number 3 was given to sherpa by Nemotron. So the number of speakers is not an independent opinion, but the split of who speaks when is. Which one is right is decided by a person reading both SRTs.

8. Making the manager start on its own: LaunchAgent

mac$ bash scripts/install-launchagent.sh
.../Library/LaunchAgents/dev.modelos.server.plist: OK
{"version":"0.1.0","running":null,"task":null,"queued":0,"free_gb":54.3}
mac$ lsof -nP -iTCP:8765 -sTCP:LISTEN
Python  94703 <usuario>  7u  IPv4  TCP 127.0.0.1:8765 (LISTEN)

What the LaunchAgent does:

• Starts the manager at login on the screen (LimitLoadToSessionType: Aqua).
• If it goes down, it relaunches it (KeepAlive). Tested: the process was killed by its PID and it came back on its own after ~21 s.
• It still listens only on 127.0.0.1.
• A 5 min job ran the same as before: 18.7 s and 2.93 GB.

The installer is idempotent; --uninstall removes it.

mac$ lsof -nP -iTCP:8765 -sTCP:LISTEN
Python  94703 <usuario>  7u  IPv4  TCP 127.0.0.1:8765 (LISTEN)
mac$ ~/modelos/envs/main/bin/python scripts/bench_speakers.py ~/modelos/audios/real-{05,20,60}.wav

Limits (read them before selling this)

• The tables in steps 4 to 6 use synthetic voices: no noise, no overlaps, no interruptions. The real audio in step 7 confirms it: there the two diarizers disagree for almost half the time.
• n = 1 per audio. The Mac was indexing Photos in the background (60–320 % CPU): the times are the ceiling.