How AI Audio Cleaning Works: A Plain-English Explanation
Updated: May 2026
The old way: noise gates and spectral subtraction
Before neural audio models, the two main noise reduction approaches were noise gating and spectral subtraction. A noise gate simply mutes audio below a set amplitude threshold -- anything quieter than X gets silenced. It works on recordings with very clear separation between speech and silence, and fails badly when noise overlaps with quiet speech, which is almost always.
Spectral subtraction is more sophisticated. You take a sample of background noise, compute its frequency spectrum, then subtract that spectrum from the whole recording. The result preserves the voice while reducing the static. The problem: subtracted spectra never cleanly cancel. You get residual artifacts -- the "musical noise" effect where the leftover noise sounds like faint metallic harmonics shifting in pitch. Anyone who used early Audacity or early iZotope RX at high attenuation settings has heard this. It is easy to mistake for the AI doing something wrong. It is actually the old method revealing its limits.
How neural networks changed the problem
Neural audio models work differently. They are trained on large datasets of clean speech -- recordings where the voice is isolated and the noise floor is negligible. From that training data the model learns to build an internal representation of what human speech looks like as a pattern of frequencies over time. Vowels, voiced consonants, sibilants, breath sounds, natural room tone -- the model develops statistical expectations for all of them.
At inference time (when you actually clean a file), the model takes your noisy recording and outputs a "mask" -- a prediction, at each point in time and at each frequency, of whether that audio signal is speech or noise. The output is not the original signal minus the noise; it is a reconstruction of what the speech probably was, filtered through the model's learned expectations. This is why neural noise reduction sounds so much better: it is not subtracting, it is predicting.
The spectral domain explained simply
Audio processing that operates in the "spectral domain" transforms the audio from a time-series waveform into a representation that shows frequency content over time -- essentially a spectrogram. The neural model operates on this spectrogram representation rather than on the raw waveform. This matters because noise and speech occupy different frequency bands in different ways, and the spectral representation makes those patterns much more separable than the waveform does.
After the model computes its mask, the result is transformed back to a waveform and that is what you hear in the output. The entire operation on a two-minute file takes a few seconds on Apple Silicon because the GPU and Neural Engine are purpose-built for the matrix operations that dominate this kind of inference.
What on-device actually means technically
Cloud audio cleaners run the same class of neural model, but on a server that your file is uploaded to. The model weights are the same type; the computation is the same math. The difference is latency and data path. When you upload a file to a cloud cleaner, your audio crosses a network, sits in a server queue, gets processed, and comes back to you. Total round-trip time for a 30-minute file on a fast connection: 30 seconds to several minutes, depending on the service and queue depth.
On-device, the inference runs on your Mac's GPU and Neural Engine directly. The audio never leaves your machine. On an M1 chip, a two-minute recording processes in roughly 20 seconds. On M4, closer to 8 seconds. No queue, no upload, no server dependency. The model weights are installed with the app and run locally.
What AI cannot fix
Neural noise reduction is not magic. It is very good at removing stationary and quasi-stationary noise -- HVAC, fan hum, line noise, constant traffic. It is less effective on highly variable noise (a dog that keeps barking, a voice in the background, intermittent machinery) because the model cannot easily distinguish variable noise from speech variation. It will not repair clipped audio -- samples that hit the ceiling of the recording range and distorted. And it will not recover frequency content that was never recorded in the first place, which is why a bad recording still sounds like a bad recording after cleaning, just a cleaner bad recording.
Aulio Studio home · Pricing · For podcasters · Alternatives · Support