Spotify Shut Off Its Audio Features API. I Rebuilt Tempo, Key and Loudness From Raw Audio
Spotify turned off Audio Features for new apps on 27 Nov 2024. This is the deterministic pipeline I built to measure tempo, key and loudness from the audio itself, inside a fixed 109-number vector per song, with the numbers from my own catalogue.
The API went away, the audio did not
On 27 November 2024 Spotify stopped serving Audio Features, Audio Analysis and Recommendations to new apps (Spotify for Developers). The Audio Features object carried tempo, key, mode, loudness, danceability, energy, valence and more. Developers who had built DJ tools and mood apps on those numbers lost them.
I run my own music service, Billboard, so I had the opposite problem: no API to lose, only audio. This is the deterministic engine I built to measure those signals from the file itself. It is part 1 of a 14 part series, one stage a day.
The whole pipeline in 40 seconds
One pass per song
- Decode mp3, FLAC or m4a to mono float32 and resample once to 22,050 Hz.
- Excerpt: the middle 60 s, peak normalised. Loudness and duration use the whole file, because they are whole-file quantities.
- One shared STFT (n_fft 2048, hop 512), a 128-band mel spectrogram and a harmonic/percussive split, computed once and handed to every analyzer.
- Nine analyzers: pitch (pYIN), tempo, rhythm, timbre, harmony (key by Krumhansl-Kessler, beat-synchronous chords), melody, loudness (ITU-R BS.1770-4), spectral summaries, a vocal proxy.
- One record whose id is sha256(track, audio, versions, excerpt), appended to a store and announced on an outbox.
- 109 numbers in a fixed order, written to one D1 row per song, the track card, beside the label and where the label came from.
The map
What it hears
From the stored record of Billie Eilish's bad guy: 135.9 BPM at confidence 0.97, G minor, -13.2 LUFS integrated, true peak -1.8 dBTP, vocal probability 0.94 (a DSP heuristic labelled low confidence), spectral centroid 1,497 Hz. The analysis took 13.5 s on one core: decode 3.27 s, the shared STFT 3.69 s, pitch 2.85 s, loudness 1.93 s.
feature_vector.py: the schema of the 109 slots
_SCALAR = [
("pitch", ("mean_hz", "median_hz", "std_hz", "range_semitones", "voiced_fraction", "stability")),
("bpm", ("estimated_bpm", "confidence", "octave_agreement", "beat_strength", "beat_interval_s_std")),
("rhythm", ("onset_rate_per_s", "onset_strength_mean", "onset_strength_std", "beat_regularity",
"percussive_ratio", "syncopation", "tempo_stability")),
("timbre", ("spectral_centroid", "spectral_centroid_std", "spectral_bandwidth", "spectral_flatness",
"spectral_rolloff", "zero_crossing_rate", "brightness")),
("harmony", ("key_correlation", "key_confidence", "tonal_clarity", "chord_change_rate_per_s", "chord_count")),
("melody", ("complexity", "note_rate_per_s", "interval_mean_semitones", "interval_std_semitones",
"range_semitones", "direction_changes_per_s")),
("dynamics_duration", ("rms_mean", "rms_std", "peak_dbfs", "crest_factor_db", "dynamic_range_db",
"loudness_lufs", "loudness_range_lu", "silence_fraction")),
("spectral", ("spectral_entropy_mean",)),
("vocals", ("probability",)),
]
N_CONTRAST, N_MFCC, N_CHROMA = 7, 20, 12
def _names() -> list[str]:
out = []
for section, keys in _SCALAR:
out += [f"{section}.{k}" for k in keys]
out += [f"timbre.spectral_contrast.{i}" for i in range(N_CONTRAST)]
out += [f"timbre.mfcc.{i}" for i in range(N_MFCC)]
out += [f"timbre.mfcc_std.{i}" for i in range(N_MFCC)]
out += [f"harmony.chroma_rel.{i}" for i in range(N_CHROMA)]
out += ["harmony.mode_is_minor"]
out += ["spectral.band_low", "spectral.band_mid", "spectral.band_high"]
return out
FEATURE_NAMES: list[str] = _names()
N_FEATURES = len(FEATURE_NAMES)Two choices in that file matter. Chroma is rotated so bin 0 is the key root: a song moved up a tone is still the same genre, so the model should see harmony relative to the tonic. And a failed stage contributes NaN: imputing quietly inside the feature code would hide a broken analyzer inside a training set, so the caller decides.
The 109 slots, by group
Tested on known answers
On every test run, synthetic signals with a known truth go through the real engine: a 440 Hz sine must read 440 Hz (within 2 %) and pitch class A; a 120 BPM click 120 BPM (within 3 %) with regularity above 0.9; C F G C must read C major and contain those chords; a -20 dBFS sine must read -23.0 LUFS (within 1 LU) and -20 dBTP; white noise must be flat and not vocal; silence and 0.3 s of audio must be refused with coded errors.
On the real catalogue: 3,141 tracks analysed, and the last pass did 668 tracks with 0 failures in 3,466 s, a median of 22.7 s per track.
What I did not rebuild
Tempo, key, mode and loudness are measurements. Danceability, energy and valence are judgments: a model's opinion of a song. My vector keeps their ingredients (beat regularity, onset strength, RMS, mode) but no score, because I would rather leave a slot empty than ship a number I cannot defend.
Part 1 of 14, next: A 118-Minute Mix Ate 9.3 GB of RAM: What It Taught Me About Decoding Audio.
Part 1 of 14, next: A 118-Minute Mix Ate 9.3 GB of RAM: What It Taught Me About Decoding Audio.
More: LinkedIn · Instagram. Portfolio and case studies: designsbyduhart.org.
If any of this saved you an afternoon, Buy me a coffee.