Social music collaboration app
A prototype lets musicians search by instrument, read from the audio itself
On a live music collaboration app, about a third of example searches asked for an instrument or role that no field held. We showed the gap sat in the data, not the ranking, and prototyped audio tagging on real tracks, designed to run inside the app's own cloud.
Marketing & MediaAudio search and discoveryAI added to a live product
The challenge
Musicians searched for roles such as piano, drummer or vocalist that no field held, and declared genres were often wrong. The client tied search directly to retention.
What we did
A paid prototype on real tracks: CLAP zero-shot tagging and librosa features read the audio, rules turn prompts into search signals, and a production design keeps it all in the client's cloud.
The outcome
On a sample of 112 real tracks, the audio recovered what tracks actually were where declared genres were wrong, and the client invited us to propose a long-term team with search first.
- Prototype data
- 112 real tracks from 47 artists
- Platform
- Live music collaboration app, over a million users
- Design
- Extends the existing GraphQL API, no fork
- Data boundary
- Models designed to run in the client's cloud
The starting point
A live social music collaboration app where musicians record parts on mobile and others add vocals or instruments to the same song. Finding the right collaborators is the core loop, and the client tied search directly to retention.
What had to change
Searches such as piano, drummer or vocalist asked for an instrument or role that no field held, and declared genres were often wrong. Tuning ranking on that metadata could not fix it.
What could not break
- Extend the existing GraphQL schema, never fork it, so the mobile app and the website keep working
- All data and models stay inside the client's own Google Cloud project
- No release that has to be coordinated with the separate audio engine developer
- Live uploads keep flowing while the catalogue is enriched
The calls we made
We framed weak search as a data problem first: the missing signal had to come from the audio, not from better ranking on bad metadata.
- 01
Read the platform before touching it
We mapped the live platform from the outside in C4 views and sequence flows, which showed that search indexed only part of the catalogue and ignored the audio attributes it already stored.
- 02
Listen to the audio
CLAP puts audio and text in one space, so a phrase like solo piano can be scored against any track with no per instrument classifier to train, and librosa adds cheap, explainable rhythm and energy features.
Ruled outTuning ranking on unreliable genre tags.
- 03
Rules first, a model later
Prompt parsing starts as deterministic, testable rules with typo correction, and a language model can take over later and emit the same signals.
- 04
Measure relevance, do not claim it
Production relevance is judged against a labeled baseline with precision and nDCG at five, with ranking weights tuned offline on labeled queries.
Ruled outClaiming relevance without a measured baseline.
- 05
Own the models
Self-hosted CLAP and librosa on the client's Kubernetes would put vectors in its own index, migrated by dual write and then deprecation, with musical key flagged as the one signal the open models do not replace cleanly.
Ruled outStaying on the third party tagging service, whose tags no search facet used.
How the system fits together
What we built
Audio analysis on real tracks
CLAP zero-shot tagging and librosa features over 112 real tracks from 47 artists, which recovered what tracks actually were where declared genres were wrong, plus sounds like similarity from the audio vectors.
Reference architecture
C4 views and sequence flows of the live platform, read from the outside, which showed that search ignored the audio attributes it already stored and indexed only part of the catalogue.
Live search demo
Prompt search with typo correction, role, genre and mood facets with live counts, and a search box over the client's real tracks.
Production design
Enrichment workers on the client's Kubernetes, hybrid lexical and vector ranking fused with RRF, a taste vector per user and suggestions for the complementary role a song still needs.
How it works
Upload
A musician uploads a track to the existing storage and transcoding pipeline.
Enrich
Workers tag instruments, mood and rhythm from the audio and store the tags and a vector in the search index.
Parse the search
Rules turn a query such as vocalist with piano into structured signals, correcting typos along the way.
Rank and explain
The existing GraphQL API returns results ranked on audio, popularity and recency, with the reasons shown.
Personalize
A taste vector from each user's plays nudges the order without overriding intent, so a drummer search still returns drummers.
A closer look
Query: "vocalist with pianno, upbeat, brazil"
typo fix pianno -> piano (edit distance 1)
concept vocals scored by CLAP
concept piano scored by CLAP
DSP upbeat tempo, beat clarity
region Brazil
free text (nothing left over)
Each signal becomes one Elasticsearch clause;
results come back scored, with the reasons shown.What changed
On the sample, the audio model proved more trustworthy than the existing metadata, recovering what tracks actually were where declared genres were wrong. The client then asked us to propose a long-term full-stack team with search and discovery as the first priority.
- Audio more reliable than declared genres
- Finding
- Asked to propose a long-term team
- Next step
Under the hood
- Datics role
- AI added to a live product
- Integrations
- Designed to sit behind the existing GraphQL API, read audio from the client's storage bucket and write tags and vectors into Elasticsearch.
- Ranking
- One weighted score over audio, popularity, recency, geography and taste, with lexical and vector results fused by RRF, weights tuned offline for nDCG at five, and learning to rank from clicks and plays later.
- Data boundary
- Audio models are designed to run inside the client's own cloud project, on a GPU node pool for CLAP and CPU workers for librosa, so nothing leaves it.
- Rollout
- Catalogue backfill as an idempotent, resumable, rate limited background batch, released on the discoverable subset first, with dual write before retiring the third party tagging service.
Data and retrieval
Apps and APIs
Also in the stack
How it shipped
Read the platform
A reference architecture of the live platform, built from the outside, located the search gap in what the index held and what it ignored.
Metadata and audio notebooks
A metadata search notebook showed what the real fields could answer, and an audio notebook ran CLAP and librosa over real tracks for the searches they could not.
Live demo and production design
A working demo on the client's real tracks brought prompt search, facets and sounds like similarity together, alongside a production design for the live stack.
Team proposal
The client then asked for a long-term team with search first, and we proposed a plan with acceptance criteria for the first search release, measured on a labeled query set.
Read before touching
We mapped the live platform from the outside before the test started, so the prototype targeted the real gap.
Proof on the client's own tracks
Every notebook and the demo ran on the client's real catalogue, not stand in data.
Production plan with a definition of done
The production plan we proposed set acceptance criteria for a first search release: behind the existing API, with explicit filters and a measured gain on a labeled query set.
What this unlocked
The same index is designed to feed recommendations, such as the drummer a song with vocals and guitar still needs, and we proposed a visual pass over sampled video frames for drummer searches, the hardest to read from a full mix.
Related work
Have a product with the same kind of constraints?
Tell us what already exists and what you want it to do. We will tell you what it takes.