SenTrac
Campaign reports use only the data each platform allows, refreshed within its limits
SenTrac depends on six social platforms whose APIs, caps and storage rules it does not control. We mapped what each one returns and permits, then built collection that fetches only what is new, stays inside platform limits and shows the operator its quotas and AI cost.
Marketing & MediaSocial data ingestion layerBuilt from zero
The challenge
Official social APIs are gated and tiered, comment providers cap what they return, and some data may not be stored at all. Reports still had to stay current and consistent across channels.
What we did
A per platform data map before design, one collector per platform behind a shared pipeline, incremental and throttled refreshes, storage rules in scope, and an operator view of quotas and AI cost.
The outcome
The founder closed about 95% of the admin screens in review before development began, and all five build milestones are now complete ahead of launch.
- Sources
- Official APIs plus vetted data providers
- Refresh
- New comments only, in staggered batches
- Rules in scope
- What may be stored, what must be deleted
- Operator view
- API quota monitor and AI cost tracker
The starting point
SenTrac is a new SaaS that collects comments, reactions and engagement from campaign posts on six social platforms and turns them into sentiment reports for agencies and brands. A super admin portal serves the team that operates it.
What had to change
Official APIs are gated and tiered, third party providers return about 200 top level comments per post for X and TikTok with no nested replies, and demographics depend on account types and permissions. A product built on one fixed assumption per platform would break the day that platform changed.
What could not break
- LinkedIn profile data is never stored beyond person IDs
- Deleted posts are removed from the platform's data
- Demographic insights stay within what each platform allows
- Refreshes stay inside each source's rate limits and quotas
- A failed fetch must never be stored as zero comments
The calls we made
We treated data access as a product risk to design around from the first week, not a detail to fix after launch.
- 01
Map each platform before designing reports
Discovery listed, channel by channel, which content types, interactions, comments, shares and audience data can really be extracted, so report metrics only promise what the data supports.
- 02
Mix official APIs and vetted providers
The AI blueprint found access gated, tiered and changing, so each platform gets its own collector behind one registry: official APIs where they serve, a scraping provider where they do not, as with Instagram, and a source can change without touching the reports.
Ruled outTying the product to a single fixed source per platform.
- 03
Platform rules as product rules
Storage limits went into the scope: LinkedIn profile data kept as person IDs only, deleted posts removed, and demographics limited to what each platform allows.
- 04
Fetch only what is new
Each refresh stores a metrics snapshot with day over day changes and inserts only comments it has not seen, with a cap on the first fetch and a daily limit after it, so quota goes to new data.
- 05
Throttle the refresh
Campaign refreshes go out in staggered batches of ten, platform fetches and AI analysis are rate limited, and slow external calls run in their own lane so quick database updates never wait on them.
Ruled outRefreshing every campaign and post at the same moment.
How the system fits together
What we built
Collectors and refresh pipeline
X and YouTube first, with a script that checks what each API actually returns, then Instagram, Facebook, TikTok and LinkedIn behind the same registry, refreshed on a schedule with deduplicated comments and day over day metric snapshots.
Operator controls
A super admin portal with an API quota monitor, an AI cost tracker, a financial hub and a data bank where the operator asks questions in plain language and gets a short summary, a confidence indicator and a chart.
Platform data map
A per channel record of the content types, interactions, comments, shares and audience data each platform really returns, including caps such as about 200 top level comments per post from third party providers.
Failures that surface
Passing errors retry with backoff, persistent ones are raised instead of stored as zero, and a post whose metrics show comments that the fetch missed is logged as a gap and can be backfilled.
How it works
Queue the posts
Each campaign post becomes its own collection task, sent out in staggered batches.
Collect within the rules
The platform's collector pulls only what its official API or vetted provider allows, inside its rate limits.
Keep what is new and permitted
New comments are added once, metrics are stored as snapshots, LinkedIn profiles stay as person IDs and deleted posts are removed.
Score and report
Only new comments go to the sentiment pass, and reports show only the metrics the collected data can support.
Track usage
API quotas and AI cost are logged for the operator view.
A closer look
What changed
The founder reviewed and closed about 95% of the admin screens before development began. The data layer was then built channel by channel, and all five build milestones are complete ahead of launch.
- About 95% of screens closed before development
- Admin design
- All five milestones complete ahead of launch
- Build
Under the hood
- Datics role
- Product built by Datics
- Sources
- X API v2 and YouTube Data API, scraping actors on Apify for Instagram posts and comments, and third party data providers where official access is limited, each behind one collector registry.
- Data rules
- Demographics limited to what each platform allows, LinkedIn profile data kept as person IDs only, and deleted posts removed from the platform's data.
- Known limits
- Third party providers return about 200 top level comments per post for X and TikTok, with no nested replies; Instagram demographics need a business account, and Facebook needs fine grained permissions.
- Runtime
- Python services with Celery workers in separate fast and slow lanes, PostgreSQL on Amazon RDS, retries with backoff and a backfill command for comment gaps.
Data and retrieval
Cloud and infrastructure
Apps and APIs
Also in the stack
How it shipped
Paid discovery
We reviewed competitor tools and mapped, platform by platform, what data can really be extracted, then wrote an AI blueprint that flagged data access as a business risk.
Scope and API research
Scope versions named each data source with a note that the product depends on those APIs, and a later round of API research updated the scope and functional breakdown after the founder dropped Pinterest.
Admin design
The super admin portal, from the data bank to the API quota monitor and AI cost tracker, was designed and reviewed until the founder had closed about 95% of its screens.
Collectors in sprints
X and YouTube collectors and the API check script came first, then Instagram, Facebook, TikTok and LinkedIn behind the same registry.
Hardening and operator portal
Refreshes were throttled and split into fast and slow lanes, failures made visible, and the operator portal completed the five build milestones ahead of launch.
API research before estimates
We researched each platform's API and data providers before sizing the build, and updated the scope when the founder dropped a channel.
Checked against live data
Development ran in two week sprints with a daily scrum, and comment deduplication and sentiment tracking were verified by cross checking live API data against the database.
What this unlocked
After launch, later scope adds keyword and trend monitoring and more channels on the same collector registry.
Related work
Have a product with the same kind of constraints?
Tell us what already exists and what you want it to do. We will tell you what it takes.