Coast Technology
Dealers see rival stock, prices and sales daily, captured from many changing websites
My Lot Spy shows RV and automotive dealers what competing lots hold, at what price and what sells. We built its crawling engine around shared crawler types per site platform, so a template change is fixed once for every dealer on it.
Automotive & DealershipsCompetitive inventory intelligence engineModernized a live product
The challenge
Dealers had no current view of rival lots. Every source is a public dealer website that changes templates, blocks bots or goes offline, and sales can only be read from complete daily snapshots.
What we did
A crawling engine built on shared crawler types, with partial crawls reported and every listing mapped to one catalog, under a portal for trends, price comparison and alerts.
The outcome
The product is live and in continuous use. One engine serves RV and automotive OEM dealer networks in the U.S. and Canada, and new dealer crawlers are added in nearly every sprint.
- Sources
- Public dealer websites, captured every day
- Crawler types
- Dozens of shared templates for site families
- Dealers see
- Stock and sales trends, price comparison, alerts
- Serves
- RV and OEM dealer networks, U.S. and Canada
The starting point
Coast Technology already sold software to dealerships and wanted a separate competitive intelligence product, My Lot Spy. Dealers use it to see what competing lots hold, how prices move and which units sell, across RV and automotive dealer networks.
What had to change
Manual research across dealer websites was slow and fragmented, so pricing and stocking calls were made on stale or partial information. The product needed every competitor listing captured daily and mapped to one make, model and trim catalog.
What could not break
- Only public listing data from dealer websites is collected
- Snapshots must be complete every day, because sales are inferred from units that disappear between them
- Every listing maps to one make, model and trim catalog, whatever its title says
- RV and OEM dealer network instances stay separate, each with its own dev, QA and production
The calls we made
We treated data capture as its own product, designed for sources that change and fail every day, and kept it apart from the portal dealers use.
- 01
Crawler families, not a scraper per dealer
Many dealer websites are built on the same site platforms, so we grouped them into shared crawler types. A template change is fixed once for every dealer on that platform, and the engine has grown to dozens of types.
Ruled outA custom scraper for each dealer site, rebuilt every time that site changed.
- 02
Browsers and proxies only where a site needs them
Splash and Selenium based crawler types render the sites that need a browser, and proxies are tuned per crawler type, so the many simpler sites stay on lighter crawlers.
- 03
Partial crawls reported, not only failures
Sales are inferred from units that disappear between daily snapshots, which only works when each snapshot is complete. So the admin panel reports partially successful crawls as well as failed ones.
- 04
Mapping as ongoing work, with a way to undo it
Listings arrive with messy titles, conditions and missing fields. Management commands map each offer to a make, model and trim and clean out wrongly mapped products, and offers can be unmapped from the dashboard.
- 05
Keep the engine and the portal apart
The dealer portal reads mapped data through REST APIs, so we could rebuild it in two week sprints, starting from approved user stories, while daily capture kept running underneath.
How the system fits together
What we built
Crawling engine with an admin panel
A Django engine that runs and tracks crawler jobs, with CSV import and export of vendors and spiders, and reports on failed and partially successful crawls.
Catalog mapping and daily statistics
Management commands map scraped offers to products by make, model and trim, delete wrongly mapped products and generate daily statistics, while every offer keeps first seen and last seen dates.
Trends, comparison and alerts for dealers
A React portal with a dashboard, stock and sales trends, My Inventory, a comparison of each unit's price against similar units at other dealers with a PDF report, and triggers that email a dealer when matching inventory appears.
Dozens of shared crawler types
Templates for families of dealer sites, including Splash, Algolia and Selenium based types, with proxies tuned per type and a Selenium crawler for the hardest sites. New dealers are found through dealer locator pages.
How it works
Capture
Each day the engine runs crawler jobs across dealer websites, each site handled by the crawler type for its platform.
Check
Failed and partially successful crawls are flagged in the admin panel and fixed in the crawler health pass of the sprint.
Map
Every scraped offer is mapped to a make, model and trim, and wrongly mapped products are cleaned out.
Infer
Units that disappear between complete daily snapshots are counted as sold, and daily statistics are generated.
Serve
APIs and dashboards turn the mapped data into stock trends, sales trends and price comparisons for dealers.
A closer look
What changed
My Lot Spy is live and in continuous use, with coverage growing as new dealer crawlers are added in nearly every sprint. One engine serves an RV instance and automotive OEM dealer networks, across U.S. and Canadian dealers.
- Continuous use, new crawlers most sprints
- In production
- U.S. and Canada from one engine
- Countries
“The team successfully completed the initial phase of the RVP project. They have now moved on to phase two. Umair and his team continue to impress us and we are so happy to be partnered with them.”
Richard DeLanceyCIOCoast TechnologyUnder the hood
- Datics role
- Product built by Datics
- Data sources
- Only public listing data from dealer websites, captured daily, with first seen and last seen dates on every offer.
- Data layer
- An early MySQL scraper database migrated into the Django engine on PostgreSQL, sped up with hashmaps and set differences, with materialized views for the heavy analysis reads.
- Deployment
- A containerized Django web service on DigitalOcean cloud servers, separate dev, QA and production for each dealer network instance, and crawler logs synced to AWS S3.
- No AI in the product
- It is data engineering and analytics with no AI model inside; it is the trusted data an AI pricing feature would need.
Data and retrieval
Cloud and infrastructure
Apps and APIs
Also in the stack
How it shipped
Plan and requirements
We started from a project plan and a software requirements specification for the RV product.
One engine for every crawler
The first scraper pipeline wrote to its own database; we migrated its data into the Django crawling engine, sped the migration up with hashmaps and set differences, and folded the crawlers into the engine.
APIs and a second instance
REST APIs for products, offers and analysis went live, along with an automotive OEM dealer network instance next to the RV one.
A new portal and wider coverage
The dealer portal was rebuilt in two week sprints on the same engine, while crawlers reached Canadian dealers, two more OEM dealer networks and new RV dealers.
Operations, sprint after sprint
Every sprint fixes failed crawlers, adds new dealer crawlers and corrects mapping, each change tested on QA or dev before production.
Owned end to end
We own scoping, design, the crawling engine, the portal, DevOps and crawler operations, so the client runs a second product without building a data engineering team of its own.
User stories before each sprint
Each portal module started as user stories reviewed and approved before code, and each two week sprint ended with changes tested on QA or dev before production.
Crawler health as routine work
Every sprint mixes failed crawler fixes, mapping fixes and new dealer crawlers, and the client receives regular progress reports.
What this unlocked
With competitor stock captured, mapped and checked every day, dealers have the trusted market data an AI pricing or stocking feature would stand on.
Related work
Have a product with the same kind of constraints?
Tell us what already exists and what you want it to do. We will tell you what it takes.