Healthcare data consultancy
Provider data from scattered practice websites, verified against NPI inside the client's AWS
A healthcare data consultancy needed provider data from medical practice websites. We split finding pages from trusting what we extract: a crawler in its own AWS account discovers pages, extracts providers with an LLM into its data model and verifies each against the NPI Registry.
Healthcare & WellnessProvider data crawlerBuilt from zero
The challenge
Every practice website is built differently and many have no sitemap, scraped text alone could not be trusted, and the crawler had to run inside an AWS account that denies public traffic.
What we did
A discovery module and a parallel crawling module, LLM extraction into the client's own provider data model, and a match of every provider against the NPI Registry, deployed in Docker on EC2 in the client's account.
The outcome
The client received provider records matched to NPI and a first provider map, and asked next for direct loading into Snowflake and HIPAA aligned hosting for the crawler.
- Output
- Provider records verified against NPI, on a map
- Sources
- Practice websites, NPI Registry, Google Places
- Parallel test
- 102 URLs extracted in 644 seconds
- Deployment
- Docker on EC2 in the client's AWS account
The starting point
The client is a data consultancy that works under HIPAA as a business associate for healthcare organizations. It needed provider details such as credentials, specialty, locations, accepted insurance and hospital affiliations, collected from medical practice websites into storage its team could use.
What had to change
Each practice site is structured differently and many have no sitemap. Scraped text alone is not reliable enough, so providers had to be matched to official records.
What could not break
- The crawler runs inside the client's AWS account, where public traffic is denied at the VPC level
- Every provider is checked against official NPI records
- Sites without a sitemap still have to be covered
- The NPI API returns at most 1,200 results and Places at most 20 per call
- Output follows the client's own provider data model
The calls we made
We treated it as two problems, finding the right pages and trusting what we extract, and built a separate answer for each inside the client's own AWS rules.
- 01
Split discovery from crawling
One module builds the list of relevant URLs and saves it in batches of 100 to 150, and a second crawls those batches in parallel. In an early test, 10 parallel processes extracted 102 URLs in 644 seconds.
- 02
Sitemap first, discovery as fallback
The crawler checks a site's main page for a sitemap and, when there is none, discovers URLs up to three levels deep, then an LLM keeps only the pages with provider details.
- 03
Verify against the registry
Every scraped provider is looked up in the NPI Registry and merged with the crawl data, so the output rests on official records, not on scraped text alone.
- 04
Fit the client's network rules
We first designed it as Lambdas with S3 triggers. The client's VPC denies public traffic, so we built a containerized Flask app with Postgres for batches and state and deployed it on EC2 in its account.
Ruled outThe first design of Lambdas triggered by S3 uploads.
How the system fits together
What we built
Crawler with sitemap fallback
Crawl4AI with LLM extraction to a provider schema, a sitemap check with discovery up to three levels deep when none exists, duplicate removal and parallel batches tracked in Postgres.
The fields the team asked for
Credentials, specialty, bio, training, languages, locations mapped to providers, accepted insurance, telehealth and new patient status, services and hospital affiliations, on the client's own data model.
NPI and Places matching
Each provider is matched to the NPI Registry, and Google Places results around a point are matched to NPI records by name and address, keeping coordinates for a provider map.
Deployment and documentation
A Docker container on EC2 inside the client's AWS account, with the code headed for its own GitHub, a recorded process video, and schema and sequence diagrams of the crawler.
HIPAA aligned hosting design
A locked down VPC with security groups, IAM roles for services and Identity Center for people, mapped to the client's HIPAA controls and compared with a public subnet locked by IP rules.
How it works
Discover
The crawler reads a site's sitemap, or discovers relevant pages up to three levels deep when there is none.
Filter
An LLM keeps only the pages that hold provider and practice details.
Extract
Pages are crawled in parallel batches and turned into structured provider records, with duplicates removed.
Verify
Each provider is matched to the NPI Registry, and Google Places results are matched to NPI by name and address.
Map
Verified records are stored with their coordinates and plotted on a provider map.
A closer look
{
"provider_name": "Provider A",
"credentials": "MD",
"specialty": "Cardiology",
"npi_match": {
"npi": "0000000000",
"matched_on": ["name", "address"]
},
"practice": "Practice A",
"location": {
"address": "100 Main St",
"lat": 39.78,
"lng": -89.65
},
"accepted_insurance": ["Plan A", "Plan B"],
"telehealth": true,
"new_patients": true,
"hospital_affiliations": ["Hospital A"]
}What changed
The client received provider data matched to official NPI records and a first provider map. It then asked for the data to load straight into Snowflake, and we offered a period with Snowflake and the existing database side by side while a queue system and HIPAA aligned hosting were designed.
- Provider records verified against NPI, on a map
- Output
- 102 URLs extracted in 644 seconds
- Parallel test
- Direct loading into Snowflake
- Next
Under the hood
- Datics role
- Engineering delivered by Datics
- Integrations
- NPI Registry API, Google Places and Geocoding APIs with the client's keys, folium for the map and Postgres for batches and state.
- Deployment
- Docker on Amazon EC2 inside the client's AWS account, where public traffic is denied at the VPC level, on a t3.medium after smaller instances ran out of memory.
- API limits
- The NPI API returns at most 1,200 results and Places up to 20 per call, so city wide queries may later use downloaded NPI files.
Data and retrieval
Cloud and infrastructure
Apps and APIs
Also in the stack
How it shipped
First crawl
A Crawl4AI scraper with LLM extraction to JSON and duplicate removal, tested on the first practice sites within days of the brief.
Parallel and modular
Parallel crawling, the sitemap fallback and the split into a discovery module and a crawling module, then deployment in Docker on EC2.
The client's data model
We adopted the client's provider data model, added the fields its team asked for and tested a structure based extraction method without an LLM.
Registry and map
NPI Registry enrichment, Google Places matching by name and address, a first provider map, and schema and sequence diagrams.
Toward the data platform
Direct loading into Snowflake, offered side by side with the existing database at first, a queue system and a HIPAA aligned hosting design.
Inside the client's AWS account
We built and deployed within the client's own AWS account and its network rules, and designed the next hosting step against the HIPAA controls its team already tracks.
Their data model, their calls
We adopted the client's provider data model, added fields as its team asked, and flagged where its diagram, which started from NPI practice locations, differed from our matching approach, so the client decided.
Phased, reviewed weekly
Each module shipped and was reviewed in weekly syncs with the client's team before the next one started.
What this unlocked
Next on the same data: loading verified provider records straight into Snowflake, with a queue system in front of the crawler.
Related work
Have a product with the same kind of constraints?
Tell us what already exists and what you want it to do. We will tell you what it takes.