Case study · Data and automation
Ingestion, deduplication, scoring and a safe driver
Implemented · tests run (58/58) · local
Collecting leads by hand from directories is slow and error-prone. This pipeline gathers listings from three sources, merges repeats into one record, spots businesses with no website, scores each one transparently, and refuses to publish a list that contains test data. Its automation only plans until a person explicitly allows it to act.
Overview
A TypeScript and Node pipeline ingests listings from three public directories, through dataset imports, operator-side capture scripts, or a browser-automation agent driven over a local socket. Each source is normalised into a common lead schema.
A unified pass deduplicates records with a precedence chain of identity keys, merging repeats into the existing record, then classifies website presence and opportunity type and assigns a transparent additive score. A test-data guard refuses to publish a feed that contains synthetic records and quarantines suspicious captures, with a backed-up cleanup tool.
The automation driver is safe by default: it only plans unless explicitly authorised, caps batches, paces runs, takes detail targets only from a canonical list artifact, and turns typed failures such as a captcha or a credential prompt into hard stops. One descriptor table defines every source, so adding a source is a row, not a new driver. Its driver and detection tests pass (58 of 58, run during this review).