Lead scraping pipeline
Lead generation for B2B sales teams · Interactive prototype with sample data, solo
Updated
Before
Sales built lead lists by hand. Simple scrapers died on the first block.
After
Jobs run to the end despite blocks and bad pages, with the leads ready through an API.
- 1Person:Operator
Sets up a job in a wizard, with target sites, fields and schedule.
- 2Software:Scraper
Opens pages in a real browser through rotating proxies.
- 3Software:Pipeline
Extracts the fields, checks them, removes duplicates and retries failures.
Person:OperatorWatches progress and the live log.
- 4Software:Platform
Stores the leads, then serves them through the API and webhooks.
Always on
- A failed page never stops the job.
- The dashboard shows proxy health and errors in real time.
- API keys are limited to one team.
Problem
Lead lists were built by hand. Scrapers at volume hit bot detection, failed requests and inconsistent pages, and stopped with half the data.
Diagnosis
At volume, failure is normal. The pipeline must expect blocks and bad pages, and keep going.
Options I weighed
What happens when a page fails?
Chosen
Retry it, then move it to a failed list and keep the job running
One bad page costs one record, not the whole run. Cost: the failed list needs someone to look at it.
Rejected
Stop the job on the first error
At volume some pages always fail, so the job would never finish.
Solution
The operator's side on sample jobs: a wizard for targets, fields and schedule, live progress and logs, proxy health, and the API and webhooks for the leads.
- Interface
- Next.js
- React
- Tailwind CSS
- Recharts
- Framer Motion
- Delivery
- Vitest
- Cloudflare Pages