Building a Web Technology Detector with Scrapy¶
This project started in the simplest way possible: avoiding having to open 15 tabs manually just to understand a company's stack.
It's probably the project that taught me the most about real-world scraping, technical signals, the limits of the modern web, and the difference between a script that works and a system that holds up over time.
The original problem¶
In my B2B prospecting workflows, I was spending a lot of time doing the same thing over and over: open a site, inspect the source code, look at the loaded scripts, find marketing tags, quickly assess the level of technical maturity.
I quickly noticed that a website tells you a lot about a company: not just what technology it uses, but also how it works, its product culture, and sometimes even its level of technical debt.
| Detected signal | What it often reveals |
|---|---|
 Klaviyo |
Modern e-commerce stack |
 HubSpot |
Inbound / CRM approach |
 Segment |
Data culture |
 Heavy GTM usage |
Tracking that's become hard to maintain |
| No CMP | Low GDPR maturity |
 Shopify + Yotpo + Gorgias |
Pretty standard DTC stack |
At first, I was doing all of this by hand. And honestly, it became unbearable very quickly.
Why not use existing tools¶
I obviously looked at what was already out there:
| Tool | Main problem | |
|---|---|---|
 |
BuiltWith | Closed, no clean API, expensive at scale |
 |
Wappalyzer | Browser extension-oriented, poorly suited to automation |
 |
Klaviyo & co | Proprietary APIs, results can't be enriched within a workflow |
The real blocker
The existing tools were either:
- Too focused on being a "browser extension"
- Too closed for clean integration into workflows
- Too expensive at scale
- Impossible to enrich or modify quickly
- Little to no use in automation
And above all:
I wanted to understand exactly how detection worked, not just consume a result.
I wanted to be able to add my own signatures, modify the rules, understand false positives, and plug everything into other systems.
The early days: everything ran on my Windows PC¶
This is worth stating clearly, because many technical articles give the impression that everything starts with Kubernetes, distributed workers, and a clean cloud architecture.
In my case:
That was more than enough to get started.
The goal wasn't to crawl the internet. I mainly wanted to understand how far I could push a relatively simple detection engine.
The first version was ridiculous¶
For about 15 minutes, I thought this was brilliant.
Then the problems started: dynamically injected scripts, shared CDNs, technologies only detectable via cookies, JS frameworks returning near-empty HTML, false positives everywhere.
That's when the project changed character entirely.
The shift in perspective
A website is not an HTML page.
It's a collection of weak signals.
From that point on, I was no longer trying to answer:
but rather:
That completely changed the architecture of the system.
Overall architecture¶
The pipeline gradually evolved into this:
flowchart TD
A[Liste de domaines] --> B[Spider Scrapy]
B --> C[HTML] & D[Scripts] & E[Headers] & F[Cookies] & G[Meta tags]
C & D & E & F & G --> H[Moteur de signatures]
H --> I[Scoring]
I --> J[Technologies détectées]
J --> K[Export JSON] & L[Supabase] & M[n8n workflows]Important point: Scrapy detects nothing. It only collects signals. The business logic lives elsewhere.
Why Scrapy¶
Native performance
Even on a standard local machine, Scrapy handles solid volumes: several hundred pages per minute without aggressive optimization, as long as JS rendering is disabled.
Everything is already included: retries, throttling, concurrency, pipelines, middlewares.
Natural project structure
Very quickly, the repo started looking like something clean:
Scrapy almost naturally forces you to structure the project correctly.
| Tool | Why I ruled it out at this stage |
|---|---|
| Playwright alone | Too heavy to start with, too costly |
| Selenium | Slow, verbose, not suited to volume |
| requests + BeautifulSoup | Fine for one-off tasks, not for production scale |
| Crawl4AI | Too recent when the project started |
Some of them ended up in the pipeline eventually, but not at the beginning.
The JavaScript problem¶
This is probably the part that frustrated me the most.
At first, I absolutely wanted to avoid Playwright: heavy, resource-hungry, complicates everything.
The problem is that part of the modern web returns a nearly empty shell. On some Next.js sites, the crawler would literally retrieve:
And almost everything else arrived after hydration.
Direct consequence
Some technologies became completely invisible to pure Scrapy.
The compromise: a conditional Playwright fallback. Scrapy first, Playwright only when necessary.
flowchart LR
A[Scrapy] --> B{Détection suffisante ?}
B -->|Oui| C[Fin]
B -->|Non| D[Fallback Playwright]
D --> E[DOM rendu]
E --> F[Analyse complémentaire]from playwright.async_api import async_playwright
async def render_page(url):
async with async_playwright() as p:
browser = await p.chromium.launch()
page = await browser.new_page()
await page.goto(url)
html = await page.content()
await browser.close()
return html
The hardest part wasn't this code: it was deciding when to trigger the fallback, for which domains, and with what timeouts.
What this means in practice:
| Mode | Pages/min (approx.) | RAM | Local scalability |
|---|---|---|---|
| Scrapy only | ~300-400 | Low | Good |
| Scrapy + Playwright | ~20-50 | High | Difficult |
This is probably the most important technical trade-off in the whole project.
The real difficulty: signatures¶
Scraping turned out not to be the main problem. The real challenge was:
how to maintain a reliable signature engine.
Result: false positives everywhere, fragile detections, no confidence concept, impossible to maintain.
The real progress
I was no longer detecting "one occurrence".
I was accumulating evidence.
The actual scoring¶
I tested more statistical, more automated approaches. They mostly made the system harder to debug.
Today, the scoring remains deliberately readable:
def compute_confidence(matches):
score = 0
for match in matches:
if match["type"] == "cookie":
score += 0.45 # (1)!
elif match["type"] == "script":
score += 0.30 # (2)!
elif match["type"] == "html":
score += 0.10 # (3)!
return min(score, 1.0) # (4)!
- A cookie is very reliable: it requires explicit installation on the site.
- A CDN script is reliable but may be shared (Cloudflare, jsdelivr...).
- Raw HTML may contain the word by chance in editorial content.
- Capped at 1.0: beyond several matching signals, detection is near-certain.
It's not "intelligent". But it's stable, readable, and easy to evolve.
Real output example¶
{
"domain": "example-store.com",
"technologies": [
{
"name": "Shopify",
"confidence": 0.97,
"signals": ["cdn.shopify.com", "_shopify_y", "shopify-checkout"]
},
{
"name": "Klaviyo",
"confidence": 0.91,
"signals": ["static.klaviyo.com", "__kla_id"]
}
],
"meta": {
"crawl_time_ms": 1840,
"javascript_rendered": false,
"redirects": 1
}
}
This is where the project goes beyond being a simple "detector". With these signals, you can:
- score companies by their technical maturity,
- enrich AI workflows,
- build ICPs,
- detect stack changes over time,
- generate commercial signals.
What broke several times¶
Web data is extremely messy.
Absurd redirects, invalid certificates, timeouts, anti-bot protections, broken HTML, strange encodings: you spend far more time handling edge cases than writing business logic.
False positives were a nightmare
Classic example: the word segment can mean Segment CDP, a word in editorial content, or a URL slug.
Same problem with certain shared CDNs (Cloudflare, jsdelivr, Google APIs).
Detecting fewer technologies but with reliable signals is far preferable.
What surprised me after several thousand domains¶
Observations
- A huge number of sites are broken or half-functional
- Far more WordPress than expected, across all sectors
- Marketing stacks are very repetitive by industry
- Some sites load dozens of dead or legacy scripts
- The modern web is much more fragile than I had imagined
What I would do differently¶
What I had:
What I would do:
/signatures
/crm
hubspot.json
salesforce.json
/marketing
klaviyo.json
mailchimp.json
/analytics
segment.json
gtm.json
With validation, tests, scoring, and proper versioning from the start.
I spent a long time testing manually on random domains.
Classic mistake.
What I would build today:
- a reference list of domains with known stacks,
- HTML snapshots,
- automated regression tests.
Signatures age extremely fast. A technology can change its CDN, its script, or its domain overnight.
What is still held together with duct tape¶
JS rendering is expensive
Playwright significantly improves results. But RAM, CPU, and crawl time spike quickly. Today, the fallback remains difficult to scale cleanly.
Some technologies remain invisible
The system mainly sees what leaks on the frontend.
Anything backend, infrastructure, internal tooling, or APIs is much harder to detect. This is a structural limitation of HTML-side scraping.
What this project actually taught me¶
I thought I was building a technology detector. In reality, I mainly built a web signal collection system.
That's a very different thing.
What the project really taught me:
- robustness first,
- trade-offs between precision and cost,
- the real limits of scraping,
- and the importance of keeping systems simple and readable.
A comprehensible and stable pipeline is often worth more than an "intelligent" system that's impossible to maintain.
What I want to explore next
- Probabilistic scoring based on historical data
- Stack evolution over time (continuous monitoring)
- Technology clustering by sector
- Embeddings on detected signals
- Detection of technical maturity changes