Crawling your own site and flagging broken links or missing meta tags shares this register's Screaming Frog build spec — genuinely bounded, well-understood work for a personal or small-business site.
Build it
Effort: a weekendRunning cost: $0/mo to start
Why CanICodeThis says Build it
A crawler following internal links from a starting URL, recording each page's status code, is a well-documented pattern
Checking each page for a title tag, meta description, and H1 is straightforward HTML parsing
A report listing issues found is ordinary CRUD and export
The parts that'll cause problems
Comprehensive technical SEO rule sets (duplicate content, canonical validation, structured data errors) accumulate into specialist depth over years
Crawling at real scale without overwhelming the target site needs rate-limiting care
JavaScript-rendered pages need a headless browser, a heavier dependency than simple HTML fetching
How I'd build this
frontend
Astro or Next.js
database
Turso (SQLite)
hosting
Cloudflare Pages
MVP scope
Crawl internal links from a starting URL
Record status code, title, meta description, H1 per page
A report flagging broken links and missing tags
Postpone to v2
JavaScript-rendered page crawling
Duplicate content and canonical tag validation
Scheduled recurring crawls with change alerts
What it actually costs to build
AI agent — with a subscription (Claude Pro/Max, Cursor, etc.)
$0 marginal
AI agent — pay-per-use API, no subscription
$43–$86 one-time
Hosting, once it's running
$0/mo (free tier)
Domain name, if you want your own
~$12/yr
Existing tools solving a similar problem typically run $22–$139.95/mo — see how they compare below.
The build prompt
Build me a site crawler: starting from a URL, follow internal links, record each page's status code, title tag, meta description, and H1. A report flagging broken links (4xx/5xx) and pages missing a title or meta description. Respect crawl-delay and robots.txt. Skip JavaScript rendering for v1.