Canadian Charity Annual Report Scraper

What I built

• Built a layered discovery workflow that walks each charity's site — homepage → About Us → Accountability / Resources / Publications — scanning every page visited for either a direct report PDF link or a dedicated 'reports' landing page, so no single fixed URL pattern is assumed.

• Designed a ranked, keyword-driven matching system (centralized in config.py) that checks URL, link text, and up to 500 characters of surrounding DOM context against curated keyword lists — entirely regex and BeautifulSoup, with zero LLM or paid API calls involved.

• Implemented latest-year resolution that parses years (including ranges like '2025-2026') out of every candidate PDF URL and automatically keeps the most recent report, while filtering out financial statements flagged by filename pattern so they're never mis-filed as annual reports.

• Engineered resilient HTTP handling — automatic retry with a spoofed browser User-Agent and extended timeout for sites that block default Python request headers, plus chunked-encoding-safe file downloads with backoff — so one unresponsive site never stalls the batch.

• Ran the pipeline end-to-end across 1,700+ Canadian charities, writing Found / Not Found / Downloading Error back into a CSV ledger and organizing every downloaded PDF into a Business-Number/year-keyed folder structure, with idempotent re-runs that skip already-resolved charities.

• Instrumented structured logging (separate info and error logs) capturing every request, discovery decision, download, and retry, keeping a 1,700+ site unattended batch run fully debuggable.

Stack