New York Surrogate Court Scraper

What I built

• Connected to an already-running Chrome instance via the Chrome DevTools Protocol, inheriting a session that had already cleared Cloudflare's invisible Turnstile behavioral checks — no exposed sitekey needed, and no external CAPTCHA solver.

• Iterated every New York county, filtering Probate & Administration proceeding types for each.

• Bypassed the portal's hard result-count limit by generating and searching granular monthly date ranges instead of one wide query per county.

• Handled multi-page result sets by tracking active pagination state and managing browser history (go_back) to keep navigation state consistent across pages.

• Wrote a custom retry mechanism with configurable backoff for the frequently unstable government portal, and an append-mode CSV writer that survives interruption mid-run.

Stack