Commit graph

21 commits

Author SHA1 Message Date
Barrett Ruth
0b40e0f33e
fix: replace curl_cffi with scrapling in codeforces metadata (#334)
## Problem

`codeforces.py` used `curl_cffi` to bypass Cloudflare when fetching
contest problem HTML, making it unavailable in the nix python env and
requiring an extra dependency across `pyproject.toml` and `flake.nix`.

## Solution

Rewrite `_fetch_problems_html` to use scrapling `StealthySession` with
`solve_cloudflare=True`, matching the existing CF submit pattern. Extend
`needs_browser` in `scraper.lua` to route CF `metadata` and `tests`
through the FHS env on NixOS. Remove `curl-cffi` from `pyproject.toml`,
`flake.nix`, and test mocks.
2026-03-06 13:25:44 -05:00
Barrett Ruth
c194f12eee
feat(atcoder): extract submit helpers; add live status notifications (#294)
## Problem

`_submit_sync` was a 170-line nested closure with `_solve_turnstile` and
the browser-install block further nested inside it. Status events went
to
stderr, which `run_scraper()` silently discards, leaving the user with a
10–30s silent hang after credential entry. The NDJSON spawn path also
lacked stdin support, so submit had no streaming path at all.

## Solution

Extract `_TURNSTILE_JS`, `_solve_turnstile`, `_ensure_browser`, and
`_submit_headless` to module level in `atcoder.py`; status events
(`installing_browser`, `checking_login`, `logging_in`, `submitting`) now
print to stdout as NDJSON. Add stdin pipe support to the NDJSON spawn
path in `scraper.lua` and switch `M.submit` to streaming with an
`on_status` callback. Wire `on_status` in `submit.lua` to fire
`vim.notify` for each phase transition.
2026-03-04 19:27:29 -05:00
1162e7046b try to fix the setup 2026-02-18 14:13:37 -05:00
f805251762 some misc fixes 2025-12-07 16:09:17 -06:00
Barrett Ruth
c9769e04b8 x 2025-11-28 01:30:48 -05:00
Barrett Ruth
2fda5a74ca feat: codechef 2025-10-25 00:26:33 -04:00
c509102b37 feat(tests): basic tests 2025-10-05 21:58:43 -04:00
f929c8e826 feat(scrapers/atcoder): atcoder scraper 2025-10-03 23:26:09 -04:00
179b333505 update pyproject 2025-10-03 22:38:24 -04:00
4498c4a7fa fix scrapers 2025-10-03 19:19:02 -04:00
1520939d4b some refactors 2025-10-03 14:34:49 -04:00
49ba922ff7 fix(scraper): use scrapling 2025-09-30 20:16:59 -04:00
abe078b73d fix(ci): temporarily add cloudscraper 2025-09-30 19:53:49 -04:00
df7896709f cloudscraper -> scrapy 2025-09-30 18:32:18 -04:00
46c615416f feat(scraper): use backoff 2025-09-21 11:26:54 -04:00
3c8b76207c feat(ci): pre-commit 2025-09-19 16:01:17 -04:00
5c2cc0d97d feat: ignore missing clourscrapers 2025-09-18 22:02:21 -04:00
51fd6e3676 feat(ci): pytest 2025-09-18 22:01:40 -04:00
ca6f8417c0 feat: scraper cleanup 2025-09-18 21:49:25 -04:00
5bf8c8960b feat: qol improvements 2025-09-18 21:28:34 -04:00
dcb7debff6 initial commit 2025-09-11 23:52:32 -05:00