Problem: the single `do_login_and_submit` page action navigated between
pages within one `session.fetch` call, which was fragile and couldn't
leverage `solve_cloudflare` for the Turnstile gate on the submit page.
The submit button click also blocked on navigation completion, causing
timeouts when CF was slow to process.
Solution: split into three separate `session.fetch` calls (homepage
login check, `/enter` login, `/contest/{id}/submit`) with
`solve_cloudflare=True` on login and submit. Use `no_wait_after=True`
on the submit click with a doubled nav timeout. Extract `span.error`
text on submit failure instead of a generic timeout message.
Problem: `_solve_turnstile` looped 6 times with ~20s per iteration
(15s bounding_box timeout + 5s wait_for_function) when no Turnstile
iframe existed on the page, causing a 120-second delay on pages that
don't require Turnstile verification.
Solution: check for existing token and iframe presence before entering
the retry loop. `iframe_loc.count()` returns immediately when no
matching elements exist, avoiding the expensive timeout cascade.
Problem: each scraper defined its own timeout constants
(`TIMEOUT_S`, `TIMEOUT_SECONDS`) with inconsistent values (15s vs 30s)
and browser timeouts were scattered as magic numbers (60000, 15000,
5000, 500).
Solution: introduce `scrapers/timeouts.py` with named constants for
HTTP requests, browser session/navigation/element/turnstile/settle
timeouts, and submission polling. All six scrapers now import from
the shared module.
Problem: Codeforces submit was a stub returning "not yet implemented".
Solution: use StealthySession (same pattern as AtCoder) to handle
Cloudflare Turnstile on the login page, fill credentials, navigate to
the contest submit form, upload source via file input, and cache
cookies at `~/.cache/cp-nvim/codeforces-cookies.json` so repeat
submits skip the login entirely. Uses a single browser page action
that checks for the submit form before navigating, avoiding redundant
page loads and Turnstile challenges.
Problem: every `:CP submit` on CSES ran the full 5-request login flow
(~1.5 s overhead) even when the token from a previous submit was still
valid.
Solution: persist the API token in credentials via a `credentials`
ndjson event. On subsequent submits, validate the cached token with a
single GET before falling back to the full login.
## Problem
CSES submit was a stub returning "not yet implemented".
## Solution
Authenticate via web login + API token bridge (POST `/login` form, then
POST `/api/login` and confirm the auth page), submit source to
`/api/courses/problemset/submissions` with base64-encoded content, and
poll for verdict. Uses the same username/password credential model as
AtCoder — no browser dependencies needed. Tested end-to-end with a real
CSES account (verdict: `ACCEPTED`).
Also updates `scraper.lua` to pass the full ndjson event object to
`on_status` and handle `credentials` events for future platform use.
## Problem
`_submit_sync` was a 170-line nested closure with `_solve_turnstile` and
the browser-install block further nested inside it. Status events went
to
stderr, which `run_scraper()` silently discards, leaving the user with a
10–30s silent hang after credential entry. The NDJSON spawn path also
lacked stdin support, so submit had no streaming path at all.
## Solution
Extract `_TURNSTILE_JS`, `_solve_turnstile`, `_ensure_browser`, and
`_submit_headless` to module level in `atcoder.py`; status events
(`installing_browser`, `checking_login`, `logging_in`, `submitting`) now
print to stdout as NDJSON. Add stdin pipe support to the NDJSON spawn
path in `scraper.lua` and switch `M.submit` to streaming with an
`on_status` callback. Wire `on_status` in `submit.lua` to fire
`vim.notify` for each phase transition.
Problem: scrape_contest_list paginated the entire Kattis problem database
(3000+ problems) treating each as a "contest". scrape_contest_metadata
only handled single-problem access. stream_tests_for_category_async could
not fetch tests for multiple problems in a real contest.
Solution: replace the paginated problem loop with a single GET to
/contests that returns ~150 real timed contests. Add contest-aware path
to scrape_contest_metadata that fetches /contests/{id}/problems and
returns all problem slugs; fall back to single-problem path when the ID
is not a contest. Add _stream_single_problem helper and update
stream_tests_for_category_async to fan out concurrently over all contest
problem slugs before falling back to the single-problem path.
Problem: lua typecheck flagged missing start_time field on ContestSummary;
ty flagged BeautifulSoup Tag/NavigableString union on csrf_input.get(),
a 3-tuple unpack where _extract_problem_info now returns 4 values in
cses.py, and an untyped list assignment in usaco.py.
Solution: add start_time? to ContestSummary LuaDoc, guard csrf_input
with hasattr check and type: ignore, unpack precision from
_extract_problem_info in cses.py callers, and use cast() in usaco.py.
Add submit.lua that reads credentials from a local JSON store (prompting
via vim.ui.input/inputsecret on first use), reads the source file, and
delegates to scraper.submit(). Add language_ids.py with platform-to-
language-ID mappings for atcoder, codeforces, and cses.
Add KattisScraper and USACOScraper with contest list, metadata, and
test case fetching. Register kattis and usaco in PLATFORMS,
PLATFORM_DISPLAY_NAMES, and default platform configs.
Problem: problem pages contain floating-point precision requirements and
contest start timestamps that were not being extracted or stored. The
submit workflow also needed a foundation in the scraper layer.
Solution: add extract_precision() to base.py and propagate through all
scrapers into cache. Add start_time to ContestSummary and extract it
from AtCoder and Codeforces. Add SubmitResult model, abstract submit()
method, submit CLI case with get_language_id() resolution, stdin/env_extra
support in run_scraper, and a full AtCoder submit implementation; stub
the remaining platforms.
