Skip to main content

​
0006. Authenticate programmatic API clients with personal access tokens

​
Context and problem statement

Using the API promises that researchers and developers can drive the whole co-construction loop over plain HTTP, and documented the entry point as POST /api/auth/register followed by POST /api/auth/login. Against the live deployment that promise did not hold: both endpoints call verify_recaptcha() before touching the database, so a request carrying no captcha token is rejected with {"detail":"CAPTCHA verification failed."}. This was not a misconfiguration to correct. reCAPTCHA v3 tokens are minted browser-side by the widget, are bound to an action, and are single-use, so no HTTP client can produce one; the documented flow was unreachable by construction for exactly the audience the page addressed. A second gate compounded it: register creates the account with email_verified=False and login returns 403 until the emailed link is followed, an undocumented step a caller would stall at even if the captcha were somehow satisfied. All four public token-issuing routes are gated the same way, including google-auth, so there was no side door either. The tension is that both gates are correct where they are: reCAPTCHA stops bots mass-registering accounts and credential-stuffing the login form, and email verification proves the address is real before an account becomes usable. Weakening either to unblock scripting would trade a genuine abuse control for developer convenience. The rest of the surface was healthy (every protected route rejected an unauthenticated call cleanly), which sharpens the diagnosis: only credential acquisition was broken, and programmatic clients were being routed through an interactive-only door.

​
Decision drivers

  • Do not weaken the interactive gates. reCAPTCHA on register/login and mandatory email verification must survive untouched.
  • A credential a script can obtain and hold. Long-lived, with no browser in the loop after issuance.
  • Independent revocation. Containing one leaked credential must not disturb any other user or session.
  • No churn on existing routes. Every deployed endpoint already depends on require_user; the change must land underneath them.
  • Auditable and attributable. A research deployment needs to separate scripted traffic from study traffic in the existing activity log.
  • Fits a public research project. The audience is researchers scripting against their own account, not third-party applications acting on behalf of other users.

​
Considered options

  1. Personal access tokens (PATs). A long-lived, hashed, revocable key minted from an authenticated session and sent as a bearer credential.
  2. Exempt API clients from captcha. Keep register/login as the entry point and relax the check by IP allowlist, shared bypass token, or header opt-out.
  3. Document the session JWT. Promote localStorage["haico-auth-token"] to a supported credential.
  4. OAuth 2.0 authorization code flow. Full delegated authorization for third-party apps.

​
Decision outcome

Chosen option: personal access tokens, because it is the only option that unblocks programmatic access by adding the credential type such access was always meant to use, rather than by removing a control or publishing an internal one. The design:
  • The gates are inherited, not bypassed. A token can only be minted from a logged-in session, and logging in already requires a verified email and a passed captcha, so both checks are enforced exactly once, interactively, at issuance, and every later programmatic call inherits that assurance. Nothing about /auth/register or /auth/login changes. This also settles whether captcha should be “disabled” for API callers: there is nothing to disable, because captcha guards two unauthenticated human-facing endpoints that token holders never call. The programmatic plane gets per-key rate limiting, scopes, expiry, revocation, and usage logging instead.
  • One insertion point. The branch lives in get_current_user (app/routers/deps.py), which already extracts a bearer credential: a token carrying the haico_pat_ marker goes to the key verifier, everything else takes the untouched jwt.decode path. Every existing endpoint therefore accepts keys with no route changes at all.
  • SHA-256 at rest, looked up by an indexed prefix. A password hash’s work factor exists to make offline brute force of a low-entropy human secret expensive; against 256 bits of CSPRNG output it buys nothing, adds ~100 ms to every authenticated request, and bcrypt would silently truncate past 72 bytes. Digests are compared with hmac.compare_digest. The row is found by a unique indexed key_prefix (the marker plus the first 8 secret characters), never by scanning.
  • The haico_pat_ marker is functional, not cosmetic. A fixed, searchable prefix is what lets GitHub secret scanning and push protection recognise a leaked key, which is why every comparable product uses one (ghp_, sk_live_, xoxb-).
  • Scopes derived from the HTTP method. A key carries read or read,write, and anything outside GET/HEAD/OPTIONS requires write. Every non-GET route in the app is a genuine mutation, so this enforces scopes in the same single place rather than scattering a dependency across ~30 routes.
  • /api/keys refuses key-authenticated callers. If a key could mint keys, one leaked credential could issue replacements, outlive its own revocation, and escalate a read-only grant into a writable one. Revocation is a soft delete (revoked_at) so the audit trail survives, and last_used_at is written at most once per minute per key so authenticating does not add a database write to the hot path.
  • Rate limiting per key, in the application. The nginx haico_api zone limits per IP, which is the wrong unit for programmatic traffic: one CI host is a single address running many legitimate keys, while an abuser rotates addresses freely.
The resulting model has two planes: The frontend keeps using the session JWT and must never use a PAT. A long-lived key in localStorage is reachable by any XSS and by anyone who opens devtools, and unlike a 24-hour JWT it does not expire out of an attacker’s hands. Keeping the planes separate is also what makes per-key revocation and separate rate limits meaningful. Both credentials authenticate, and that is normal: GitHub accepts session cookies for the web UI and personal access tokens for the API, and Stripe does the equivalent. The distinction is not which one the server accepts, it is which one is specified, supported, and documented.

​
Positive consequences

  • The documented flow becomes true: a developer with no source access can drive the full loop over HTTP.
  • Leaked credentials are contained by revoking one row, with no effect on other users or sessions.
  • The session JWT stays a private implementation detail, so shortening its lifetime, rotating JWT_SECRET, adding refresh tokens, or migrating to httpOnly cookies all remain non-breaking. The httpOnly migration matters most, because it is the more secure option and it would delete the localStorage trick outright.
  • Programmatic traffic is attributable per key in user_activity, which a research deployment needs in order to separate study traffic from scripted traffic.
  • The documentation playground becomes usable, since a key can be pasted into its auth field where /api/auth/login could never have succeeded.

​
Negative consequences

  • A long-lived credential now exists. Mitigated by hashing at rest, a 90-day default expiry, one-time display, a per-user cap on active keys, and a visible last_used_at.
  • Two credential types reach get_current_user, so that branch is a place authentication bugs could hide; it is covered by tests asserting the JWT path and the key path side by side.
  • Users can now leak a durable secret into a public repository. Mitigated by the scannable haico_pat_ prefix.
  • Rate-limit counters are held in process memory, so running multiple workers multiplies the effective ceiling and a restart clears them. Acceptable at this deployment’s shape; the fix is a shared counter behind the same interface.
  • New surface to maintain: a table, a router, a settings screen, and docs.

​
Pros and cons of the options

​
Option 1: personal access tokens

  • + Industry standard: GitHub, Stripe, OpenAI, Anthropic, Slack, and Linear all mint API credentials from an authenticated settings page, display them once, and support revocation.
  • + Inherits captcha and email verification instead of bypassing them.
  • + Individually revocable, scopable, expirable, rate-limitable, auditable.
  • + Lands underneath the existing require_user dependency, so no route changes.
  • − Requires a migration, a router, and UI work, and introduces a durable secret users can mishandle.

​
Option 2: exempt API clients from captcha

  • + Cheapest to implement.
  • − Reopens exactly the abuse vector captcha was added to close. An IP allowlist does not scale to a public research tool, and a shared bypass token is a single secret whose leak silently disables the control for everyone.
  • − Leaves email verification still blocking, so it does not even finish the job.
  • − Still hands out a 24-hour credential, so callers must re-authenticate constantly.

​
Option 3: document the session JWT

  • + Zero implementation cost; it already works today.
  • − Not obtainable without a browser, a human, and devtools, repeated every 24 hours, so the “API” is not automatable end to end.
  • − Freezes session internals into the public contract.
  • − No individual revocation: containment means rotating JWT_SECRET and logging every user out at once.
  • − Teaches a phishable habit. “Copy the token out of your browser storage” is the shape of a well-known attack, and a bad reflex to train into a research user base.

​
Option 4: OAuth 2.0 authorization code flow

  • + The correct answer when third-party applications need delegated access on behalf of other users, with consent screens and scoped grants.
  • − Solves a problem this project does not have; client registration, redirect URIs, and refresh-token handling are pure overhead on both sides for researchers scripting against their own account.
  • − Substantially more surface to build and secure.
  • − Can be added later without invalidating PATs; the two coexist in every product cited above.