opencli-adapter-author

opencli-adapter-author

You are an agent writing an adapter for a site. Goal: from zero to passing opencli browser verify within 30 minutes.

Use existing tools throughout: opencli browser * / opencli doctor / opencli browser init / opencli browser verify. No new commands.

When debugging a browser-type adapter, prefer running with --trace on --keep-tab true --window foreground. --trace on writes trace artifacts each round; summary.md is the entry point for failure/success review. --keep-tab true --window foreground keeps the tab lease alive and the browser window in the foreground for easy final page state inspection.

---

Prerequisite: Where you are

Quick self-test with coverage-matrix.md. Three questions:

1. Is the data visible in the browser? (No → resolve auth first)
2. Is the data HTTP/JSON/HTML? (No → out of scope)
3. Does it require real-time push? (Yes → find an HTTP endpoint for the same data; if none, give up)

If all three are yes, continue.

---

Top-level decision tree

Decide strategy first, then write adapter. Before writing code after entering Step 3/4, you must produce a strategy note. Without this note, do not start writing clis/<site>/<name>.js.

The core judgment is not "API is better than DOM", but whether the data source has an external contract. Maintenance cost data shows: public/official APIs are most stable; UI/DOM semantics usually have a user-visible contract; undocumented internal XHR/GraphQL/signed endpoints drift most easily. Do not blindly migrate stable UI/DOM implementations to contract-free internal APIs just to be "API-first".

Strategy: PUBLIC_API | COOKIE_API | PAGE_FETCH | INTERCEPT | DOM_STATE | UI_SELECTOR
Contract: stable | visible-ui | internal-unstable
Evidence:
- observed request/state: <endpoint / state global / UI-only signal>
- auth source: <none / browser cookie / csrf from meta / localStorage / page runtime>
- replay result: <status + content-type + non-empty sample shape>

If Strategy is PAGE_FETCH or INTERCEPT:
- why PUBLIC_API / COOKIE_API are unavailable:
- why UI_SELECTOR / DOM_STATE are not safer:
- why the maintenance cost is acceptable:

Strategy classes:

Strategy	Contract level	When to use	Evidence required
PUBLIC_API	stable	No login needed; Node-side fetch directly gets target data	200 + JSON/HTML containing target data, not tracking/ads
COOKIE_API	stable	Node-side fetch + page.getCookies() / header helper can get data	cookie/CSRF source clear, replay non-empty
UI_SELECTOR	visible-ui	Publish/upload/click/forms, or page semantics more stable than internal APIs	selector has semantic anchor; error path is typed error
DOM_STATE	visible-ui	Data in hydration state / bootstrap JSON / SSR HTML	state key / script JSON / HTML structure clear
PAGE_FETCH	internal-unstable	Only fetch in page context can reuse same-origin/session/runtime	opencli browser eval fetch(...) non-empty; must explain why internal API is unavoidable
INTERCEPT	internal-unstable	Request signing is complex, but the page naturally makes the request	Can intercept target response after triggering UI; must explain why UI/DOM is insufficient

Selection rules: Prefer PUBLIC_API / COOKIE_API. If UI/DOM semantics are stable, do not force upgrade to PAGE_FETCH / INTERCEPT. Only take on the maintenance cost of contract-free internal APIs when public/official APIs are unavailable and UI/DOM cannot express the target data or operation.

Empirical data: PAGE_FETCH / INTERCEPT fix frequency is about 7-8 times that of PUBLIC_API; UI_SELECTOR is on par with COOKIE_API. Detailed ladder derivation, how to fill api_candidates evidence, and counterexamples like booking #1680 are in references/strategy-selection.md.

Boundary: Only reuse data/capabilities the page has legitimately obtained. Do not teach signature cracking, bypass captchas/risk controls/access controls; if encountering non-reusable signatures (e.g., must be generated by page runtime and cannot be safely abstracted), downgrade to UI_SELECTOR / DOM_STATE / INTERCEPT.

START
  │
  ▼
┌──────────────────────────┐
│ opencli doctor passes?   │── no ──→ Fix bridge (hints in doctor output)
└──────────────────────────┘
  │ yes
  ▼
┌────────────────────────────────────────────────────┐
│ Read site memory:                                   │
│   1. ~/.opencli/sites/<site>/endpoints.json         │
│   2. ~/.opencli/sites/<site>/notes.md               │
│   3. references/site-memory/<site>.md               │
└────────────────────────────────────────────────────┘
  │ Hit endpoint + fields → jump to [endpoint verify] (do not skip writing adapter! memory may be stale)
  │ No hit → continue
  ▼
┌──────────────────────────┐
│ Site recon (site-recon)   │  → Pattern A/B/C/D/E
└──────────────────────────┘
  │
  ▼
┌──────────────────────────┐
│ API discovery (api-discovery)│  §1 network → §2 state → §3 bundle → §4 token → §5 intercept
└──────────────────────────┘
  │ Got candidate endpoint
  ▼
┌────────────────────────────────────────────┐
│ Direct fetch verify endpoint (run even if memory hit)│── 401/403 ──→ back to §4 resolve token
│ Data non-empty + 200                              │── empty/HTML ──→ back to site-recon change Pattern
│ Is memory value still alive?                     │── site version change ──→ mark old endpoint, back to api-discovery
└────────────────────────────────────────────┘
  │ OK
  ▼
┌───────────────────────────────────────┐
│ Field decode (spot-check field-map in memory too)│  Self-explanatory → direct / known code → field-conventions / unknown → decode-playbook
│ Compare one known field with visible web value to confirm no misalignment     │
└───────────────────────────────────────┘
  │
  ▼
┌──────────────────────────┐
│ Design columns (output)   │  Follow output-design.md naming / types / order
└──────────────────────────┘
  │
  ▼
┌──────────────────────────┐
│ opencli browser init      │  Generate ~/.opencli/clis/<site>/<name>.js skeleton
│ Copy most similar neighbor adapter    │
│ Change name / URL / mapping three places  │
└──────────────────────────┘
  │
  ▼
┌──────────────────────────┐
│ opencli browser verify    │── fail ──→ autofix skill, use --trace retain-on-failure back to corresponding step
└──────────────────────────┘
  │ success
  ▼
┌──────────────────────────┐
│ Visually compare fields vs web page   │── values wrong ──→ back to field decode
└──────────────────────────┘
  │ match
  ▼
┌──────────────────────────┐
│ Write back ~/.opencli/sites/   │  endpoints / field-map / notes / fixtures
└──────────────────────────┘
  │
  ▼
DONE

---

Runbook (check step by step)

[ ] 1. opencli doctor returns "Everything looks good"
[ ] 2. Read site memory:
       [ ] ~/.opencli/sites/<site>/endpoints.json exists? Contains desired endpoint?
       [ ] references/site-memory/<site>.md exists? Check "Known endpoints" section
       [ ] If hit: **jump to Step 5 (endpoint verify) + Step 7 (field check)**, do not jump directly to Step 9 write adapter
       [ ] Memory written >30 days ago (check `verified_at`) → treat as stale, cold start via Step 3 → 4
[ ] 3. Recon (site-recon.md):
       [ ] **Preferred**: `opencli browser analyze <url>` to get pattern + anti-scrape + nearest adapter + next step in one go
       [ ] If `analyze` conclusion is vague, manually run: `open` → `wait time 2` (or `wait xhr <regex>`) → `network`
       [ ] Determine Pattern (A / B / C / D / E)
[ ] 4. API discovery (api-discovery.md) select § by Pattern:
       [ ] Pattern A → §1 network deep read
       [ ] Pattern B → §2 state extraction + §1 deep data
       [ ] Pattern C → §3 bundle / script src search
       [ ] Pattern D → §4 token source + downgrade §5
       [ ] Pattern E → find HTTP polling endpoint; only if not found then §5
[ ] 5. Direct fetch verify candidate endpoint:
       [ ] Returns 200
       [ ] Response contains target data (not HTML / ads)
[ ] 6. Write strategy note (mandatory before writing code):
       [ ] Choose from `PUBLIC_API / COOKIE_API / PAGE_FETCH / INTERCEPT / DOM_STATE / UI_SELECTOR`
       [ ] Fill Contract: `stable / visible-ui / internal-unstable`
       [ ] Fill Evidence: observed request/state, auth source, replay result
       [ ] If choosing `PAGE_FETCH` / `INTERCEPT`, must explain why `PUBLIC_API` / `COOKIE_API` / `UI_SELECTOR` / `DOM_STATE` are all unsuitable
       [ ] If choosing `UI_SELECTOR` / `DOM_STATE`, no need to over-justify "why not API"; just explain semantic anchor and typed error path
[ ] 7. Field decode:
       [ ] Self-explanatory → use key directly
       [ ] Known code → field-conventions.md lookup
       [ ] Unknown code → field-decode-playbook.md (sort key comparison / structural diff / constant elimination)
[ ] 8. Design columns (output-design.md):
       [ ] Naming camelCase and aligned with neighbor adapters
       [ ] Types / units / percentage format clear
       [ ] Order: identifier column → business numbers → metadata
[ ] 9. Write adapter (adapter-template.md):
       [ ] opencli browser init <site>/<name>
       [ ] Find most similar adapter from same site or same type, cp it over
       [ ] Change name / URL / field mapping
[ ] 10. opencli browser verify <site>/<name>
        [ ] After first pass, immediately `--write-fixture` to generate `~/.opencli/sites/<site>/verify/<cmd>.json` seed
        [ ] Manually edit seed: add `patterns` (URL / date / ID format) + `notEmpty` (core fields) + tighten `rowCount`
        [ ] Run `opencli browser verify <site>/<name>` again, confirm ✓ matches fixture
[ ] 11. Visually compare field values vs web page (don't just look at "Adapter works!")
[ ] 12. Write back site memory (**after verify passes + visual comparison matches**, schema in `references/site-memory.md`):
        [ ] `endpoints.json`: key = short name of endpoint, value = `{url, method, params.{required,optional}, response, verified_at: YYYY-MM-DD, notes}`
        [ ] `field-map.json`: only append new codes. key = field code, value = `{meaning, verified_at: YYYY-MM-DD, source}`; **do not overwrite existing keys**, if conflict, align with web visual value first then write
        [ ] `notes.md`: prepend a section `## YYYY-MM-DD by <agent/user>`, write new pitfalls / conclusions encountered while writing this adapter
        [ ] `verify/<cmd>.json`: **Required.** Expected values for `opencli browser verify` (args / rowCount / columns / types / patterns / notEmpty), already generated in Step 10, just checklist here
        [ ] `fixtures/<cmd>-<YYYYMMDDHHMM>.json`: Save a full response sample of this endpoint (remove cookie / token / user private fields before saving), for future field comparison / offline replay
        [ ] If temporary files were dumped during debugging in repo / adapter directory (`.dbg-*.html` / `raw-*.json` / etc.), **clean them before commit** — they should have been placed in `~/.opencli/sites/<site>/fixtures/` or `/tmp/`

---

Degradation paths (where to jump if stuck)

Stuck at	Symptom	Jump to
Step 4 API discovery	network empty, __INITIAL_STATE__ also empty	§3 bundle search baseURL
	bundle search no baseURL	§5 intercept
Step 5 endpoint verify	401 / 403	§4 token investigation
	200 but response is HTML	Back to Step 3 change Pattern
	200 but data: [] empty	Wrong params / endpoint version changed, back to §1 check real request headers in network
Step 7 field decode	Sort key comparison inconclusive	field-decode-playbook.md §3 structural diff
	Still inconclusive	Output raw first, iterate after adapter runs
Step 10 verify fail	fltt missing / field mapping wrong	autofix skill; add --trace retain-on-failure to reproduce command
	A column always null	Wrong field path, back to Step 7
Step 10 verify fixture mismatch	[pattern] row[i] error	First visually compare with web value; if value correct → fixture pattern too strict, loosen; if value wrong → field mapping wrong
	[column] missing column "X"	Actual response missing this column (site version change or args effect); re---update-fixture or fix adapter
	[type] actual null / undefined	Field extraction failed, back to Step 7 re-extract; temporary fallback union type string|null only when semantics truly nullable
Step 11 values wrong	Off by factor 10000	Unit mismatch ("万" vs "元")
	Percentage off by factor 100	Response already 0.025, do not × 100

---

Reference files

File	When to consult
references/coverage-matrix.md	Before starting: self-test "is it in scope"
references/site-recon.md	Step 3 determine site type
references/api-discovery.md	Step 4 find endpoint
references/strategy-selection.md	Before Step 6 fill strategy note: contract model + empirical fix frequency + api_candidates evidence usage + counterexamples
references/field-conventions.md	Step 7 look up known field codes
references/field-decode-playbook.md	Step 7 when field not in dictionary
references/output-design.md	Step 8 naming / types / order
references/adapter-template.md	Step 9 file structure + live example convertible.js
references/site-memory.md	Overview: in-repo seeds + local ~/.opencli/sites/ two-layer structure
references/site-memory/<site>.md	Step 2 read site public knowledge (eastmoney / xueqiu / bilibili / tonghuashun already seeded)
references/success-rate-pitfalls.md	Before Step 7 / 11: 11 silent failures where "verify passes but data is wrong" (including aria-label locale-dependence)
references/jsdom-fixture-pattern.md	When adapter uses DOM extraction inside page.evaluate and mocked-evaluate unit tests miss silent bugs — freeze HTML into clis/<site>/__fixtures__/ and run with JSDOM (includes mandatory awk 'NF>0' tightening for fixture creation + reverse-validate discipline)
references/typed-errors.md	Must read before writing func body: 5 typed error landing table (ArgumentError / EmptyResultError / CommandExecutionError / AuthRequiredError / TimeoutError) + three silent anti-patterns (silent-clamp / sentinel-row / generic CliError) with fix examples

---

Key conventions

• Adapter only imports @jackwener/opencli/registry + @jackwener/opencli/errors, no third-party
• columns array and func return object keys must be fully aligned (including order)
• Intermediate parse object keys must not overlap with any columns entry (otherwise silent-column-drop audit misjudgment, PR #1329 R1 actually hit this; rename to dedicated names + destructure aliasing when pushing rows)
• browser: field determines func signature: browser:false → (args), browser:true → (page, args). If reversed, args is actually a debug flag and all external parameters silently fallback to defaults (PR #1329 upstream, 8 non-browser adapters all hit this)
• Known failures: throw corresponding typed error per references/typed-errors.md 5-classification; do not silently return [], do not silently return [{sentinel}], do not Math.max/min silently clamp external parameters
• Write personal adapters in ~/.opencli/clis/<site>/<name>.js (no build needed); only copy to clis/<site>/<name>.js when submitting a PR
• Site memory is rewritten each round: no memory → use skill → produce memory → next time becomes 5 minutes
• During debugging, raw dumps / captures / HTML samples must only be placed in ~/.opencli/sites/<site>/fixtures/ or /tmp/. It is strictly forbidden to leave .dbg-*.html / raw-*.json / sample.* temporary files in the repo root, clis/<site>/, or current working directory (they will appear in PR diffs, annoying reviewers).
• JSDOM unit-test fixtures (clis/<site>/__fixtures__/<command>.html) are an exception to the above — they are intentionally committed review artifacts, not temporary dumps. But therefore the quality bar is higher: must complete the 5 steps in references/jsdom-fixture-pattern.md (including mandatory awk 'NF>0' blank line tightening) and reverse-validate to prove the regression guard actually catches.

---

Stuck

• Diagnostic: opencli doctor → check notes.md → search autofix skill
• Field decode: field-decode-playbook.md all three sections → output raw and iterate
• Endpoint not found: api-discovery §5 intercept as fallback

Do not guess. If you guess wrong, verify may pass but data is wrong, and users will discover garbled output.