RFC: Event-Driven Discovery & Agent-Facing Capability Search

Status: Draft · Scope: architecture / design only (no code in this RFC)

Turn x402DiscoveryAnalysis from a polling batch ETL into an event-driven discovery platform whose product surface is an RRF faceted search that agents call via a documented HTTP path and an llms.txt manifest, with a tool-rating loop that feeds ranking quality back into search.

The unit of value shifts from "rows in endpoints" to discoverable building blocks an agent can use — x402 endpoints today, MCP servers and other tools next.

1. Goals / Non-goals

Goals

• Emit a domain event the moment a new capability is discovered (x402 endpoint, MCP server, generic tool).
• Decouple discovery from downstream work (embed, cluster, analytics, index) so each reacts independently and is independently replayable.
• Expose a single agent-facing contract: GET /v1/search (RRF + facets) advertised through /llms.txt, optionally fronted by an MCP server.
• Capture quality signals — starting from probe HTTP status — and fold them into ranking as a tool rating.

Non-goals (this RFC)

• Rewriting clustering/revenue math. Those stay batch aggregations, event-triggered (debounced), not per-event recomputation.
• Picking a final broker vendor (a recommendation is given; the decision is an open question).
• Any code. This is the design contract teams build against.

2. Current state (grounding)

• etl service loops on a timer: run_cycle() → scrape → pricing → embed, all inline, all mutating one shared Postgres (src/x402_discovery/etl/main.py).
• The scraper already computes the discovery moment — the endpoint is None branch in EndpointScraper.run is "I found a new one" (src/x402_discovery/etl/scraper.py).
• catalog_events already models a discovery event log (event_type, resource_url, source, before_json, after_json) but is not currently written — it is a ready-made outbox table (src/x402_discovery/models.py).
• Embeddings already exist: endpoint_text.embedding is Vector(384) (pgvector) — the dense half of RRF is in place.
• The probe (probe_endpoint in src/x402_discovery/etl/x402_parse.py) already observes HTTP status (402, last_status, error_code) but does not persist it — a prerequisite for cold-start scoring (see §9).

3. Architecture overview

flowchart LR
    subgraph Producers
      S1[x402 catalog scraper]
      S2[MCP registry scraper]
      S3[other tool sources]
    end
    S1 & S2 & S3 -->|capability + outbox row<br/>one DB transaction| PG[(Postgres + catalog_events outbox)]
    PG -->|relay tails outbox| B{{Event broker}}
    B --> EMB[Embedding worker]
    B --> CLU[Clustering trigger<br/>debounced]
    B --> ANA[Analytics / revenue<br/>on ClusterRunCompleted]
    B --> IDX[Search index projection<br/>dense + sparse + facets]
    B --> SCORE[Tool-score projection<br/>Wilson / Bayesian + decay]
    PROBE[Probe health/status] -->|capability.health| B
    EMB --> B
    IDX --> SRCH[/v1/search RRF + /llms.txt + MCP/]
    SCORE -->|quality ranker| IDX
    SRCH --> AG[Agents]
    AG -->|tool.invoked / .succeeded / .failed / .rated| B

The broker is the backbone; Postgres becomes a set of projections (read models) rebuildable from the event log, not the source of truth for downstream stages.

4. Event model

4.1 Envelope (CloudEvents 1.0)

4.2 Event taxonomy

4.3 data payload for capability.*

Generalization note: the x402-specific endpoints table is fine as a producer-side store, but the search projection should be a kind-discriminated capabilities model so MCP servers and tools are first-class.

5. Producer: transactional outbox

Do not publish to the broker directly from the scraper (broker downtime would drop discoveries; dual-write is inconsistent). Instead:

1. In the same DB transaction that upserts a capability, write a catalog_events row (event_type = discovered|updated|removed, with before_json/after_json).
2. A small relay process tails undispatched catalog_events rows (poll, or Postgres LISTEN/NOTIFY, or logical-replication/CDC) and publishes them to the broker, marking each dispatched.

This gives exactly-once capture and at-least-once delivery with zero change to scrape correctness. catalog_events doubles as the audit log.

6. Broker (recommendation, not final)

Recommendation: NATS JetStream for the current footprint; revisit Redpanda if projection rebuild/replay becomes central.

7. Consumers & projections

• Embedding worker — on capability.discovered → embed → emit capability.embedded. (Peels the inline embed step out of run_cycle first; lowest risk.)
• Search index projection — maintains the dense + sparse + facet read model (§8) from capability.* / capability.embedded / tool.* events. Idempotent, rebuildable from the log.
• Tool-score projection — maintains tool_score from capability.health + tool.* events (§9).
• Clustering trigger — debounced: fire a run when "N new embedded capabilities since last run" or on a window; clustering itself stays batch and emits cluster.run_completed.
• Wallet sync — daily incremental Alchemy pull per pay_to wallet (per-wallet block cursor, global deduped wallet_transfers); emits wallet.volume_completed. Initial catalog bootstrap may use a one-shot full-window seed.
• Analytics attribution — debounced Postgres-only projection (bounds → certainty → revenue) on cluster.run_completed and wallet.volume_completed; reads transfers by rolling time window, not per sync run.

8. RRF faceted search

Reciprocal Rank Fusion merges several ranked lists:

rrf_score(d) = Σ_i  weight_i / (k + rank_i(d))      # k ≈ 60

Ranked lists (input rankers)

Facets (filter applied before/with fusion): kind, source, category, pay_to, network, price range, cluster label, certainty.

The index projection is downstream of events — never the system of record.

9. Tool rating

Ratings are events flowing back into the platform. The loop: search → agent uses a tool → outcome/rating events → tool_score projection → ranking quality → better search.

9.1 Cold-start: probe HTTP status → initial health score

Before any agent feedback exists, seed quality from what the probe already sees. The probe must persist the observed HTTP status (currently only transient in ProbeAttempt) — that is the one prerequisite.

These seed tool_score. Repeated probes update it (uptime ratio over a rolling window), so a flaky 5xx endpoint decays while a steady 402 stays high.

9.2 Aggregation (avoid small-sample bias)

• Combine implicit (tool.succeeded / tool.failed ratio, latency), explicit (tool.rated), and probe health.
• Use a Wilson lower bound or Bayesian average (prior = the §9.1 health seed) so a tool with 2 ratings does not outrank one with 200.
• Apply time decay so stale uptime and old ratings fade.

9.3 Feedback into RRF

tool_score enters §8 either as a third ranked list or a bounded multiplicative boost on rrf_score, so frequently-used, healthy building blocks surface higher over time.

10. Agent contract

• GET /v1/search — documented, stable. Query params: q, kind, category, source, network, price range, paging. Returns structured results: id, kind, canonical_url, facets, score, doc_url.
• /llms.txt — the manifest agents read first: what the platform is, the /v1/search path + facet vocabulary, auth, and links to per-capability markdown docs. Generated from the search projection (not hand-maintained) so new discoveries appear automatically. llms-full.txt for the expanded variant.
• MCP server (optional second front door) — exposes the same projection as callable tools (search_capabilities, get_capability). llms.txt = "read/HTTP"; MCP = "call me as a tool."

11. Cross-cutting concerns

• Idempotency: at-least-once delivery → consumers dedup on subject + hash (reuse _content_hash / canonical_url).
• Ordering: partition/key by subject so updates for one capability stay ordered.
• DLQ: poison messages and probe failures route to a dead-letter stream (probe errors are already counted today).
• Schema versioning: version the envelope (dataschema); a schema registry if Kafka/Redpanda is chosen.
• Observability: consumer lag + DLQ depth on the existing Grafana stack.

12. Phased rollout (strangler — no big bang)

1. Populate catalog_events inside the scraper's upsert transaction (closes the existing gap; zero behavior change downstream).
2. Persist probe HTTP status on each capability/text row (prerequisite for §9.1).
3. Add broker + relay; emit events while run_cycle keeps working unchanged.
4. Move embedding to an event consumer.
5. Stand up the search index projection + GET /v1/search + generated /llms.txt.
6. Add the tool-score projection seeded from probe status, then wire tool.* feedback.
7. Convert clustering/analytics to event-triggered (debounced).
8. Optional: MCP front door over the same projection.

13. Open questions

1. Lexical engine: Postgres ts_rank (no new infra) vs ParadeDB pg_search BM25 (better relevance, still in-Postgres)?
2. Broker: confirm NATS JetStream vs Redpanda/Kafka.
3. llms.txt: static file regenerated on events vs dynamically rendered per request?
4. Capabilities model: when do we promote the x402-specific endpoints table to a kind-discriminated capabilities projection for MCP/tools?
5. RRF weighting: quality as a separate ranker vs a multiplicative boost, and the k / weight values.

Appendix A — Probe status → health seed (quick reference)

200 or 402 ⇒ good · 401/403 ⇒ auth-gated / neutral · 404 ⇒ bad · 5xx ⇒ bad but transient (decayed/retried) · connection error ⇒ worst.