Ablyably.com
Ably's API program shows strength in core API design and schema quality, but two areas are holding back partner success: onboarding is friction-heavy (no self-service portal, no quickstart), and AI agents lack the structured context they need to understand and act on Ably's APIs without guesswork. Fixing the self-service onboarding gap and enriching agent-facing documentation should be the immediate priorities — together they determine whether partners and their agents can get to productive integrations independently, or whether every engagement still requires a sales conversation.
API DesignA clean, typed, well-governed API contract agents can reason about2 pass2 warn1 fail73B
| Signal | Points | Findings | Rationale | |
|---|---|---|---|---|
| warn | Schema coverage & depthvia sdk | 22.9/25 | 31/34 (91%) public params are fully typed (no any/**kwargs). Analyzer reports not all methods have a derivable input schema. Investigated: sdk 92%, spec 83%. | Typed, complete request/response schemas are what make agent function-calling possible. |
| warn | Machine-readable, versioned contractvia spec | 18.8/25 | info.version="4.0.0" set but no versioning scheme (URL / header / media type) detected. Investigated: spec 75%, docs 50%. | A current OpenAPI version with a declared versioning scheme lets agents reason about the contract. |
| pass | Security & governance hygienevia spec | 15/15 | No credential-shaped strings detected in spec. Investigated: spec 100%, wellknown 100%. | No leaked secrets, no critical lint violations, no OWASP API Top-10 spec smells, and a published vulnerability-disclosure channel. |
| pass | Example coveragevia spec | 10/10 | 74% example coverage (target 80%+). Investigated: spec 100%, sdk 100%. | Examples carry shape semantics schemas under-specify — for humans and agents alike. |
| fail | Auth declared & discoverablevia spec | 0/25 | No securitySchemes declared. Investigated: spec 0%, docs 0%. | Agents can only authenticate when auth is declared, scoped, and discoverable. |
Developer ExperienceThe context both developers and agents need to integrate fast — onboarding, code samples, complete descriptions and worked examples2 pass0 warn2 fail52D
| Signal | Points | Findings | Rationale | |
|---|---|---|---|---|
| pass | Description completenessvia spec | 15/15 | 100% of operations + parameters have substantive descriptions. Investigated: spec 100%, sdk 68%. | Complete descriptions are the context humans and agents need to use endpoints. |
| pass | Changelog publishedvia sdk | 13/13 | Newest official SDK activity 0 day(s) ago. Investigated: sdk 100%, spec 0%, docs 0%. | A published changelog lets partners track changes without surprise. |
| fail | Self-service developer portalvia docs | 0/29 | No signup page detected across conventional paths (/signup, /sign-up, /register, /get-started, /console/signup, /dashboard/signup, /try, /try-free, /free, /free-trial, /start, /start-free). Investigated: docs 0%. | Self-serve key/account creation is the fast first call for partners, with no sales gate. |
| fail | Quickstart presentvia docs | 0/25 | No quickstart/getting-started page found at the conventional paths. Investigated: docs 0%. | A quickstart is the fastest path from landing page to first successful call. |
| na | Code samples in docs | —/18 | No surface produced evidence for this capability in this run. | Multi-language samples shorten time-to-first-call. |
Agent DiscoveryPartners and their agents can find your APIs — llms.txt, registries, crawlable and reachable docs1 pass3 warn0 fail75B
| Signal | Points | Findings | Rationale | |
|---|---|---|---|---|
| pass | Registry & SDK presencevia docs | 28/28 | Indexed on Context7 (websites/ably, 45656 snippets). Investigated: docs 100%, cli 100%, sdk 92%. | Listing in MCP registries and publishing SDKs puts the API where agents and their tooling look. |
| warn | Docs reachable, not hard auth-gatedvia docs | 22.5/30 | Server ignores Accept: text/markdown header (0/50 sampled pages return markdown); 50 rate-limited (HTTP 429). Investigated: docs 75%. | Agents can only index and fetch docs they can reach — past auth gates and over correct HTTP semantics. |
| warn | llms.txt present, valid & comprehensivevia docs | 12.9/30 | Could not test any pages; 50 failed to fetch. Investigated: docs 43%, sdk 0%. | A valid, comprehensive llms.txt is the machine-readable entry point for agents. |
| warn | Crawlable / AEOvia wellknown | 6/12 | No sitemap discoverable via robots.txt or /sitemap.xml on the docs host. Investigated: wellknown 50%. | Bots allowed plus a fresh sitemap make docs findable by agent crawlers. |
Agent UnderstandingAgents can correctly interpret your APIs — machine-readable errors, consistent descriptions, structured data, parseable docs1 pass2 warn2 fail60C
| Signal | Points | Findings | Rationale | |
|---|---|---|---|---|
| pass | Machine-readable errors (RFC 9457)via docs | 28/28 | 5 distinct 4xx/5xx response codes documented. Investigated: docs 100%, sdk 100%, spec 50%. | RFC 9457 problem details and a documented error-code inventory let agents parse failures without burning tokens. |
| warn | Agent-navigable, token-efficient docsvia docs | 15.4/22 | Could not fetch HTML for any pages to compare; 25 failed to fetch. Investigated: docs 70%. | Server-rendered, clean, small-footprint docs are what an agent can cheaply fetch and parse correctly. |
| warn | Description consistency across surfaces | 0.5/7 | Mean pairwise description similarity across 2 surfaces (spec, sdk) is 2% (threshold 35% for full credit). | Every surface tells the same story about what the product is. |
| fail | Operation purpose clarityvia spec | 0/25 | 0% of operations are agent-inferable. Investigated: spec 0%. | Agents select the right endpoint from its summary + operationId; clear, named operations make tool-selection reliable — the strongest driver of correct tool choice. |
| fail | Agent instructions file (AGENTS.md)via wellknown | 0/10 | No AGENTS.md at the site root or /.well-known/. Investigated: wellknown 0%. | An AGENTS.md gives coding agents explicit setup, auth, and usage instructions to interpret and operate the API — beyond llms.txt's link index. |
| na | Docs structured data | —/8 | No surface produced evidence for this capability in this run. | Structured data (JSON-LD/schema.org) on docs pages gives agents an unambiguous parse target and is what answer engines cite. Detected on the JS-rendered head (Firecrawl) for a bounded page budget, so JS-injected JSON-LD is now caught; pages we can't render are excluded rather than failed. |
Agent UsabilityAgents have the context to use your APIs reliably, not just find them3 pass1 warn1 fail78B
| Signal | Points | Findings | Rationale | |
|---|---|---|---|---|
| pass | Idempotency documentedvia sdk | 27/27 | Built-in retry machinery is present (grep-derived). Investigated: sdk 100%, spec 0%, docs 0%. | Documented idempotency lets agents retry safely. |
| pass | Pagination documented & consistentvia docs | 22/22 | Pagination params detected on 1 list endpoint(s). Investigated: docs 100%, spec 75%, sdk 50%. | Consistent, documented pagination lets agents traverse collections. |
| pass | Sandbox separationvia sdk | 20/20 | Sandbox environments are exposed (sandbox). Investigated: sdk 100%, spec 0%, docs 0%. | An isolated environment lets agents exercise destructive operations safely. |
| warn | Runnable collection with test scriptsvia platform | 4.5/9 | No test scripts found in the workspace's collections. Investigated: platform 50%. | A public, maintained collection with assertions is runnable truth agents validate against. |
| fail | Rate-limit signalingvia spec | 0/22 | No rate-limit response headers documented. Investigated: spec 0%, docs 0%. | Machine-readable rate-limit headers let agents throttle adaptively. |
Resources Discovered
The public resources we found for Ably — the evidence behind the score. All discovered from public sources; nothing here requires access to your systems.
| Agent hints | Context7 (45,656) |
|---|---|
| APIs analyzed | 2 — Ably Chat REST API, Ably LiveObjects REST API |