Railwayrailway.app
Railway's API Discovery and Understanding capabilities are strong, showing that partners and agents can find and parse your APIs. However, the program's two critical weaknesses — a missing machine-readable contract with no declared auth or security model, and no runnable collections for partners to execute against — mean partners and AI agents cannot confidently build integrations today.
API DesignA clean, typed, well-governed API contract agents can reason about1 pass1 warn3 fail52D
| Signal | Points | Findings | Rationale | |
|---|---|---|---|---|
| warn | Schema coverage & depthvia sdk | 24.8/25 | 76/77 (99%) public params are fully typed (no any/**kwargs). Analyzer reports not all methods have a derivable input schema. Investigated: sdk 99%. | Typed, complete request/response schemas are what make agent function-calling possible. |
| pass | Example coveragevia docs | 10/10 | Code samples in 4 language(s) across docs (top: shell, typescript, javascript); 2/20 pages have ≥ 2 languages on the same page. Investigated: docs 100%, sdk 100%, cli 0%. | Examples carry shape semantics schemas under-specify — for humans and agents alike. |
| fail | Machine-readable, versioned contractvia cli | 0/25 | no machine-readable help (`--help --format json` unsupported). Investigated: cli 0%. | A current OpenAPI version with a declared versioning scheme lets agents reason about the contract. |
| fail | Auth declared & discoverablevia cli | 0/25 | no auth/config hint in --help. Investigated: cli 0%. | Agents can only authenticate when auth is declared, scoped, and discoverable. |
| fail | Security & governance hygienevia wellknown | 0/15 | No security.txt at /.well-known/security.txt or /security.txt. Investigated: wellknown 0%. | No leaked secrets, no critical lint violations, no OWASP API Top-10 spec smells, and a published vulnerability-disclosure channel. |
Developer ExperienceThe context both developers and agents need to integrate fast — onboarding, code samples, complete descriptions and worked examples1 pass2 warn1 fail55D
| Signal | Points | Findings | Rationale | |
|---|---|---|---|---|
| pass | Code samples in docsvia docs | 18/18 | Code samples in 4 language(s) across docs (top: shell, typescript, javascript); 2/20 pages have ≥ 2 languages on the same page. Investigated: docs 100%. | Multi-language samples shorten time-to-first-call. |
| warn | Quickstart presentvia docs | 12.5/25 | Quickstart page reachable (1 variants scanned starting at https://docs.railway.com/quick-start) but no runnable code sample detected in HTML or .md variant. Investigated: docs 50%. | A quickstart is the fastest path from landing page to first successful call. |
| warn | Description completenessvia sdk | 3.5/15 | 42/55 operations lack a description. Investigated: sdk 24%. | Complete descriptions are the context humans and agents need to use endpoints. |
| fail | Self-service developer portalvia docs | 0/29 | No signup page detected across conventional paths (/signup, /sign-up, /register, /get-started, /console/signup, /dashboard/signup, /try, /try-free, /free, /free-trial, /start, /start-free). Investigated: docs 0%. | Self-serve key/account creation is the fast first call for partners, with no sales gate. |
| na | Changelog published | —/13 | No surface produced evidence for this capability in this run. | A published changelog lets partners track changes without surprise. |
Agent DiscoveryPartners and their agents can find your APIs — llms.txt, registries, crawlable and reachable docs1 pass3 warn0 fail87A
| Signal | Points | Findings | Rationale | |
|---|---|---|---|---|
| pass | Registry & SDK presencevia docs | 28/28 | Indexed on Context7 (websites/railway, 4917 snippets). Investigated: docs 100%, sdk 100%, cli 100%, mcp 100%, wellknown 0%. | Listing in MCP registries and publishing SDKs puts the API where agents and their tooling look. |
| warn | Docs reachable, not hard auth-gatedvia docs | 27/30 | 1 of 50 sampled pages require authentication (49 accessible). Investigated: docs 90%. | Agents can only index and fetch docs they can reach — past auth gates and over correct HTTP semantics. |
| warn | llms.txt present, valid & comprehensivevia docs | 20.6/30 | No llms.txt directive found in HTML of any of 50 sampled pages. Investigated: docs 69%, sdk 0%. | A valid, comprehensive llms.txt is the machine-readable entry point for agents. |
| warn | Crawlable / AEOvia wellknown | 9/12 | Sitemap present (1 entries) but carries no lastmod values. Investigated: wellknown 75%. | Bots allowed plus a fresh sitemap make docs findable by agent crawlers. |
Agent UnderstandingAgents can correctly interpret your APIs — machine-readable errors, consistent descriptions, structured data, parseable docs3 pass2 warn0 fail88A
| Signal | Points | Findings | Rationale | |
|---|---|---|---|---|
| pass | Machine-readable errors (RFC 9457)via sdk | 28/28 | Single base error "RailwayError" with 32 subclasses. Investigated: sdk 100%, cli 0%. | RFC 9457 problem details and a documented error-code inventory let agents parse failures without burning tokens. |
| warn | Agent-navigable, token-efficient docsvia docs | 17.6/22 | 2 of 50 pages have substantive content differences between markdown and HTML (avg 4% missing). Investigated: docs 80%. | Server-rendered, clean, small-footprint docs are what an agent can cheaply fetch and parse correctly. |
| pass | Agent instructions file (AGENTS.md)via wellknown | 10/10 | Agent instructions file found at https://railway.app/AGENTS.md. Investigated: wellknown 100%. | An AGENTS.md gives coding agents explicit setup, auth, and usage instructions to interpret and operate the API — beyond llms.txt's link index. |
| pass | Docs structured datavia docs | 8/8 | JSON-LD Article markup on 13/13 assessed pages (100%) with dateModified present. Investigated: docs 100%. | Structured data (JSON-LD/schema.org) on docs pages gives agents an unambiguous parse target and is what answer engines cite. Detected on the JS-rendered head (Firecrawl) for a bounded page budget, so JS-injected JSON-LD is now caught; pages we can't render are excluded rather than failed. |
| warn | Description consistency across surfaces | 1.1/7 | Mean pairwise description similarity across 2 surfaces (docs, sdk) is 6% (threshold 35% for full credit). | Every surface tells the same story about what the product is. |
| na | Operation purpose clarity | —/25 | No surface produced evidence for this capability in this run. | Agents select the right endpoint from its summary + operationId; clear, named operations make tool-selection reliable — the strongest driver of correct tool choice. |
Agent UsabilityAgents have the context to use your APIs reliably, not just find them2 pass0 warn1 fail86A
| Signal | Points | Findings | Rationale | |
|---|---|---|---|---|
| pass | Idempotency documentedvia sdk | 27/27 | Built-in retry machinery is present (grep-derived). Investigated: sdk 100%, cli 0%. | Documented idempotency lets agents retry safely. |
| pass | Sandbox separationvia sdk | 20/20 | Sandbox environments are exposed (sandbox). Investigated: sdk 100%. | An isolated environment lets agents exercise destructive operations safely. |
| fail | Runnable collection with test scriptsvia platform | 0/9 | No public Postman workspace discovered for the org. Investigated: platform 0%. | A public, maintained collection with assertions is runnable truth agents validate against. |
| na | Rate-limit signaling | —/22 | No surface produced evidence for this capability in this run. | Machine-readable rate-limit headers let agents throttle adaptively. |
| na | Pagination documented & consistent | —/22 | No surface produced evidence for this capability in this run. | Consistent, documented pagination lets agents traverse collections. |
Resources Discovered
The public resources we found for Railway — the evidence behind the score. All discovered from public sources; nothing here requires access to your systems.
| Agent hints | Context7 (4,917) |
|---|