Skip to content
Devsible
Sign inGet started

DevOps for AI inference · AWS Bedrock & vLLM

Find, explain and fix expensive or slow AI inference.

Devsible reads the telemetry you already have, flags spend spikes, latency regressions, error bursts and saturated deployments, and shows the evidence behind every claim.

Create a workspace Explore the sample workspace

Read-only by design. Devsible never changes your infrastructure.

findings / latency-regression
Illustrative · synthetic
HighP95 latency regression · vLLM

P95 latency up 108% on llama-3.1-8b

7-day baseline 2.1 sLast 6h 4.4 s
Measured
  • Waiting requests18.6 vs 0.3
  • KV-cache usage96%
  • Requests / hour+45%
Hypothesis

Deployment is saturated: requests queue before scheduling under KV-cache pressure.

Spend is a lagging signal

By the time the invoice arrives, a prompt change has been doubling output tokens for weeks.

Generic dashboards don't diagnose

CPU and HTTP metrics can't tell queueing from longer generations or quota throttling.

Percentiles get averaged wrong

Averaging P95s across intervals produces numbers that never happened. We merge histograms instead.

From raw metrics to a finding you can act on

  1. 01

    Connect read-only

    A cross-account IAM role with an external ID for Bedrock, or a TLS metrics endpoint for vLLM. No access keys, no agents in your request path.

  2. 02

    Normalise telemetry

    Provider metrics become canonical series with units, sources and freshness. Unknown stays unknown — never silently zero.

  3. 03

    Detect deterministically

    Transparent rules compare each window to a comparable baseline and refuse to fire without minimum evidence.

  4. 04

    Explain and verify

    Each finding separates measured evidence from hypotheses, with non-destructive next steps and a validation checklist.

Integrations, with honest coverage

Every integration shows a capability matrix: what the source exports, what it doesn't, and what needs extra instrumentation.

AWS Bedrock

CloudWatch via cross-account role

  • Invocations, errors, throttles
  • Input and output tokens per model
  • Provider-computed P50 / P95 latency
  • Time-to-first-token requires client instrumentation.

vLLM

Prometheus endpoint, SSRF-guarded

  • Token throughput and request counts
  • Latency and TTFT histograms
  • Queue depth and KV-cache usage
  • vLLM does not export error counts.

GPU telemetry

Optional DCGM exporter

  • Measured GPU utilisation
  • Under-utilisation vs provisioned GPUs
  • GPU idle time is never inferred from throughput.

Read-only by design

Devsible observes and advises. Every recommendation is a step for a human to take, with risks and a way to verify the outcome.

  • No write permissions

    The Bedrock policy grants only cloudwatch:GetMetricData and ListMetrics.

  • No long-lived keys

    Short-lived STS sessions bound to a server-generated external ID.

  • Secrets encrypted at rest

    Endpoint tokens are AES-256-GCM encrypted and never returned by the API.

  • Tenant isolation server-side

    Every query is scoped to a verified membership, not to IDs from the browser.

What we won't claim

No guaranteed savings, no "AI root cause" certainty, no prompt capture. Spend is shown as an estimate from pricing you configure, with its coverage. If a metric isn't available, the product says so instead of showing a number.

See it with synthetic data first

The sample workspace runs the full detection pipeline on clearly labelled demo telemetry.

Explore the sample workspace