Observability

What it is: TramAI provides OpenTelemetry instrumentation through OpenTelemetryOperationObserver. It records one span per provider attempt, emits metrics for latency, tokens, and parse failures, and integrates with your existing OpenTelemetry pipeline.

When to use it: Production deployments where you need visibility into AI call latency, token consumption, error rates, retry behavior, and provider attribution. The module is optional — without it, TramAI runs without any OpenTelemetry dependencies.

Minimum Setup

Add the dependency:

dependencies {
   implementation("dev.tramai:tramai-observability:0.3.1")
}

Create the observer with your OpenTelemetry instance and attach it to the builder:

val observer = OpenTelemetryOperationObserver(openTelemetry)

val tramai = Tramai {
   provider(OpenAiProvider(System.getenv("OPENAI_API_KEY")), name = "openai", default = true)
   model("gpt-4o", "openai")
   observer(observer)
}

That's it. Every provider attempt now produces spans and metrics.

What Is Recorded

Spans (Tracing)

One span per provider attempt with the name ai.<methodName>. Attributes recorded on every span:

Attribute	Example	Description
`gen_ai.system`	`openai`	Provider identifier
`gen_ai.request.model`	`gpt-4o`	Model requested by the operation
`gen_ai.response.model`	`gpt-4o-2024-08-06`	Model actually used (when reported)
`gen_ai.usage.input_tokens`	142	Input tokens consumed
`gen_ai.usage.output_tokens`	37	Output tokens generated
`tramai.operation.interface`	`com.example.InvoiceAnalyzer`	Service interface name
`tramai.operation.method`	`analyze`	Method name
`tramai.retry.attempt`	0	Retry attempt index (0-based)
`tramai.structured.parse_success`	`true`	Whether structured parsing succeeded
`tramai.outcome`	`success`	Overall outcome (success/failure/parse_failure)
`tramai.error.type`	`ProviderTransportFailure`	Error type on failure

Span events (additional detail):

tramai.parse.failure: Emitted when structured parsing fails (includes raw response length and validation error summary)
Engine events (via onEngineEvent): Retry scheduling, fallback routing, circuit-open transitions, token budget warnings

Metrics

Recorded when a meter provider is configured in your OpenTelemetry SDK:

Instrument Name	Type	Description
`tramai.operation.attempts`	Counter	Completed provider attempts
`tramai.operation.duration`	Histogram	Duration of provider attempts (ms)
`tramai.operation.input_tokens`	Counter	Total input tokens across all attempts
`tramai.operation.output_tokens`	Counter	Total output tokens across all attempts
`tramai.operation.input_tokens.per_attempt`	Histogram	Distribution of input tokens per attempt
`tramai.operation.output_tokens.per_attempt`	Histogram	Distribution of output tokens per attempt
`tramai.operation.parse_failures`	Counter	Structured parse failures
`tramai.engine.events`	Counter	Engine-owned resilience and routing events

The tramai.engine.events counter carries a tramai.event.name attribute identifying the specific event type (e.g., retry_scheduled, fallback_activated, circuit_open, token_budget_warning).

Token Usage Tracking

Token counts arrive on span attributes and counter/histogram metrics when the provider reports them. Supported providers:

OpenAI: Input and output tokens reported per response
Anthropic: Input and output tokens, plus thinking tokens
Gemini: Input and output tokens
Ollama: Varies by model
Others: When the provider reports usage, it's captured

// Token data appears automatically in:
// - Span attributes: gen_ai.usage.input_tokens, gen_ai.usage.output_tokens
// - Counter metrics: tramai.operation.input_tokens, tramai.operation.output_tokens
// - Histogram metrics: tramai.operation.input_tokens.per_attempt, tramai.operation.output_tokens.per_attempt
// - StreamChunk.Complete: usage field for streaming operations

Error Rate Monitoring

Errors are captured through three mechanisms:

Span status: Set to StatusCode.ERROR on provider failures with exception recording
Metrics: tramai.operation.parse_failures counter for structured output failures
Span attributes: tramai.outcome and tramai.error.type enable filtering and aggregation

Common error scenarios you can monitor:

Error Pattern	How to Detect
Provider unavailable	`tramai.outcome = "failure"`, `tramai.error.type = "ProviderTransportFailure"`
Rate limiting	`tramai.engine.events` with `tramai.event.name = "retry_scheduled"`
Circuit breaker open	`tramai.engine.events` with `tramai.event.name = "circuit_open"`
Parse failures	`tramai.operation.parse_failures` counter
Token budget exceeded	`tramai.engine.events` with `tramai.event.name = "token_budget_warning"`

Kotlin + Java Examples

Kotlin

import io.opentelemetry.api.OpenTelemetry
import io.opentelemetry.sdk.OpenTelemetrySdk

val openTelemetry: OpenTelemetry = OpenTelemetrySdk.builder().build()
// Configure your exporter pipeline (OTLP, Jaeger, Prometheus, etc.)

val observer = OpenTelemetryOperationObserver(openTelemetry)

val tramai = Tramai {
   provider(OpenAiProvider(System.getenv("OPENAI_API_KEY")), name = "openai", default = true)
   model("gpt-4o", "openai")
   observer(observer)
}

Java

import io.opentelemetry.api.OpenTelemetry;
import dev.tramai.observability.OpenTelemetryOperationObserver;

OpenTelemetry openTelemetry = OpenTelemetrySdk.builder().build();
// Configure your exporter pipeline

OpenTelemetryOperationObserver observer =
   new OpenTelemetryOperationObserver(openTelemetry);

Tramai tramai = Tramai.builder()
   .provider(new OpenAiProvider(System.getenv("OPENAI_API_KEY")), "openai", true)
   .model("gpt-4o", "openai")
   .observer(observer)
   .build();

Why The Observer Sits Outside Providers

TramAI keeps providers small. Providers focus on:

Request creation (provider-specific format)
HTTP transport
Response mapping

The engine wraps those calls with observation hooks so providers do not need tracing-specific logic. This means:

Adding a new provider automatically gets observability without provider changes
Observability is optional — remove the observer, and no tracing code runs
The observer API is stable across provider implementations

Integration with Spring Boot

In Spring Boot, configure OpenTelemetryOperationObserver as a bean and it will be picked up by auto-configuration:

@Configuration
class ObservabilityConfig {
   @Bean
   fun tramaiObserver(openTelemetry: OpenTelemetry): OpenTelemetryOperationObserver =
       OpenTelemetryOperationObserver(openTelemetry)
}

Spring Boot's OpenTelemetry auto-configuration (via io.opentelemetry:opentelemetry-spring-boot-starter) will provide the OpenTelemetry instance. TramAI's auto-configuration picks up any OperationObserver bean.

Verifying Traces in Tests

To assert that TramAI emitted the right spans and metrics during a test, use InMemorySpanExporter and InMemoryMetricReader from the OpenTelemetry SDK testing artifacts:

dependencies {
   testImplementation("io.opentelemetry:opentelemetry-sdk-testing:1.48.0")
   testImplementation("dev.tramai:tramai-testing:0.3.1")
}

Setup

import io.opentelemetry.sdk.testing.exporter.InMemorySpanExporter
import io.opentelemetry.sdk.testing.exporter.InMemoryMetricReader
import io.opentelemetry.sdk.trace.SdkTracerProvider
import io.opentelemetry.sdk.trace.export.SimpleSpanProcessor
import io.opentelemetry.sdk.metrics.SdkMeterProvider
import io.opentelemetry.sdk.OpenTelemetrySdk

class MyObserverTest {
   private val spanExporter = InMemorySpanExporter.create()
   private val metricReader = InMemoryMetricReader.create()

   private val tracerProvider = SdkTracerProvider.builder()
       .addSpanProcessor(SimpleSpanProcessor.create(spanExporter))
       .build()

   private val meterProvider = SdkMeterProvider.builder()
       .registerMetricReader(metricReader)
       .build()

   private val openTelemetry = OpenTelemetrySdk.builder()
       .setTracerProvider(tracerProvider)
       .setMeterProvider(meterProvider)
       .build()

   @AfterEach
   fun tearDown() {
       spanExporter.reset()
   }
}

Asserting Spans

@Test
fun `records span attributes for successful provider call`() = runBlocking {
   val provider = MockAiProvider {
       onMethod("classify") respondWith """{"priority":"high"}"""
   }

   val tramai = Tramai {
       provider(provider, default = true)
       model("gpt-4o", "mock")
       observer(OpenTelemetryOperationObserver(openTelemetry))
   }

   val service = tramai.create<Classifier>()
   val result = service.classify("Urgent issue")

   // Assert exported span
   val span = spanExporter.finishedSpanItems.single()
   assertThat(span.name).isEqualTo("ai.classify")
   assertThat(span.attributes.asMap())
       .containsEntry(AttributeKey.stringKey("gen_ai.system"), "mock")
       .containsEntry(AttributeKey.stringKey("gen_ai.request.model"), "gpt-4o")
       .containsEntry(AttributeKey.stringKey("tramai.operation.method"), "classify")
}

Asserting Metrics

@Test
fun `records attempt and token metrics`() = runBlocking {
   val provider = MockAiProvider {
       onMethod("classify") respondWith """{"priority":"high"}"""
   }

   val tramai = Tramai {
       provider(provider, default = true)
       model("gpt-4o", "mock")
       observer(OpenTelemetryOperationObserver(openTelemetry))
   }

   val service = tramai.create<Classifier>()
   service.classify("Urgent issue")

   val metrics = metricReader.collectAllMetrics()

   // Assert counter values
   val attemptsMetric = metrics.single { it.name == "tramai.operation.attempts" }
   assertThat(attemptsMetric.longSumData.points.single().value).isEqualTo(1L)

   // Assert histogram exists
   val durationMetric = metrics.single { it.name == "tramai.operation.duration" }
   assertThat(durationMetric.histogramData.points.single().count).isEqualTo(1L)
}

Asserting Parse Failures and Retries

@Test
fun `records parse failure span event on retry`() = runBlocking {
   val provider = MockAiProvider {
       onMethod("classify") respondWith "not json"
       onMethod("classify") respondWith """{"priority":"high"}"""
   }

   val tramai = Tramai {
       provider(provider, default = true)
       model("gpt-4o", "mock")
       observer(OpenTelemetryOperationObserver(openTelemetry))
   }

   val service = tramai.create<Classifier>()
   val result = service.classify("Urgent issue")

   // Two spans — first attempt (parse failure) + retry (success)
   assertThat(spanExporter.finishedSpanItems).hasSize(2)

   val firstAttempt = spanExporter.finishedSpanItems.first()
   assertThat(firstAttempt.events).anySatisfy { event ->
       assertThat(event.name).isEqualTo("tramai.parse.failure")
   }

   // Parse failure counter recorded
   val parseFailures = metricReader.collectAllMetrics()
       .single { it.name == "tramai.operation.parse_failures" }
   assertThat(parseFailures.longSumData.points.single().value).isEqualTo(1L)
}

Asserting Engine Events

@Test
fun `records engine events as span events`() = runBlocking {
   val provider = SimulatedFailureProvider {
       onMethod("classify").retryableFailure("rate limited", statusCode = 429)
       onMethod("classify") respondWith """{"priority":"high"}"""
   }

   val tramai = Tramai {
       provider(provider, default = true)
       model("gpt-4o", "mock")
       observer(OpenTelemetryOperationObserver(openTelemetry))
   }

   val service = tramai.create<Classifier>()
   val result = service.classify("Urgent issue")

   // Engine events like retry_scheduled appear as span events
   val span = spanExporter.finishedSpanItems.first()
   assertThat(span.events).anySatisfy { event ->
       assertThat(event.name).isEqualTo("tramai.retry.scheduled")
   }
}

This pattern lets you write deterministic, in-process assertions against the exact OpenTelemetry data your application will emit in production, without needing an exporter or collector running.

Limitations

No automatic exporter setup: The observer does not configure exporters for you. You must set up your own OpenTelemetry SDK, meter provider, and exporter pipeline (OTLP, Jaeger, Prometheus, etc.).
No dashboards: The module provides instrumentation, not pre-built dashboards. You configure those in your observability platform.
No trace correlation helpers: Outside standard OpenTelemetry usage, no additional trace ID propagation helpers are provided.
Streaming-specific metrics: Streaming operations produce completion metrics but do not emit per-chunk metrics.
Provider-specific attributes: Provider-specific details (e.g., Anthropic thinking tokens, OpenAI system fingerprint) are captured when available but not all attributes are available from all providers.

Next Steps

Testing — Compare observed behavior with expected metrics
Structured Output — Monitor parse failure rates
Providers & Routing — Attribute cost and latency per provider
Spring Boot — Auto-configure observability in Spring Boot