Skip to content

Observability Features

Overview

tRPC-Agent-Go provides comprehensive observability features built on the OpenTelemetry standard, offering powerful observability capabilities for Agent applications. With observability enabled, developers can achieve end-to-end monitoring of Agent runtime status, including tracing, performance metrics collection, and logging.

🎯 Key Features

  • Tracing: Fully records call chains during Agent execution.
  • Metrics: Collects key runtime performance data for Agents.
  • Logging: Unified log collection and management.
  • Multi-platform Support: Supports mainstream monitoring platforms such as Jaeger, Prometheus, Galileo, and ZhiYan Monitoring Bao.
  • Flexible Configuration: Supports multiple configuration methods and custom extensions.

Integration with Different Monitoring Platforms

Langfuse Integration

Langfuse is an observability platform designed for LLM applications and supports collecting tracing data via the OpenTelemetry protocol. tRPC-Agent-Go can export Trace data to Langfuse via OpenTelemetry.

1. Deploy Langfuse

Refer to the Langfuse self-hosting guide for local or cloud deployment. For a quick start, see the Docker Compose deployment guide.

2. Go Code Integration Example

1
2
3
4
export LANGFUSE_PUBLIC_KEY="your-public-key"
export LANGFUSE_SECRET_KEY="your-secret-key"
export LANGFUSE_HOST="your-langfuse-host" # In host:port format (no scheme), e.g. "cloud.langfuse.com:443" or "localhost:3000".
export LANGFUSE_INSECURE="true" # Use "true" for local http (development only).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
import (
    "context"
    "log"

    "trpc.group/trpc-go/trpc-agent-go/telemetry/langfuse"
)

func main() {
    // Start trace with Langfuse integration using environment variables
    clean, err := langfuse.Start(context.Background())
    if err != nil {
        log.Fatalf("Failed to start trace telemetry: %v", err)
    }
    defer func() {
        if err := clean(context.Background()); err != nil {
            log.Printf("Failed to clean up trace telemetry: %v", err)
        }
    }()

See the complete example at examples/telemetry/langfuse.

Note: LANGFUSE_HOST is passed to OpenTelemetry otlptracehttp.WithEndpoint, so it must not include http:// or https://. The scheme is controlled by LANGFUSE_INSECURE, and the path is fixed to /api/public/otel/v1/traces.

Run the example:

1
go run .

You can view tracing data in the Langfuse console.

For message payload compatibility and the recommended OTel role + parts fields, see Multimodal Telemetry Messages.

Integration Code Description

Langfuse supports receiving Trace data via the /api/public/otel (OTLP) endpoint, supporting HTTP/protobuf only, not gRPC. The above code integrates with Langfuse by setting OTEL_EXPORTER_OTLP_HEADERS and OTEL_EXPORTER_OTLP_TRACES_ENDPOINT.

1
2
3
4
5
6
7
8
9
# EU data region
OTEL_EXPORTER_OTLP_ENDPOINT="https://cloud.langfuse.com/api/public/otel"
# US data region
# OTEL_EXPORTER_OTLP_ENDPOINT="https://us.cloud.langfuse.com/api/public/otel"
# Local deployment (>= v3.22.0)
# OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:3000/api/public/otel"

# Set Basic Auth authentication
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic ${AUTH_STRING}"

AUTH_STRING is the base64 encoding of public_key:secret_key, which can be generated using the following command:

1
2
echo -n "pk-lf-xxxx:sk-lf-xxxx" | base64
# On GNU systems, add -w 0 to avoid line breaks

To specify the endpoint for traces only, set:

1
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://localhost:3000/api/public/otel/v1/traces"

Jaeger, Prometheus, and Other Open-Source Monitoring Platforms

Refer to code examples in examples/telemetry.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
package main

import (
    "context"
    "log"

    ametric "trpc.group/trpc-go/trpc-agent-go/telemetry/metric"
    atrace "trpc.group/trpc-go/trpc-agent-go/telemetry/trace"
)

func main() {
    // Start metrics collection.
    mp, err := ametric.NewMeterProvider(
        context.Background(),
        ametric.WithEndpoint("localhost:4318"),
        ametric.WithProtocol("http"),
    )
    if err != nil {
        log.Fatalf("Failed to create meter provider: %v", err)
    }
    defer mp.Shutdown(context.Background())
    ametric.InitMeterProvider(mp)

    // Start tracing.
    traceClean, err := atrace.Start(
        context.Background(),
        atrace.WithEndpoint("localhost:4317"), // Trace export address.
    )
    if err != nil {
        log.Fatalf("Failed to start trace telemetry: %v", err)
    }
    defer traceClean()

    // Your Agent application code.
    // ...
    // You can add custom traces and metrics.
}

Jaeger trace example

trace-jaeger

Prometheus metrics example

metric-prometheus

Practical Application Examples

Basic Metrics and Tracing

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
package main

import (
    "context"
    "fmt"
    "time"

    ametric "trpc.group/trpc-go/trpc-agent-go/telemetry/metric"
    atrace "trpc.group/trpc-go/trpc-agent-go/telemetry/trace"
    "trpc.group/trpc-go/trpc-agent-go/log"

    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/metric"
    "go.opentelemetry.io/otel/trace"
)

func main() {
    mp, err := ametric.NewMeterProvider(
        context.Background(),
        ametric.WithEndpoint("localhost:4318"),
        ametric.WithProtocol("http"),
    )
    if err != nil {
        log.Fatalf("Failed to create meter provider: %v", err)
    }
    defer mp.Shutdown(context.Background())
    ametric.InitMeterProvider(mp)
    meter := mp.Meter("trpc_agent_go.app")

    if err := processAgentRequest(context.Background(), meter); err != nil {
        log.Errorf("processAgentRequest failed: %v", err)
    }
}

func processAgentRequest(ctx context.Context, meter metric.Meter) error {
    // Create tracing span.
    ctx, span := atrace.Tracer.Start(
        ctx,
        "process-agent-request",
        trace.WithAttributes(
            attribute.String("agent.type", "chat"),
            attribute.String("user.id", "user123"),
        ),
    )
    defer span.End()

    // Create metrics counter.
    requestCounter, err := meter.Int64Counter(
        "agent.requests.total",
        metric.WithDescription("Total number of agent requests"),
    )
    if err != nil {
        return err
    }

    // Record request.
    requestCounter.Add(ctx, 1, metric.WithAttributes(
        attribute.String("agent.type", "chat"),
        attribute.String("status", "success"),
    ))

    // Simulate processing.
    time.Sleep(100 * time.Millisecond)

    return nil
}

Agent Execution Tracing

The framework automatically instruments key components of Agents:

1
2
3
4
5
6
// Agent execution will automatically generate the following observability data:
// 
// Traces:
// - agent.execution: Overall Agent execution process.
// - tool.invocation: Tool invocation process.  
// - model.api_call: Model API call process.

Telemetry Data Analysis

Trace Analysis

A typical Agent execution trace structure:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
Agent Request
β”œβ”€β”€ Planning Phase
β”‚   β”œβ”€β”€ Model API Call (DeepSeek)
β”‚   └── Response Processing
β”œβ”€β”€ Tool Execution Phase  
β”‚   β”œβ”€β”€ Tool: web_search
β”‚   β”œβ”€β”€ Tool: knowledge_base
β”‚   └── Result Processing
└── Response Generation Phase
    β”œβ”€β”€ Model API Call (DeepSeek)
    └── Final Response Formatting

Trace data can be used to analyze:

  • Performance Bottlenecks: Identify the most time-consuming operations.
  • Error Localization: Quickly locate the exact failing step.
  • Dependencies: Understand relationships between components.
  • Concurrency Analysis: Observe the effects of concurrent execution.

Span Attribute Policy (Production Side)

telemetry/trace provides an opt-in SpanAttributePolicy that controls collection and size of large payload span attributes at span creation time.

Default (unset) behavior is unchanged.

Two capabilities

  • Drop() and unconditional Omit() (without MaxBytes) skip json.Marshal.
  • MaxBytes() + Omit(): raw []byte paths (for example execute_tool tool arguments) compare length without an extra marshal; JSON-backed paths (chat / workflow / invoke_agent) still require a full marshal and do not reduce heap peaks.
  • Truncate() always marshals the full payload; it only limits the value written to the span.
Rule Reduces marshal heap peak? Notes
Drop() Yes Skips marshaling; attribute not written
Omit() (no MaxBytes) Yes Skips payload marshaling
MaxBytes(n) + Omit() JSON: no; []byte: yes JSON paths marshal fully before comparing size; []byte paths use len only
Truncate(n) No Full marshal, then truncate on export

To reduce memory, prefer Drop() on attributes you do not need (for example redundant *.otel).

Coverage

Large payload attributes are wired for chat, invoke_agent, workflow, and execute_tool (messages, llm request/response, workflow request/response, tool arguments/result, and related keys).

Configuration

1
2
3
4
5
6
7
8
9
import atrace "trpc.group/trpc-go/trpc-agent-go/telemetry/trace"

clean, err := atrace.Start(ctx,
    atrace.WithSpanAttributePolicy(
        atrace.WithAttributeRule(atrace.OperationChat, atrace.AttrInputMessagesOTel, atrace.Drop()),
        atrace.WithAttributeRule(atrace.OperationChat, atrace.AttrOutputMessagesOTel, atrace.Drop()),
        atrace.WithAttributeRule(atrace.OperationInvokeAgent, atrace.AttrInputMessagesOTel, atrace.Drop()),
    ),
)

If trace.Start is not used, call atrace.SetSpanAttributePolicy(...) before the first LLM invocation. trace.Start installs the policy only after tracer initialization succeeds; clean() restores the previous policy.

MaxBytes + Omit() example (limits attribute size; JSON paths still marshal):

1
2
3
atrace.WithAttributeRule(atrace.OperationWorkflow, atrace.AttributeKey("gen_ai.workflow.request"),
    atrace.MaxBytes(16<<10), atrace.Omit(),
)

Truncate example (full marshal, limited export size):

1
atrace.WithAttributeRule(atrace.OperationWorkflow, atrace.AttributeKey("gen_ai.workflow.response"), atrace.Truncate(64<<10))

Compatibility

Drop/Omit/Truncate may prevent some backends from reconstructing structured full text from attributes. Opt in based on your backend and memory budget.

Advanced Features

Custom Exporter

If you need to send observability data to a custom monitoring system:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
import (
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "go.opentelemetry.io/otel/sdk/trace"
)

func setupCustomExporter() error {
    exporter, err := otlptracehttp.New(
        context.Background(),
        otlptracehttp.WithEndpoint("https://your-custom-endpoint.com"),
        otlptracehttp.WithHeaders(map[string]string{
            "Authorization": "Bearer your-token",
        }),
    )
    if err != nil {
        return err
    }

    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
    )

    // Set as the global TracerProvider.
    otel.SetTracerProvider(tp)

    return nil
}

References

  • OpenTelemetry documentation.
  • tRPC-Agent-Go telemetry examples.

By using observability features properly, you can establish a complete monitoring system for Agent applications, discover and resolve issues in time, and continuously optimize system performance.