Rust's std::io::Stdout uses internal BufWriter that bypasses libc,
making stdbuf -oL useless. stderr is unbuffered by default in Rust,
so tracing output appears immediately in container logs.
Also reverts n_episodes debug cap back to 16384.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The batch span processor needs a tokio runtime for gRPC transport and
periodic flush. Async services already have one via #[tokio::main], but
sync training binaries (hyperopt, train, evaluate) don't.
Previous approach (making binaries async with #[tokio::main]) caused
"Cannot start a runtime from within a runtime" panics because the ML
crate's internal code creates its own tokio runtimes for block_on().
New approach: build_otel_tracer() detects runtime context via
Handle::try_current(). If absent, it creates a dedicated 1-worker
multi-thread runtime stored in a process-lifetime OnceLock. The worker
thread actively polls the OTLP batch export task.
Reverts training binaries to sync fn main() so internal runtime creation
(hyperopt adapters, DQN/PPO trainers) continues working as before.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove unnecessary async from init_observability -- body was fully sync.
Change otlp_endpoint from &str to Option<&str> -- when None, OTLP layer
is skipped (fmt-only mode). Update all 8 service callers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
OTLP fix: opentelemetry-otlp 0.27 bundled tonic 0.12 while the workspace
uses tonic 0.14, making with_channel() impossible (type mismatch). Upgrading
to otel-otlp 0.31 aligns both on tonic 0.14, enabling explicit Channel
construction that respects http:// scheme (no spurious TLS negotiation).
API migrations (otel 0.27→0.31):
- TracerProvider → SdkTracerProvider
- with_batch_exporter(exporter, runtime) → with_batch_exporter(exporter)
- Resource::new(vec![...]) → Resource::builder().with_service_name().build()
- global::shutdown_tracer_provider() removed (Drop-based shutdown)
- opentelemetry-otlp feature "tonic" → "grpc-tonic"
DCGM fix: remove runtimeClassName: nvidia from DaemonSet — Scaleway Kapsule
GPU pools use nvidia runtime as default containerd handler. The RuntimeClass
CRD is only created by the full GPU Operator, not the device plugin alone.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously init_observability() called init_json_logger() (which set the
global subscriber with a fmt layer) and then init_tracing() (which only
set the global tracer provider). Because no tracing_opentelemetry::layer()
was ever added to the subscriber, #[instrument] spans never reached Tempo.
Now init_observability() builds a single Registry subscriber that
composes an EnvFilter, a JSON fmt layer, and an optional OTLP layer
(via tracing_opentelemetry). The OTLP layer gracefully degrades to None
when Tempo is unavailable, so logging always works.
Also adds build_otel_tracer() to tracing_config.rs which returns
Option<Tracer> for the caller to wrap in an OpenTelemetryLayer inside
their subscriber stack, letting the S type parameter be inferred
correctly by the compiler.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Disable file logging in init_observability — use console-only (stdout).
Containerd captures stdout, Promtail ships to Loki. Eliminates
"Failed to create log directory: Permission denied" errors on 3 services.
- Remove LOG_DIR env vars and /app/logs emptyDir volumes from 5 deployments.
- Add OTEL_EXPORTER_OTLP_ENDPOINT=tempo to 5 services that were missing it
(api-gateway, web-gateway, data-acquisition, trading-agent, broker-gateway).
- Make OTLP tracing init non-fatal — services degrade gracefully if Tempo is
unavailable instead of failing entire observability stack.
- Stop web-gateway gRPC stream retry on Unimplemented status. When trading-service
doesn't implement streaming endpoints, log once and stop instead of retrying
every 60s indefinitely.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- init_observability now reads LOG_DIR env var for log directory (falls back
to relative logs/ path). Gracefully disables file logging if directory
cannot be created instead of failing the entire observability init.
- Add ServiceAccount, Role, and RoleBinding for ml-training-service to
create/get/list/watch/delete batch/v1 Jobs (K8s training dispatch).
- Add LOG_DIR=/app/logs env to ml-training-service, trading-service, and
backtesting-service deployment YAMLs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move 17 library crates into crates/, CLI binary into bin/fxt,
consolidate 10 test crates into testing/, split config crate
from deployment config files.
Root directory reduced from 38+ to ~17 directories.
All Cargo.toml paths and build.rs proto refs updated.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
