gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,866 Commits 13 Branches 0 Tags
7b60fd5f8636ce144abbbc04631328993c942e60
Commit Graph

17 Commits

Author SHA1 Message Date
jgrusewski
68b6aa8313 feat(infra): migrate container registry from SCW to internal GitLab 
Full migration off Scaleway Container Registry to internal GitLab
registry backed by MinIO S3. All 4 images (ci-builder, ci-builder-cpu,
foxhunt-runtime, foxhunt-training-runtime) rebuilt in internal registry.

Registry & images:
- All image refs → gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/
- imagePullSecrets: scw-registry → gitlab-registry
- Kaniko build template: two-step DAG (git-clone → kaniko-build) with shared PVC
- Kaniko layer cache enabled at root/foxhunt/cache
- AWS_ACCESS_KEY_ID: $SCW_ACCESS_KEY → $MINIO_ACCESS_KEY in .gitlab-ci.yml

Network policies:
- ci-pipeline: add HTTP/80, registry/5000, webservice/8181 egress rules

DNS & Tailscale proxy cleanup:
- Remove ci, prometheus, monitor DNS records (no longer exposed)
- Rename s3 → minio DNS record
- Remove Argo UI, Prometheus, monitor nginx server blocks
- Remove argo-htpasswd volume mount
- Tailscale proxy nodeSelector: infra → platform

Terraform cleanup:
- Delete infra/modules/registry/ (SCW CR namespace)
- Delete infra/modules/object-storage/ (SCW S3 buckets)
- Delete infra/modules/secrets/ (SCW secrets)
- Delete corresponding live configs
- TF state backend: S3 → GitLab HTTP

Argo workflows:
- Add events/ (GitLab push eventsource + ci-pipeline sensor)
- ci-pipeline + training templates: SCW → internal registry
- Delete obsolete compile-training-template.yaml

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-05 13:52:24 +01:00
jgrusewski
08d070bb95 refactor: rename JWT issuer foxhunt-api-gateway → foxhunt-api, drop compat aliases 
- JWT issuer now foxhunt-api across all 16 files (services, tests, config, docker-compose)
- Remove serde alias api_gateway_url from FxtConfig (no backwards compat)
- Remove api_gateway CLI alias from e2e orchestrator
- All services must deploy simultaneously for JWT validation to match

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-05 00:22:04 +01:00
jgrusewski
075f715fa1 refactor: replace all api_gateway/web-gateway references across codebase 
- Rename test functions, variables, bench names (api_gateway → api)
- Update e2e orchestrator executable path (target/debug/api)
- Clean Grafana dashboards: remove dead web-gateway panels, fix trailing
  pipes/commas, update pod selectors
- Update metrics_validation test metric prefixes (api_gateway_ → api_)
- Fix .gitlab-ci.yml remaining path reference
- Fix FXT CLI test flag and function names
- Keep JWT issuer foxhunt-api-gateway (cross-service auth compat)
- Keep serde alias api_gateway_url (config backwards compat)

cargo check --workspace passes cleanly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-05 00:18:58 +01:00
jgrusewski
d25c82f8f3 refactor: rename api_gateway → api across workspace, tests, and load crate 
- Workspace Cargo.toml: remove web-gateway + api_gateway members, keep api
- trading_service: dep api-gateway → api, update test imports
- testing/api-gateway-load → testing/api-load (crate renamed)
- All test crates: get_api_gateway_addr → get_api_addr + variable renames

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-04 23:38:21 +01:00
jgrusewski
e047c1eea3 refactor: rename all service crates to kebab-case 
Rename 7 service binaries from snake_case to kebab-case to match
K8s deployment names. Update Cargo.toml package/bin names, K8s
manifest S3 paths and commands, and cross-crate dependency keys.

- api_gateway → api-gateway
- trading_service → trading-service
- broker_gateway_service → broker-gateway
- ml_training_service → ml-training-service
- backtesting_service → backtesting-service
- trading_agent_service → trading-agent-service
- data_acquisition_service → data-acquisition-service

broker-gateway gets an explicit [lib] name = "broker_gateway_service"
since its new package name maps to broker_gateway (not the original
broker_gateway_service used in source code). All other services map
correctly with Rust's automatic hyphen-to-underscore conversion.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-04 22:06:00 +01:00
jgrusewski
463b2cd130 refactor(fxt): complete TLI→FXT rename + fix token storage panic 
- Rename TliConfig→FxtConfig, TliError→FxtError, TliResult→FxtResult
- Migrate token storage path foxhunt-tli→foxhunt-fxt with auto-migration
- Fix FileTokenStorage::Default panic (fallback to /tmp on missing HOME)
- Update all doc comments, login prompt, config.toml.example, env vars
- Update keyring service names foxhunt-tli-access→foxhunt-fxt-access
- Add TOKEN_REFRESH_BUFFER_SECS named constant
- Add clarifying comments for JWT dummy key and IBKR default client ID

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-04 14:43:07 +01:00
jgrusewski
5fe3608d92 fix(fxt,infra): production hardening — OTLP telemetry, TUI fixes, K8s infra 
- Remove opentelemetry-otlp internal-logs feature (OTLP feedback loop)
- Switch trace sampling from AlwaysOn to 10% ratio-based
- Add RUST_LOG filtering (opentelemetry/h2/tonic/hyper=warn) to all 8 services
- Wire per-service latency measurement via health check → proto metadata → TUI
- Replace Vec::remove(0) with VecDeque ring buffers (O(1) vs O(n))
- Add Arc<AtomicBool> connected_sent for first-connected detection across 12 streams
- Add MAX_RECONNECT_ATTEMPTS (10) uniformly to all stream spawners
- Change kill switch/circuit breaker fields to Option types with N/A display
- Wire data_cache to real download status stream, remove dead cluster_events
- Remove ServiceData::new() hardcoded stubs, add honest placeholders
- Fix nanos_to_hms zero/negative guard, total_records semantic fix
- Fix RwLock held across yield in broker_gateway stream_account_state
- Add break after yield Err in broker/trading stream generators
- Fix connected_at advancing per tick in stream_session_status
- Tempo: replace emptyDir with 10Gi PVC, bump memory to 512Mi/2Gi
- Remove Prometheus gitlab-annotated-pods duplicate scrape job
- Wire 6 new gRPC streaming adapters (risk, trading, ml, data-acquisition)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-04 10:39:56 +01:00
jgrusewski
609f533abc feat: implement all 15 fxt CLI commands with real gRPC calls 
- 13 commands with full gRPC implementations: service, train, tune,
  model, trade, broker, agent, data, risk, config, cluster, auth, backtest
- Streaming support: train logs --follow, broker executions --follow
- --json output on every command via OutputFormat/HumanReadable
- Fix web-gateway monitoring URL default (50057 → 50051, API Gateway)
- Rewire 7 remaining build.rs to consolidated proto/ root (web-gateway,
  backtesting_service, training_uploader, 3 test crates, e2e)
- Fix web-gateway ml_training.proto new fields (mode, max_epochs, resume)
- 75 fxt tests + 139 web-gateway tests, 0 clippy warnings, workspace clean

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-03 22:16:35 +01:00
jgrusewski
eed268c230 chore: update Cargo.lock and regenerated e2e proto 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-03 02:02:20 +01:00
jgrusewski
2fbb19d9df refactor(ml): rename real_data_loader to data_loader 
The real_ prefix was misleading — there is no fake data loader.
Mechanical rename across 18 source files, no logic changes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-02 15:41:23 +01:00
jgrusewski
2aedc2ae1a feat(ml): comprehensive GPU saturation audit — 58 fixes across all 10 models 
Phase 1 — Fix broken models (P0):
- Diffusion: wire optimizer_step to actually apply gradients (was no-op)
- TLOB: connect forward pass to projection layers (was Tensor::zeros)
- Mamba2: F64→F32 migration across 5 files (~30x faster on L40S tensor cores)

Phase 2 — Eliminate hot-path GPU sync stalls:
- Mamba2: keep dt on GPU in discretize_ssm (4 functions, no CPU round-trip)
- TFT: gate attention weight logging to eval only (8 syncs/forward eliminated)
- Mamba2: defer loss scalar after backward (pipeline stall removed)
- Mamba2: delete dead gradient clipping (4N wasted GPU syncs removed)

Phase 3 — Enable BF16 for supervised models:
- Flip mixed_precision defaults to true in 4 config locations
- Fix cuda_layer_norm to support BF16/F16 via F32 intermediate

Phase 4 — Raise hyperopt bounds for datacenter GPUs:
- 7 adapters with VRAM-aware tiers (TFT, Liquid, TGGN, KAN, xLSTM,
  Diffusion, TLOB) — L40S gets full hidden_dim range
- Fix L40S tier boundary (was excluded at <48000, now >=40000)

Phase 5 — Update memory estimates:
- 10 param_count estimates updated (DQN 200K→12M, TFT 2M→50M, etc.)
- Fix power-of-two rounding (was wasting up to 49% of budget)
- Correct MODEL_OVERHEAD_MB in DQN/PPO/TFT adapters

Phase 6 — Fix per-epoch CPU bottlenecks:
- PPO: deduplicate double advantage normalization (correctness fix)
- PPO: GPU tensor reward normalization + explained variance
- Fuse per-parameter grad norm to single GPU sync (xLSTM, KAN, TGGN)

Phase 7 — Data pipeline:
- GpuBufferPool: use from_slice (eliminate staging buffer copy)

Phase 8 — Correctness:
- TFT: remove broken .detach() in forward_checkpointed (restore gradients)
- Update stale RTX 3050 Ti doc references

33 files changed, 2451 tests pass, 0 clippy warnings.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-02 15:41:23 +01:00
jgrusewski
e2a0576ef5 docs: create/update README.md for all services, CLI, and testing crates 
Create 3 missing service READMEs (api_gateway, data_acquisition_service,
trading_agent_service). Create bin/fxt/README.md. Create
testing/service-integration/README.md. Update existing service and testing
READMEs to standard template. Delete 4 subdirectory READMEs from
testing/integration/ and testing/e2e/.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-01 22:50:38 +01:00
jgrusewski
4a1add5806 cleanup: delete 92 scattered .md files and .serena artifacts 
Remove stale test reports, quick-start guides, benchmark analyses,
profiling reports, and tool artifacts from across the workspace.
Keeps only root README.md per crate/service.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-01 22:42:36 +01:00
jgrusewski
afd85b2f8f chore: clean up examples, update ML binaries and risk tests 
- Delete 14 unused example files (-3,543 lines): config, adaptive-strategy,
  data, storage, trading_engine, api_gateway, backtesting, trading_service, chaos
- Update ML training/eval binaries: improved CLI args, completion tracking,
  CUDA test cleanup, hyperopt enhancements
- Fix KAN network and TFT module adjustments
- Update risk test assertions for consistency
- Fix backtesting repositories and promotion manager
- Update .serena project config and Cargo dependencies

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-27 01:33:18 +01:00
jgrusewski
de8d7b86f8 chore: commit generated proto stubs for on-demand training RPCs 
The e2e test crate uses pre-generated proto code (no build.rs).
Missing stubs for: ReportJobCompletion, ListPendingPromotions,
ApprovePromotion, RejectPromotion + message types.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-26 13:55:40 +01:00
jgrusewski
d7f11449f4 fix(e2e): mark infrastructure-dependent tests as #[ignore] 
E2E framework tests require running services (database, JWT, ML
pipeline) that aren't available in CI unit test stage.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-25 14:00:49 +01:00
jgrusewski
9c3d741a08 refactor: restructure repo — crates/, bin/, testing/ layout 
Move 17 library crates into crates/, CLI binary into bin/fxt,
consolidate 10 test crates into testing/, split config crate
from deployment config files.

Root directory reduced from 38+ to ~17 directories.
All Cargo.toml paths and build.rs proto refs updated.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-25 11:56:00 +01:00