Config v2 steel-thread landing plan

[MrAlias]

Reference PoC: #2080

Summary

#2080 proves the config v2 steel thread, but it is too large to review and should not be treated as the final implementation shape. The plan is to extract it into serial, reviewable PRs. Each implementation PR is based on main after the previous PR merges.

Because this may span multiple releases, no public config v2 behavior should ship until the final gated release group. Early PRs must stay internal: no public Go package, no CLI, no runtime auto-detection, no Collector receiver support, and no user-facing docs that tell users to use config v2.

Parallel Workstream: Config Drift CI

This can happen independently of the implementation split.

• Move the existing parity checker into cmd/check-config-v2-parity.
  • Treat this mostly as a move, not new review burden.
  • Keep it simple and short-lived.
• Add make check-config-v2-parity.
• Wire the target into PR checks.
• Fail CI when v1 defaults and merged config v2 definitions drift unintentionally.

Estimated review burden: ~60-150 changed lines, plus moved checker code.

Serial Implementation PRs

Review-size rule: target ~300 changed logic/test lines; do not exceed ~800. Pure moves and generated artifacts do not count as review burden when they are not mixed with logic changes.

1. Internal config schema package

   • Add hidden parser/model under internal/config/schema.
   • Include v2 document/extension structs, version detection, standalone vs receiver validation, and tests.
   • No pkg/config/v2.
   • Estimate: ~600 lines.

2. Internal v1-to-v2 conversion foundation

   • Add internal/config/convert.
   • Convert default/runtime config into hidden v2 document shape for core capture fields, protocol enablement, basic network capture, and daemon basics.
   • Estimate: ~500-700 lines.

3. Export conversion parity

   • Split the original export parity scope into two review-sized PRs:
     • Capture export parity (Expand internal config v2 capture export parity #2269): discovery rule export, rule refinements, application/network/stats filters, HTTP routes, HTTP enrichment, GenAI payload extraction, SQL MSSQL fields, network CIDRs, network enrichment, stats, diagnostics, and capture engine additions.
     • Standalone/enrichment/pipeline export parity (Add standalone config v2 export parity #2288): top-level resource, tracer_provider, meter_provider, capture telemetry reporter cache/TTL fields, Kubernetes/name-resolution/attribute enrichment, correlation/log trace annotation, and daemon Prometheus telemetry fields.
   • Keep changes internal and test-driven.
   • Estimate: ~500-750 lines per PR.

4. Typed v2 export builders

   • Replace ad hoc multi-level map construction in the export converter with typed builders or typed intermediate structs for stable v2 sections.
   • Keep the rendered YAML/document output compatible with the current config v2 shape.
   • Render to maps only at schema boundaries that need to preserve arbitrary declarative fields.
   • Focus on exporter-owned sections before import conversion so exported field names and nesting get compile-time coverage.
   • Keep the change internal and review-sized.
   • Estimate: ~500-750 lines.

5. Internal v2-to-runtime conversion foundation

   • Convert hidden v2 schema into obi.Config for core capture, protocol enablement, network basics, and standalone-only sections.
   • No runtime wiring yet.
   • Estimate: ~500-750 lines.

6. Expand import conversion parity

   • Add advanced rule/selectors, resource identity, Kubernetes fields, attributes, stats, HTTP enrichment, payload extraction, and pipeline import behavior.
   • Estimate: ~500-750 lines.

7. Hidden artifact/tooling support

   • Add internal generation/check plumbing needed by tests and CI, but do not publish user-facing v2 docs or expose a release binary command.
   • Generated artifacts should be clearly marked and reviewed separately from logic.

Final Release-Gated Group

These PRs expose the feature and must land in the same release window.

1. Expose CLI

   • Add obi config validate and obi config migrate.
   • Mark as release-gated.

2. Expose standalone runtime support

   • Add v2 detection in obi startup with fallback to v1 only on explicit not-v2 errors.
   • Mark as release-gated.

3. Expose Collector receiver support

   • Add receiver-mode v2 parsing.
   • Reject standalone-only sections in receiver mode.
   • Mark as release-gated.

4. Publish docs and artifacts

   • Publish config v2 docs, generated schema/example artifacts, and migration guidance.
   • Promote any public Go API only if maintainers explicitly decide one is needed.
   • Mark as release-gated.

Acceptance Criteria

• Parity CI is merged independently and catches v1/v2 config drift.
• All pre-gate PRs are internal-only and safe to ship across releases.
• No pkg/config/v2, CLI command, runtime v2 parsing, receiver v2 parsing, or user-facing config v2 docs ship before the gated group.
• Each serial PR is green on its own.
• Final gated PRs expose and document the feature together in one release.

[MrAlias]

Implementation note for serial PR 2 ("Internal v1-to-v2 conversion foundation"):

I’m adjusting the package name from the issue shorthand internal/configconv to internal/config/convert.

Decision:

• Use path internal/config/convert.
• Use package name convert.
• Use direction-explicit entry points such as RuntimeToV2(cfg) for the forward conversion foundation.

Rationale:

• It keeps config-v2 internals grouped under internal/config/..., matching the already-merged internal/config/schema package from PR Add internal config v2 schema parser #2252.
• It avoids the stutter that would come from internal/config/configconv.
• It gives clear future call sites: convert.RuntimeToV2(cfg) now, and the later import-side PR can naturally add convert.V2ToRuntime(...).
• It keeps the conversion layer internal-only and does not imply a public migration API or CLI surface.

Scope impact:

• This is a naming/organization decision only. The implementation slice remains the same: internal v1 runtime/default config to hidden v2 document shape for capture foundation fields, protocol enablement, basic network capture, and daemon basics.
• No public pkg/config/v2, CLI command, runtime auto-detection, receiver support, or user-facing docs are introduced by this naming change.

[MrAlias]

Plan adjustment for serial PR 3 after drafting #2269:

#2269 currently exceeds the review-size ceiling because the originally planned "Expand export conversion parity" slice contains two fairly independent review areas. To keep the series reviewable, I’m splitting that item into two export-parity PRs:

1. Capture export parity (Expand internal config v2 capture export parity #2269)

   • discovery rule export
   • rule refinements for per-selector exports and HTTP routes
   • application/network/stats filters
   • HTTP routes, payload extraction, HTTP enrichment, and GenAI payload extraction
   • SQL MSSQL export fields
   • network CIDRs, enrichment, stats, diagnostics, and guess_ports
   • capture engine additions such as force_map_reader and map scale factor

2. Standalone/enrichment/pipeline export parity (follow-up PR)

   • top-level resource
   • tracer_provider and meter_provider
   • capture telemetry reporter cache/TTL fields
   • Kubernetes/name-resolution/attribute enrichment
   • correlation/log trace annotation
   • daemon Prometheus telemetry fields

I’m going to trim #2269 to the first scope with a follow-up cleanup commit rather than amending the original commit, so the original full draft remains available as a concrete reference for the next PR.

[MrAlias]

Follow-up from the #2269 capture export parity slice: v1 per-selector HTTP routes distinguish incoming and outgoing, while the current v2 capture.rules[].refine.http.routes shape only has directionless route patterns. #2269 should warn and skip those directional route overrides rather than exporting a widened directionless override.

Action item for a later config-v2 slice: define a direction-aware per-rule HTTP route refinement shape, then add export/import parity for v1 routes.incoming and routes.outgoing.