Thanos compact iterating way too slowly

[bck01215]

I was using thanos compact and noticed compactions regularly completing. After adding another (not the first) otel services writing to our prometheus HA pair, we got an alert for compactions not completing. This may be unrelated but occurred around the same time.

Our original args were:

```yaml
      containers:
        - args:
            - compact
            - --wait
            - --wait-interval=12h
            - --log.level=warn
            - --log.format=json
            - --objstore.config=$(OBJSTORE_CONFIG)
            - --data-dir=/var/thanos/compact
            - --debug.accept-malformed-index
            - --retention.resolution-raw=0d
            - --retention.resolution-5m=30d
            - --retention.resolution-1h=3650d
            - --delete-delay=48h
            - --compact.concurrency=1
            - --compact.enable-vertical-compaction
            - --downsample.concurrency=1
            - --block-files-concurrency=4
            - --deduplication.replica-label=prometheus_replica
            - --deduplication.replica-label=rule_replica
            - --deduplication.replica-label=replica
            - --deduplication.replica-label=prom_team
            - --deduplication.replica-label=receiver_replica
            - --deduplication.func=penalty
            - --min-time=-180d

To attempt to speed up compactions our new args are:

      containers:
        - args:
            - compact
            - --wait
            - --wait-interval=12h
            - --log.level=debug
            - --log.format=json
            - --objstore.config=$(OBJSTORE_CONFIG)
            - --data-dir=/var/thanos/compact
            - --retention.resolution-raw=0d
            - --retention.resolution-5m=30d
            - --retention.resolution-1h=3650d
            - --compact.cleanup-interval=0s
            - --compact.progress-interval=0s
            - --delete-delay=48h
            - --compact.concurrency=8
            - --compact.enable-vertical-compaction
            - --downsample.concurrency=4
            - --block-files-concurrency=8
            - --deduplication.replica-label=prometheus_replica
            - --deduplication.replica-label=rule_replica
            - --deduplication.replica-label=replica
            - --deduplication.replica-label=prom_team
            - --deduplication.replica-label=receiver_replica
            - --deduplication.func=penalty
            - --min-time=-30d
            - --max-time=-3d
            - |
              --tracing.config=type: ELASTIC_APM
              config:
                service_name: "thanos-compact"
                service_version: "0.40.1"
                service_environment: "prod"
                sample_rate: 1.00

We are seeing no errors in the logs (just continuous attempts to down), the compactions never complete, blocks still sync, and the data directory gets stuck on one compaction using one ~ 1 CPU memory maxing out.

The blocks do not seem too large.

Details of overlapping blocks

Start Time: May 11, 2026 6:00 AM
End Time: May 11, 2026 8:00 AM
Duration: 2 hours
Series: 5940369
Samples: 1781368010
Chunks: 14021472
Total size: 5.02 GiB
Chunks: 4.16 GiB (82.88%)
Index: 879.48 MiB (17.12%)
Daily: 60.21 GiB / day
Resolution: 0
Level: 1
Source: sidecar
Labels:
prom_team: obs
replica: prometheus-server-1

Start Time: May 11, 2026 6:00 AM
End Time: May 11, 2026 8:00 AM
Duration: 2 hours
Series: 763776
Samples: 27848014
Chunks: 1389430
Total size: 216.70 MiB
Chunks: 79.56 MiB (36.72%)
Index: 137.13 MiB (63.28%)
Daily: 2.54 GiB / day
Resolution: 0
Level: 1
Source: sidecar
Labels:
prom_team: obs
replica: prometheus-server-1

Start Time: May 11, 2026 6:00 AM
End Time: May 11, 2026 8:00 AM
Duration: 2 hours
Series: 5940369
Samples: 1781368010
Chunks: 14021472
Total size: 5.02 GiB
Chunks: 4.16 GiB (82.88%)
Index: 879.48 MiB (17.12%)
Daily: 60.21 GiB / day
Resolution: 0
Level: 1
Source: sidecar
Labels:
prom_team: obs
replica: prometheus-server-1

Then similarly repeated for
prom_team: obs
replica: prometheus-server-1
The two smaller blocks have this hint in the meta,json

    "hints": [
      "from-out-of-order"
    ]

The longest I have attempted to let it run with the new config is roughly 48 hours without a single compaction finished.

[bck01215]

profile.pb.gz