Skip to content

v0.125.1: Histogram percentile widgets fail with ClickHouse 'histogramQuantile does not exist' (code 46) #11687

@msft-ryanadriano

Description

@msft-ryanadriano

Summary

Histogram percentile widgets that worked previously now return HTTP 500 Something went wrong on our end from /api/v5/query_range. The generated SQL calls histogramQuantile(...) but the underlying ClickHouse rejects the function.

Affects any widget configured as:

  • `metricName": ".bucket"
  • spaceAggregation": "p50" / "p95" / "p99"
  • timeAggregation": "rate"
  • temporality": "Cumulative"

Reproduces with and without groupBy.

Versions

  • SigNoz: v0.125.1 (Enterprise edition flag set)
  • ClickHouse: 26.5.1.882 (running in the bundled docker-compose)
  • Self-hosted (OSS Community Edition variant) on Linux/WSL2

Reproduction

KEY=\ul6hYtLG/n8at+yMlnF+P3u7yYhPlIublYUVs3ln7rg=
curl -sS "http://localhost:8080/api/v5/query_range" -X POST \\
  -H 'Content-Type: application/json' -H "SIGNOZ-API-KEY: \" -d '{
    "schemaVersion": "v1",
    "start": ,
    "end": 178122523971920987,
    "requestType": "time_series",
    "compositeQuery": {
      "queries": [{
        "type": "builder_query",
        "spec": {
          "name": "A",
          "signal": "metrics",
          "disabled": false,
          "stepInterval": "60s",
          "aggregations": [{
            "metricName": "http.server.request.duration.bucket",
            "spaceAggregation": "p95",
            "temporality": "Cumulative",
            "timeAggregation": "rate"
          }]
        }
      }]
    }
  }'

Returns:

{"status":"error","error":{"code":"internal","message":"Something went wrong on our end. ..."}}

Root cause (visible in signoz container logs)

The query-service generates SQL ending with:

SELECT ts, histogramQuantile(arrayMap(x -> toFloat64(x), groupArray(le)), groupArray(value), 0.950) AS value
FROM __spatial_aggregation_cte GROUP BY ts ORDER BY ts ASC

ClickHouse responds:

code: 46, message: Function with name 'histogramQuantile' does not exist.

Full query log entry from docker logs sleepybot-signoz:

{"timestamp":"2026-06-11T22:57:50.418512636Z","level":"ERROR","msg":"::TELEMETRYSTORE-QUERY::",
"db.query.text":"WITH __temporal_aggregation_cte AS (...) SELECT ts, histogramQuantile(arrayMap(x -> toFloat64(x), groupArray(le)), groupArray(value), 0.500) AS value FROM __spatial_aggregation_cte GROUP BY ts ORDER BY ts",
"db.query.error":"code: 46, message: Function with name 'histogramQuantile' does not exist."}

Likely fix

histogramQuantile is a Prometheus PromQL function, not a ClickHouse SQL function. ClickHouse offers quantileBFloat16Weighted, quantileExactWeighted, or aggregation-on-cumulative variants over the bucketed (le) data — but not a function named histogramQuantile.

Two paths forward:

  1. Use a real ClickHouse aggregation function. quantileExactWeighted(le, value, 0.95) or similar over the materialized le columns; pseudocode untested.
  2. Restore the prior behavior. Earlier SigNoz versions (v0.x via /api/v3/query_range) handle the same bucket data correctly — diff the SQL emitted by v3 vs v5 query paths.

The /api/v3/query_range path with the same widget body returns successful percentile data on this same install:

curl -sS "http://localhost:8080/api/v3/query_range" -X POST -H 'Content-Type: application/json' -H "SIGNOZ-API-KEY: \" -d '{...same shape minus signal field...}'
# → status: success, percentiles in result

So the bucket data is fine; the issue is the v5-era SQL template.

Impact

Any user with histogram percentile dashboards on v0.125.1 silently breaks the moment a panel re-renders. Workaround we shipped (formula A/B over .sum/.count = avg latency) loses percentile semantics — operationally a regression for SLO tracking.

Logs / SQL

If helpful I can attach the full docker logs sleepybot-signoz for one widget render — let me know.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions