Thanos Store : strange permissions in created directory

[JohnGrime]

Docker image:tag: quay.io/thanos/thanos:v0.41.0

Object Storage Provider: Containerized instance of MinIO (image: mini/minio - slightly old, but functional)

What happened:

I'm trying to run the Thanos Store gateway as part of a Docker Compose ensemble.

The relevant compose.yaml service entries:

  minio:
    image: ${MINIO_CONTAINER_IMAGE}
    container_name: minio
    command: server /data --console-address ":9001"
    environment:
      MINIO_ROOT_USER: ${MINIO_ADMIN_USER}
      MINIO_ROOT_PASSWORD: ${MINIO_ADMIN_PASSWORD}
    volumes:
      - minio-data:/data
    ports:
      - "9000:9000"
      - "9001:9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 10s
      timeout: 5s
      retries: 12

  minio-init:
    image: ${MINIO_CONTAINER_IMAGE}
    container_name: minio-init
    depends_on:
      minio:
        condition: service_healthy
    entrypoint: >
      /bin/sh -c "
      mc alias set local http://minio:9000 ${MINIO_ADMIN_USER} ${MINIO_ADMIN_PASSWORD};
      mc mb --ignore-existing local/${THANOS_BUCKET};
      mc anonymous set none local/${THANOS_BUCKET};
      exit 0;
      "
    restart: "no"

  thanos-store:
    image: ${THANOS_CONTAINER_IMAGE}
    container_name: thanos-store
    command:
      - store
      - --data-dir=/var/thanos/store
      - --objstore.config-file=/etc/thanos/objstore.yml
      - --grpc-address=0.0.0.0:10901
      - --http-address=0.0.0.0:10902
      - --log.level=info
    depends_on:
      - minio-init
    volumes:
      - thanos-store-data:/var/thanos/store
      - ./configs/objstore.yml:/etc/thanos/objstore.yml:ro
    ports:
      - "31901:10901"
      - "31902:10902"

Relevant compose.yaml volume entries:

  minio-data:
  thanos-store-data:

After running docker compose up I get the following messages from the Thanos Store gateway container:

thanos-store        | ts=2026-06-08T20:58:30.55548007Z caller=factory.go:39 level=info msg="loading bucket configuration"
thanos-store        | ts=2026-06-08T20:58:30.556369637Z caller=inmemory.go:185 level=info msg="created in-memory index cache" maxItemSizeBytes=131072000 maxSizeBytes=262144000 maxItems=maxInt
thanos-store        | ts=2026-06-08T20:58:30.556634112Z caller=main.go:151 level=error err="mkdir /var/thanos/store/meta-syncer: permission denied\nmeta fetcher\nmain.runStore\n\t/app/cmd/thanos/store.go:407\nmain.registerStore.func1\n\t/app/cmd/thanos/store.go:264\nmain.main\n\t/app/cmd/thanos/main.go:149\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:285\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693\npreparing store command failed\nmain.main\n\t/app/cmd/thanos/main.go:151\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:285\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693"
thanos-store exited with code 1

As a check, I ran chmod -R 777 /path/to/thanos-store-data/volume (where the path was the directory indicated by docker volume inspect) after calling docker compose down.

Running the ensemble again, the thanos-store container apparently succeeded with creating the /var/thanos/store/meta-syncer directory. The output was now:

thanos-store        | ts=2026-06-08T21:04:56.97709054Z caller=factory.go:39 level=info msg="loading bucket configuration"
thanos-store        | ts=2026-06-08T21:04:56.977835518Z caller=inmemory.go:185 level=info msg="created in-memory index cache" maxItemSizeBytes=131072000 maxSizeBytes=262144000 maxItems=maxInt
thanos-store        | ts=2026-06-08T21:04:56.978712244Z caller=options.go:29 level=info protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
thanos-store        | ts=2026-06-08T21:04:56.981789936Z caller=store.go:594 level=info msg="starting store node"
thanos-store        | ts=2026-06-08T21:04:56.983188446Z caller=store.go:492 level=info msg="initializing bucket store"
thanos-store        | ts=2026-06-08T21:04:56.992159987Z caller=intrumentation.go:75 level=info msg="changing probe status" status=healthy
thanos-store        | ts=2026-06-08T21:04:56.993787363Z caller=http.go:72 level=info service=http/server component=store msg="listening for requests and metrics" address=0.0.0.0:10902
thanos-store        | ts=2026-06-08T21:04:56.994398074Z caller=handler.go:87 level=info service=http/server component=store caller=tls_config.go:354 time=2026-06-08T21:04:56.994369978Z msg="Listening on" address=[::]:10902
thanos-store        | ts=2026-06-08T21:04:56.994441392Z caller=handler.go:87 level=info service=http/server component=store caller=tls_config.go:357 time=2026-06-08T21:04:56.99443017Z msg="TLS is disabled." http2=false address=[::]:10902
thanos-store        | ts=2026-06-08T21:04:57.000466789Z caller=fetcher.go:691 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=8.115256ms duration_ms=8 cached=0 returned=0 partial=0
thanos-store        | ts=2026-06-08T21:04:57.000619669Z caller=store.go:509 level=info msg="bucket store ready" init_duration=8.360706ms
thanos-store        | ts=2026-06-08T21:04:57.000974921Z caller=intrumentation.go:56 level=info msg="changing probe status" status=ready
thanos-store        | ts=2026-06-08T21:04:57.001055518Z caller=grpc.go:158 level=info service=gRPC/server component=store msg="listening for serving gRPC" address=0.0.0.0:10901
thanos-store        | ts=2026-06-08T21:04:57.003424599Z caller=fetcher.go:691 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=2.752827ms duration_ms=2 cached=0 returned=0 partial=0

I looked into the local directory containing the thanos-store-data volume while the containers were running to examine the meta-syncer directory:

[me]$ stat /path/to/volumes/blah_thanos-store-data/_data/meta-syncer/
  File: /path/to/volumes/blah_thanos-store-data/_data/meta-syncer/
  Size: 6               Blocks: 0          IO Block: 4096   directory
Device: fd04h/64772d    Inode: 190747      Links: 2
Access: (0755/drwxr-xr-x)  Uid: (101000/ UNKNOWN)   Gid: (101000/pubarch-borr-ro)
Access: 2026-06-08 17:04:56.998694408 -0400
Modify: 2026-06-08 17:04:56.976694410 -0400
Change: 2026-06-08 17:04:56.976694410 -0400
 Birth: 2026-06-08 17:04:56.976694410 -0400

Hmmm. Who is the pubarch-borr-ro user? That's not the user thanos-store is running as in the container:

[me]$ docker ps
CONTAINER ID   IMAGE                           COMMAND                  CREATED         STATUS                   PORTS                                                                                              NAMES
...
d9647e93d295   quay.io/thanos/thanos:v0.41.0   "/bin/thanos store -…"   2 minutes ago   Up 2 minutes             0.0.0.0:31901->10901/tcp, [::]:31901->10901/tcp, 0.0.0.0:31902->10902/tcp, [::]:31902->10902/tcp   thanos-store
...
0.0.0.0:9000-9001->9000-9001/tcp, [::]:9000-9001->9000-9001/tcp                                    minio
[me]$ docker exec -it d9647e93d295 sh
/ $ ps
PID   USER     TIME  COMMAND
    1 thanos    0:00 /bin/thanos store --data-dir=/var/thanos/store --objstore.config-file=/etc/thanos/objstore.yml --grpc-address=0.0.0.0:10901 --http-address=0.0.0.0:10902 --log.level=info
   17 thanos    0:00 sh
   23 thanos    0:00 ps
/ $ 

Why is the meta-syncer directory created by the user pubarch-borr-ro? Who is that user? A quick Google turned up nothing useful, and nor did I find anything useful by searching the Thanos repo ...

What you expected to happen: I'd expect:

1. The Thanos Store gateway to be able to create the meta-syncer directory in the /var/thanos/store directory; that directory is actually mounted in from a Docker volume defined in compose.yaml. The same volume creation/mapping approach does not break Thanos commands in other containers run from the same source image (for example, Sidecar instances).
2. The Thanos Store gateway might also create the meta-suncer directory under e.g. the thanos user and not the mysterious pub arch-borr-ro user.

How to reproduce it (as minimally and precisely as possible):

Contents of .env file :

MINIO_ADMIN_USER=whatever
MINIO_ADMIN_PASSWORD=whatever

THANOS_CONTAINER_IMAGE=quay.io/thanos/thanos:v0.41.0
MINIO_CONTAINER_IMAGE=minio/minio

THANOS_BUCKET=thanos

compose.yaml:

services:

  minio:
    image: ${MINIO_CONTAINER_IMAGE}
    container_name: minio
    command: server /data --console-address ":9001"
    environment:
      MINIO_ROOT_USER: ${MINIO_ADMIN_USER}
      MINIO_ROOT_PASSWORD: ${MINIO_ADMIN_PASSWORD}
    volumes:
      - minio-data:/data
    ports:
      - "9000:9000"
      - "9001:9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 10s
      timeout: 5s
      retries: 12

  minio-init:
    image: ${MINIO_CONTAINER_IMAGE}
    container_name: minio-init
    depends_on:
      minio:
        condition: service_healthy
    entrypoint: >
      /bin/sh -c "
      mc alias set local http://minio:9000/ ${MINIO_ADMIN_USER} ${MINIO_ADMIN_PASSWORD};
      mc mb --ignore-existing local/${THANOS_BUCKET};
      mc anonymous set none local/${THANOS_BUCKET};
      exit 0;
      "
    restart: "no"

  thanos-store:
    image: ${THANOS_CONTAINER_IMAGE}
    container_name: thanos-store
      #    user: jgrimeadmin # <- avoid permissions issues where Prometheus/Sidecar container share a volume
    command:
      - store
      - --data-dir=/var/thanos/store
      - --objstore.config-file=/etc/thanos/objstore.yml
      - --grpc-address=0.0.0.0:10901
      - --http-address=0.0.0.0:10902
      - --log.level=info
    depends_on:
      - minio-init
    volumes:
      - thanos-store-data:/var/thanos/store:rw
      - ./configs/objstore.yml:/etc/thanos/objstore.yml:ro
    ports:
      - "31901:10901"
      - "31902:10902"

volumes:
  minio-data:
  thanos-store-data:

Then docker compose up

Full logs to relevant components:

See log messages, previous.

Anything else we need to know:

Red Hat Enterprise Linux 9.7, Docker version 29.5.3, build d1c06ef