Control sharding codec read coalescing with ArrayConfig and runtime config options

[aldenks]

Follow up #3004 by adding ArrayConfig and runtime configuration options for the thresholds that control how requests are coalesced when reading in the sharding codec.

Two new fields on ArrayConfig control how the sharding codec coalesces partial-shard reads: sharding_coalesce_max_gap_bytes (default 1 MiB) and sharding_coalesce_max_bytes (default 16 MiB). When reading multiple chunks from the same shard, nearby byte ranges are merged into a single request to the store if separated by no more than sharding_coalesce_max_gap_bytes and the merged read stays within sharding_coalesce_max_bytes. Defaults are seeded from the matching array.sharding_coalesce_max_gap_bytes / array.sharding_coalesce_max_bytes keys in zarr.config at array-creation time, and can be overridden per array by passing config={...} to zarr.create_array.

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/user-guide/*.md
• Changes documented as a new file in changes/
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.53%. Comparing base (7bc3017) to head (bde0ee3).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3987   +/-   ##
=======================================
  Coverage   93.52%   93.53%           
=======================================
  Files          90       90           
  Lines       11926    11938   +12     
=======================================
+ Hits        11154    11166   +12     
  Misses        772      772           

Files with missing lines	Coverage Δ
src/zarr/codecs/sharding.py	91.71% <ø> (ø)
src/zarr/core/array_spec.py	100.00% <100.00%> (ø)
src/zarr/core/common.py	88.82% <100.00%> (+0.26%)	⬆️
src/zarr/core/config.py	100.00% <ø> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[d-v-b]

disclaimer: I'm not a big fan of our global config object, so I'd like to explore some alternative ways for the sharding reads to access this configuration.

A few options:

• new attributes on the sharding codec
  I'm not a big fan of this, because declaring the sharding codec explicitly from create_array is tedious, and also because we want to move away from the codecs knowing too much about IO operations.
• new fields on ArrayConfig
  probably the best option. we still use the global config for setting the defaults, but the sharding codec gets these parameters from the array config object, which is tied to the array, not a mutable global.

[aldenks]

@d-v-b I also like new fields on ArrayConfig. Thinking that through:

• This allows you to set coalesce options differently per array
• If you only use global configs, whatever global setting you have at the time of array open is what is used for the life of that array.
• The global config field becomes something like array.sharding_coalesce_max_gap_bytes and array.sharding_coalesce_max_bytes matching the ArrayConfig convention of pulling from a singly nested field under array..
• For the time being, if you're interacting with zarr via xarray then you still can only set these via the global config but that's a pre-existing inability to specify the ArrayConfig via when opening via xarray.

That sound alright?

[d-v-b]

@d-v-b I also like new fields on ArrayConfig. Thinking that through:

* This allows you to set coalesce options differently per array

* If you only use global configs, whatever global setting you have _at the time of array open_ is what is used for the life of that array.

* The global config field becomes something like `array.sharding_coalesce_max_gap_bytes` and `array.sharding_coalesce_max_bytes` matching the ArrayConfig convention of pulling from a singly nested field under `array.`.

* For the time being, if you're interacting with zarr via xarray then you still can only set these via the global config but that's a pre-existing inability to specify the ArrayConfig via when opening via xarray.

That sound alright?

yeah, that sounds right. the array config object is designed to make it easy to get a cheap copy of an array with a new config, using the with_config method. Unfortunately, xarray makes it very hard to use this, because xarray doesn't give access to the base zarr array. So until xarray adds a zarr array-config-aware API, the global config is the only knob xarray users have, without re-creating the dataarray entirely.

[aldenks]

ArrayConfig options implemented, pulling from the global config if not set explicitly. This is ready for review!

[d-v-b]

flagging this as an old problem we need to deal with, not a blocker: the default values appear twice, once here, and once in the global config object. that's a latent bug. the defaults should be defined in exactly 1 place.

[aldenks]

+1 For now I added test_array_config_init_defaults_match_global_config which makes sure they don't drift.

[aldenks]

@d-v-b ready for merge! thank you for the review