This release contains 578 PRs from 62 authors, including new contributors Andrew Hall, Frank Kloeker, Gregor Zeitlinger, Ilia Lazebnik, Julien, Maruth Goyal, Matt Clegg, Mattia Barbon, Mohijeet, Oleg Vorobev, Oliver Herrmann, Phil Kates, Pranshu Raj, Rob B, Timon Engelke, Vadim Stepanov, dannyc-grafana, unkls ben. Thank you!

Grafana Mimir version 3.0.0-rc.0 release notes

Changelog

3.0.0-rc.0

Grafana Mimir

• [CHANGE] Build: Include updated Mozilla CA bundle from Debian Testing. #12247
• [CHANGE] Query-frontend: Add support for UTF-8 label and metric names in /api/v1/cardinality/{label_values|label_values|active_series} endpoints. #11848.
• [CHANGE] Querier: Add support for UTF-8 label and metric names in label_join, label_replace and count_values PromQL functions. #11848.
• [CHANGE] Remove support for Redis as a cache backend. #12163
• [CHANGE] Memcached: Remove experimental -<prefix>.memcached.addresses-provider flag to use alternate DNS service discovery backends. The more reliable backend introduced in 2.16.0 (#10895) is now the default. As a result of this change, DNS-based cache service discovery no longer supports search domains. #12175 #12385
• [CHANGE] Query-frontend: Remove the CLI flag -query-frontend.downstream-url and corresponding YAML configuration and the ability to use the query-frontend to proxy arbitrary Prometheus backends. #12191 #12517
• [CHANGE] Query-frontend: Remove experimental instant query splitting feature. #12267
• [CHANGE] Query-frontend, querier: Replace query-frontend.prune-queries flag with querier.mimir-query-engine.enable-prune-toggles as pruning middleware has been moved into MQE. #12303 #12375
• [CHANGE] Distributor: Remove deprecated global HA tracker timeout configuration flags. #12321
• [CHANGE] Query-frontend: Use the Mimir Query Engine (MQE) by default. #12361
• [CHANGE] Query-frontend: Remove the CLI flags -querier.frontend-address, -querier.max-outstanding-requests-per-tenant, and -query-frontend.querier-forget-delay and corresponding YAML configurations. This is part of a change that makes the query-scheduler a required component. This removes the ability to run the query-frontend with an embedded query-scheduler. Instead, you must run a dedicated query-scheduler component. #12200
• [CHANGE] Ingester: Remove deprecated -ingester.stream-chunks-when-using-blocks CLI flag and ingester_stream_chunks_when_using_blocks runtime configuration option. #12615
• [CHANGE] Querier: Require non-zero values for -querier.streaming-chunks-per-ingester-buffer-size and -querier.streaming-chunks-per-store-gateway-buffer-size CLI flags and corresponding YAML configurations. This is part of a change that makes streaming required between queriers, ingesters, and store-gateways. Streaming has been the default since Mimir 2.14. #12790 #12818 #12897 #12929 #12973
• [CHANGE] Remove support for the experimental read-write deployment mode. #12584
• [CHANGE] Store-gateway: Update default value of -store-gateway.dynamic-replication.multiple to 5 to increase replication of recent blocks. #12433
• [CHANGE] Cost attribution: Reduce the default maximum per-user cardinality of cost attribution labels to 2000. #12625
• [CHANGE] Querier, query-frontend: Add _total suffix to cortex_mimir_query_engine_common_subexpression_elimination_duplication_nodes_introduced, cortex_mimir_query_engine_common_subexpression_elimination_selectors_eliminated and cortex_mimir_query_engine_common_subexpression_elimination_selectors_inspected metric names. #12636
• [CHANGE] Distributor: Remove the experimental setting service_overload_status_code_on_rate_limit_enabled which used an HTTP 529 error (non-standard) instead of HTTP 429 for rate limiting. #13012
• [CHANGE] Alertmanager: Change the severity for InitialSyncFailed from 'critical' to 'warning'. #12824
• [CHANGE] Ingester: Renamed experimental reactive limiter options. #12773
• [CHANGE] Distributor: gRPC errors with the mimirpb.ERROR_CAUSE_INSTANCE_LIMIT cause are now mapped to codes.Unavailable and http.StatusServiceUnavailable instead of codes.Internal and http.StatusInternalServerError. #13003 #13032
• [CHANGE] Admin: use relative links instead of absolute ones in the administration web UI. #13034
• [CHANGE] Distributor: Use memberlist by default for the HA tracker. #12998
• [CHANGE] Block-builder: Remove cortex_blockbuilder_process_partition_duration_seconds metric and related dashboard panels. #12631
• [FEATURE] Ingester: Expose the number of active series ingested via OTLP as cortex_ingester_active_otlp_series. #12678
• [FEATURE] Distributor, ruler: Add experimental -validation.name-validation-scheme flag to specify the validation scheme for metric and label names. #12215
• [FEATURE] Ruler: Add support to use a Prometheus-compatible HTTP endpoint for remote rule evaluation. See remote evaluation mode for more details. This feature can be used to federate data from multiple Mimir instances. #11415 #11833
• [FEATURE] Distributor: Add experimental -distributor.otel-translation-strategy flag to support configuring the metric and label name translation strategy in the OTLP endpoint. #12284 #12306 #12369
• [FEATURE] Query-frontend: Add query-frontend.rewrite-propagate-matchers flag that enables a new MQE AST optimization pass that copies relevant label matchers across binary operations. #12304
• [FEATURE] Query-frontend: Add query-frontend.rewrite-histogram-queries flag that enables a new MQE AST optimization pass that rewrites histogram queries for a more efficient order of execution. #12305
• [FEATURE] Query-frontend: Support delayed name removal (Prometheus experimental feature) in MQE. #12509
• [FEATURE] Usage-tracker: Introduce a new experimental service to enforce active series limits before Kafka ingestion. #12358 #12895 #12940 #12942 #12970
• [FEATURE] Ingester: Add experimental -include-tenant-id-in-profile-labels flag to include tenant ID in pprof profiling labels for sampled traces. Currently only supported by the ingester. This can help debug performance issues for specific tenants. #12404
• [FEATURE] Alertmanager: Add experimental -alertmanager.storage.state-read-timeout flag to configure the timeout for reading the Alertmanager state (notification log, silences) from object storage during the initial sync. #12425
• [FEATURE] Ingester: Add experimental -blocks-storage.tsdb.index-lookup-planning.* flags to configure use of a cost-based index lookup planner. This should reduce the cost of queries in the ingester. #12197 #12199 #12245 #12248 #12457 #12530 #12407 #12460 #12550 #12597 #12603 #12608 #12658 #12696 #12731 #12755 #12738 #12752 #12807 #12830 #12896 #13039
• [FEATURE] MQE: Add support for applying extra selectors to one side of a binary operation to reduce data fetched. #12577
• [FEATURE] Query-frontend: Add a native histogram presenting the length of query expressions handled by the query-frontend #12571
• [FEATURE] Query-frontend and querier: Add experimental support for performing query planning in query-frontends and distributing portions of the plan to queriers for execution. #12302 #12551 #12665 #12687 #12745 #12757 #12798 #12808 #12809 #12835 #12856 #12870 #12883 #12885 #12886 #12911 #12933 #12934 #12961 #13016 #13027
• [FEATURE] Alertmanager: add Microsoft Teams V2 as a supported integration. #12680
• [FEATURE] Distributor: Add experimental flag -validation.label-value-length-over-limit-strategy to configure how to handle label values over the length limit. #12627 #12844
• [FEATURE] Ingester: Introduce metric cortex_ingester_owned_target_info_series for counting the number of owned target_info series by tenant. #12681
• [FEATURE] MQE: Add support for step invariant expression handling in query planning and evaluation. #12931
• [FEATURE] MQE: Add support for experimental ts_of_min_over_time, ts_of_max_over_time, ts_of_first_over_time and ts_of_last_over_time PromQL functions. #12819
• [FEATURE] Ingester: Add experimental flags -ingest-storage.write-logs-fsync-before-kafka-commit-enabled and -ingest-storage.write-logs-fsync-before-kafka-commit-concurrency to fsync write logs before the offset is committed to Kafka. This is enabled by default. #12816
• [FEATURE] MQE: Add support for experimental mad_over_time PromQL function. #12995
• [FEATURE] Continuous test: Add experimental -tests.ingest-storage-record.enabled flag to verify ingest-storage record correctness by validating the V2 record format against live write requests. #12500
• [ENHANCEMENT] Query-frontend: CLI flag -query-frontend.enabled-promql-experimental-functions and its associated YAML configuration is now stable. #12368
• [ENHANCEMENT] Query-scheduler/query-frontend: Add native histogram definitions to cortex_query_{scheduler|frontend}_queue_duration_seconds. #12288
• [ENHANCEMENT] Querier: Add native histogram definition to cortex_bucket_index_load_duration_seconds. #12094
• [ENHANCEMENT] Query-frontend: Allow users to set the query-frontend.extra-propagated-headers flag to specify the extra headers allowed to pass through to the rest of the query path. #12174
• [ENHANCEMENT] MQE: Add support for applying common subexpression elimination to range vector expressions in instant queries. #12236
• [ENHANCEMENT] Ingester: Improve the performance of active series custom trackers matchers. #12184
• [ENHANCEMENT] Ingester: Add postings cache sharing and invalidation. You can enable sharing and head cache invalidation via -blocks-storage.tsdb.shared-postings-for-matchers-cache and -blocks-storage.tsdb.head-postings-for-matchers-cache-invalidation respectively, and you can configure the number of metric versions per cache via -blocks-storage.tsdb.head-postings-for-matchers-cache-versions. #12333 ...

This release contains 7 PRs from 5 authors. Thank you!

Changelog

2.17.1

Grafana Mimir

• [BUGFIX] Ingester: Fix a bug ingesters would get stuck in read-only mode after compactions. #12538
• [BUGFIX] Update to Go v1.24.6 to address CVE-2025-4674, CVE-2025-47907. #12580

All changes in this release: mimir-2.17.0...mimir-2.17.1

This release contains 735 PRs from 78 authors, including new contributors Bernaud Vincent, Carrie Edwards, danieleandreatta, David Vávra, Edgaras Giedrė, Gabija Bruzgaitė, Henrique Lourenço, Innokentii Konstantinov, Jasper Maes, Jeanette Tan, Joobi S B, Julius Hinze, Lukas Bischofberger, mihaelmiklec, mimir-vendoring[bot], Nasiel, pierremahot, rektabhi, sam clulow, Shay Pletcher, Thor K. Høgås, Toni Cárdenas, Yuran Ou, zhuoyuan-liu. Thank you!

Grafana Mimir version 2.17.0 release notes

Grafana Labs is excited to announce version 2.17 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release.
For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

MQE is enabled by default in queriers. MQE provides benefits over the Prometheus engine, inluding reduced memory and CPU consumption and improved performance. To use the Prometheus engine instead of MQE, set -querier.query-engine=prometheus.

Grafana Mimir now supports using the Mimir Query Engine (MQE) in query-frontends in addition to queriers. You can enable MQE in query-frontends by setting the experimental CLI flag -query-frontend.query-engine=mimir or through the corresponding YAML option.

You can export the cortex_ingester_attributed_active_native_histogram_series and cortex_ingester_attributed_active_native_histogram_buckets native histogram cost attribution metrics to a custom Prometheus registry with user-specified labels.

Grafana Mimir supports converting OTel explicit bucket histograms to Prometheus native histograms with custom buckets using the distributor.otel-convert-histograms-to-nhcb flag.

The following experimental features have been removed:

• The max_cost_attribution_labels_per_user cost attribution limit
• Read-write deployment mode in the mixin

Important changes

In Grafana Mimir 2.17, the following behavior has changed:

The following default configuration values now apply to the memberlist KV store:

These values perform better but might cause long-running packets to be dropped in high-latency networks.

The -ruler-storage.cache.rule-group-enabled experimental CLI flag has been removed. Caching rule group contents is now always enabled when a cache is configured for the ruler.

The -ingester.ooo-native-histograms-ingestion-enabled CLI flag and corresponding ooo_native_histograms_ingestion_enabled runtime configuration option have been removed. Out-of-order native histograms are now enabled whenever both native histogram and out-of-order ingestion is enabled.

The -ingester.stream-chunks-when-using-blocks CLI flag and corresponding ingester_stream_chunks_when_using_blocks runtime configuration option have been deprecated and will be removed in a future release.

The cortex_distributor_label_values_with_newlines_total metric has been removed.

In the distributor, memberlist is marked as a stable option for backend storage for the high availability tracker. etcd has been deprecated for this purpose.

Experimental features

Grafana Mimir 2.17 includes some features that are experimental and disabled by default.
Use these features with caution and report any issues that you encounter:

• Prometheus Remote-Write 2.0 protocol.
• Duration expressions in PromQL. These are simple arithmetics on numbers in offset and range specification. For example, rate(http_requests_total[5m * 2]).
• Promoting OTel scope metadata, including name, version, schema URL, and attributes, to metric labels, prefixed with otel_scope_. Enable this feature through the -distributor.otel-promote-scope-metadata flag.
• Allowing primitive delta metrics ingestion through the OTLP endpoint with the -distributor.otel-native-delta-ingestion option.
• Support for sort_by_label and sort_by_label_desc PromQL functions.
• Support for cluster validation in HTTP calls. When enabled, the HTTP server verifies if a request coming from an HTTP client comes from an expected cluster. You can configure this validation with the following options:
  • -server.cluster-validation.label
  • -server.cluster-validation.http.enabled
  • -server.cluster-validation.http.soft-validation
  • -server.cluster-validation.http.exclude-paths

Bug fixes

For a detailed list of bug fixes, refer to the CHANGELOG.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is released independently.
Refer to the Grafana Mimir Helm chart documentation.

Changelog

2.17.0

Grafana Mimir

• [CHANGE] Query-frontend: Ensure that cache keys generated from cardinality estimate middleware are less than 250 bytes in length by hashing the tenant IDs that are included in them. This change invalidates all cardinality estimates in the cache. #11568
• [CHANGE] Ruler: Remove experimental CLI flag -ruler-storage.cache.rule-group-enabled to enable or disable caching the contents of rule groups. Caching rule group contents is now always enabled when a cache is configured for the ruler. #10949
• [CHANGE] Ingester: Out-of-order native histograms are now enabled whenever both native histogram and out-of-order ingestion is enabled. The -ingester.ooo-native-histograms-ingestion-enabled CLI flag and corresponding ooo_native_histograms_ingestion_enabled runtime configuration option have been removed. #10956
• [CHANGE] Distributor: removed the cortex_distributor_label_values_with_newlines_total metric. #10977
• [CHANGE] Ingester/Distributor: renamed the experimental max_cost_attribution_cardinality_per_user config to max_cost_attribution_cardinality. #11092
• [CHANGE] Frontend: The subquery spin-off feature is now enabled with -query-frontend.subquery-spin-off-enabled=true instead of -query-frontend.instant-queries-with-subquery-spin-off=.* #11153
• [CHANGE] Overrides-exporter: Don't export per-tenant overrides that are set to their default values. #11173
• [CHANGE] gRPC/HTTP clients: Rename metric cortex_client_request_invalid_cluster_validation_labels_total to cortex_client_invalid_cluster_validation_label_requests_total. #11237
• [CHANGE] Querier: Use Mimir Query Engine (MQE) by default. Set -querier.query-engine=prometheus to continue using Prometheus' engine. #11501
• [CHANGE] Memcached: Ignore initial DNS resolution failure, meaning don't depend on Memcached on startup. #11602
• [CHANGE] Ingester: The -ingester.stream-chunks-when-using-blocks CLI flag and ingester_stream_chunks_when_using_blocks runtime configuration option have been deprecated and will be removed in a future release. #11711
• [CHANGE] Distributor: track cortex_ingest_storage_writer_latency_seconds metric for failed writes too. Added outcome label to distinguish between success and failure. #11770
• [CHANGE] Distributor: renamed few metrics used by experimental ingest storage. #11766
  • Renamed cortex_ingest_storage_writer_produce_requests_total to cortex_ingest_storage_writer_produce_records_enqueued_total
  • Renamed cortex_ingest_storage_writer_produce_failures_total to cortex_ingest_storage_writer_produce_records_failed_total
• [CHANGE] Distributor: moved HA tracker timeout config to limits. #11774
  • Moved distributor.ha_tracker.ha_tracker_update_timeout to limits.ha_tracker_update_timeout.
  • Moved distributor.ha_tracker.ha_tracker_update_timeout_jitter_max to limits.ha_tracker_update_timeout_jitter_max.
  • Moved 8000 distributor.ha_tracker.ha_tracker_failover_timeout to limits.ha_tracker_failover_timeout.
• [CHANGE] Distributor: Memberlist marked as stable as an option for backend storage for the HA tracker. #11861
• [CHANGE] Distributor: etcd deprecated as an option for backend storage for the HA tracker. #12047
• [CHANGE] Memberlist: Apply new default configuration values for MemberlistKV. This unlocks using it as backend storage for the HA Tracker. We have observed better performance with these defaults across different production loads. #11874
  • memberlist.packet-dial-timeout: 500ms
  • memberlist.packet-write-timeout: 500ms
  • memberlist.max-concurrent-writes: 5
  • memberlist.acquire-writer-timeout: 1s
    These defaults perform better but may cause long-running packets to be dropped in high-latency networks.
• [CHANGE] Query-frontend: Apply query pruning and check for disabled experimental functions earlier in query processing. #11939
• [FEATURE] Distributor: Experimental support for Prometheus Remote-Write 2.0 protocol. Limitations: Created timestamp is ignored, per series metadata is merged on metric family level automatically, ingestion might fail if client sends ProtoBuf fields out of order. The label version is added to the metric cortex_distributor_requests_in_total with a value of either 1.0 or 2.0 depending on the detected Remote-Write protocol. #11100 #11101 #11192 #11143
• [FEATURE] Query-frontend: expand query-frontend.cache-errors and query-frontend.results-cache-ttl-for-errors configuration options to cache non-transient response failures for instant queries. #11120
• [FEATURE] Query-frontend: Allow use of Mimir Query Engine (MQE) via the experimental CLI flags -query-frontend.query-engine or -query-frontend.enable-query-engine-fallback or corresponding YAML. #11417 #11775
• [FEATURE] Querier, query-frontend, ruler: Enable experimental support for duration expressions in Pro...

This release contains 7 PRs from 4 authors. Thank you!

Grafana Mimir version 2.17.0-rc.2 release notes

Grafana Labs is excited to announce version 2.17 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release.
For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

MQE is enabled by default in queriers. MQE provides benefits over the Prometheus engine, inluding reduced memory and CPU consumption and improved performance. To use the Prometheus engine instead of MQE, set -querier.query-engine=prometheus.

Grafana Mimir now supports using the Mimir Query Engine (MQE) in query-frontends in addition to queriers. You can enable MQE in query-frontends by setting the experimental CLI flag -query-frontend.query-engine=mimir or through the corresponding YAML option.

You can export the cortex_ingester_attributed_active_native_histogram_series and cortex_ingester_attributed_active_native_histogram_buckets native histogram cost attribution metrics to a custom Prometheus registry with user-specified labels.

Grafana Mimir supports converting OTel explicit bucket histograms to Prometheus native histograms with custom buckets using the distributor.otel-convert-histograms-to-nhcb flag.

The following experimental features have been removed:

• The max_cost_attribution_labels_per_user cost attribution limit
• Read-write deployment mode in the mixin

Important changes

In Grafana Mimir 2.17, the following behavior has changed:

The following default configuration values now apply to the memberlist KV store:

These values perform better but might cause long-running packets to be dropped in high-latency networks.

The -ruler-storage.cache.rule-group-enabled experimental CLI flag has been removed. Caching rule group contents is now always enabled when a cache is configured for the ruler.

The -ingester.ooo-native-histograms-ingestion-enabled CLI flag and corresponding ooo_native_histograms_ingestion_enabled runtime configuration option have been removed. Out-of-order native histograms are now enabled whenever both native histogram and out-of-order ingestion is enabled.

The -ingester.stream-chunks-when-using-blocks CLI flag and corresponding ingester_stream_chunks_when_using_blocks runtime configuration option have been deprecated and will be removed in a future release.

The cortex_distributor_label_values_with_newlines_total metric has been removed.

In the distributor, memberlist is marked as a stable option for backend storage for the high availability tracker. etcd has been deprecated for this purpose.

Experimental features

Grafana Mimir 2.17 includes some features that are experimental and disabled by default.
Use these features with caution and report any issues that you encounter:

• Prometheus Remote-Write 2.0 protocol.
• Duration expressions in PromQL. These are simple arithmetics on numbers in offset and range specification. For example, rate(http_requests_total[5m * 2]).
• Promoting OTel scope metadata, including name, version, schema URL, and attributes, to metric labels, prefixed with otel_scope_. Enable this feature through the -distributor.otel-promote-scope-metadata flag.
• Allowing primitive delta metrics ingestion through the OTLP endpoint with the -distributor.otel-native-delta-ingestion option.
• Support for sort_by_label and sort_by_label_desc PromQL functions.
• Support for cluster validation in HTTP calls. When enabled, the HTTP server verifies if a request coming from an HTTP client comes from an expected cluster. You can configure this validation with the following options:
  • -server.cluster-validation.label
  • -server.cluster-validation.http.enabled
  • -server.cluster-validation.http.soft-validation
  • -server.cluster-validation.http.exclude-paths

Bug fixes

For a detailed list of bug fixes, refer to the CHANGELOG.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is released independently.
Refer to the Grafana Mimir Helm chart documentation.

Changelog

2.17.0-rc.2

Grafana Mimir

• [ENHANCEMENT] Stagger head compaction intervals across zones to prevent compactions from aligning simultaneously, which could otherwise cause strong consistency queries to fail when experimental ingest storage is enabled. #12090
• [ENHANCEMENT] Compactor: Add -compactor.update-blocks-concurrency flag to control concurrency for updating block metadata during bucket index updates, separate from deletion marker concurrency. #12117
• [ENHANCEMENT] Query-frontend: Allow users to set the query-frontend.extra-propagated-headers flag to specify the extra headers allowed to pass through to the rest of the query path. #12174
• [BUGFIX] Ingester: Fix issue where ingesters can exit read-only mode during idle compactions, resulting in write errors. #12128
• [BUGFIX] otlp: Reverts #11889 which has a pooled memory re-use bug. #12266

All changes in this release: mimir-2.17.0-rc.1...mimir-2.17.0-rc.2

This release contains 81 PRs from 27 authors, including new contributors Edgaras Giedrė, Julius Hinze. Thank you!

Grafana Mimir version 2.17.0-rc.1 release notes

Grafana Labs is excited to announce version 2.17 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release.
For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

MQE is enabled by default in queriers. MQE provides benefits over the Prometheus engine, inluding reduced memory and CPU consumption and improved performance. To use the Prometheus engine instead of MQE, set -querier.query-engine=prometheus.

Grafana Mimir now supports using the Mimir Query Engine (MQE) in query-frontends in addition to queriers. You can enable MQE in query-frontends by setting the experimental CLI flag -query-frontend.query-engine=mimir or through the corresponding YAML option.

You can export the cortex_ingester_attributed_active_native_histogram_series and cortex_ingester_attributed_active_native_histogram_buckets native histogram cost attribution metrics to a custom Prometheus registry with user-specified labels.

Grafana Mimir supports converting OTel explicit bucket histograms to Prometheus native histograms with custom buckets using the distributor.otel-convert-histograms-to-nhcb flag.

The following experimental features have been removed:

• The max_cost_attribution_labels_per_user cost attribution limit
• Read-write deployment mode in the mixin

Important changes

In Grafana Mimir 2.17, the following behavior has changed:

The following default configuration values now apply to the memberlist KV store:

These values perform better but might cause long-running packets to be dropped in high-latency networks.

The -ruler-storage.cache.rule-group-enabled experimental CLI flag has been removed. Caching rule group contents is now always enabled when a cache is configured for the ruler.

The -ingester.ooo-native-histograms-ingestion-enabled CLI flag and corresponding ooo_native_histograms_ingestion_enabled runtime configuration option have been removed. Out-of-order native histograms are now enabled whenever both native histogram and out-of-order ingestion is enabled.

The -ingester.stream-chunks-when-using-blocks CLI flag and corresponding ingester_stream_chunks_when_using_blocks runtime configuration option have been deprecated and will be removed in a future release.

The cortex_distributor_label_values_with_newlines_total metric has been removed.

In the distributor, memberlist is marked as a stable option for backend storage for the high availability tracker. etcd has been deprecated for this purpose.

Experimental features

Grafana Mimir 2.17 includes some features that are experimental and disabled by default.
Use these features with caution and report any issues that you encounter:

• Prometheus Remote-Write 2.0 protocol.
• Duration expressions in PromQL. These are simple arithmetics on numbers in offset and range specification. For example, rate(http_requests_total[5m * 2]).
• Promoting OTel scope metadata, including name, version, schema URL, and attributes, to metric labels, prefixed with otel_scope_. Enable this feature through the -distributor.otel-promote-scope-metadata flag.
• Allowing primitive delta metrics ingestion through the OTLP endpoint with the -distributor.otel-native-delta-ingestion option.
• Support for sort_by_label and sort_by_label_desc PromQL functions.
• Support for cluster validation in HTTP calls. When enabled, the HTTP server verifies if a request coming from an HTTP client comes from an expected cluster. You can configure this validation with the following options:
  • -server.cluster-validation.label
  • -server.cluster-validation.http.enabled
  • -server.cluster-validation.http.soft-validation
  • -server.cluster-validation.http.exclude-paths

Bug fixes

For a detailed list of bug fixes, refer to the CHANGELOG.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is released independently.
Refer to the Grafana Mimir Helm chart documentation.

Changelog

2.17.0-rc.1

Grafana Mimir

• [FEATURE] Distributor: Add experimental -distributor.otel-native-delta-ingestion option to allow primitive delta metrics ingestion via the OTLP endpoint. #11631
• [FEATURE] MQE: Add support for experimental sort_by_label and sort_by_label_desc PromQL functions. #11930
• [FEATURE] Ingester/Block-builder: Handle the created timestamp field for remote-write requests. #11977
• [FEATURE] Cost attribution: Labels specified in the limit configuration may specify an output label in order to override emitted label names. #12035
• [ENHANCEMENT] Ingester: Display user grace interval in the tenant list obtained through the /ingester/tenants endpoint. #11961
• [ENHANCEMENT] kafkatool: add consumer-group delete-offset command as a way to delete the committed offset for a consumer group. #11988
• [ENHANCEMENT] Block-builder-scheduler: Detect gaps in scheduled and completed jobs. #11867
• [ENHANCEMENT] Distributor: Experimental support for Prometheus Remote-Write 2.0 protocol has been updated. Created timestamps are now supported. This feature includes some limitations. If samples in a write request aren't ordered by time, the created timestamp might be dropped. Additionally, per-series metadata is automatically merged on the metric family level. Ingestion might fail if the client sends ProtoBuf fields out-of-order. The label version is added to the metric cortex_distributor_requests_in_total with a value of either 1.0 or 2.0, depending on the detected remote-write protocol. #11977
• [ENHANCEMENT] Query-frontend: Added labels query optimizer that automatically removes redundant __name__!="" matchers from label names and label values queries, improving query performance. You can enable the optimizer per-tenant with the labels_query_optimizer_enabled runtime configuration flag. #12054 #12066 #12076 #12080
• [ENHANCEMENT] Query-frontend: Standardise non-regex patterns in query blocking upon loading of config. #12102
• [ENHANCEMENT] Ruler: Propagate GCS object mutation rate limit for rule group uploads. #12086
• [BUGFIX] Distributor: Validate the RW2 symbols field and reject invalid requests that don't have an empty string as the first symbol. #11953
• [BUGFIX] Distributor: Check max_inflight_push_requests_bytes before decompressing incoming requests. #11967
• [BUGFIX] Query-frontend: Allow limit parameter to be 0 in label queries to explicitly request unlimited results. #12054
• [BUGFIX] Distributor: Fix a possible panic in the OTLP push path while handling a gRPC status error. #12072
• [BUGFIX] Query-frontend: Evaluate experimental duration expressions before sharding, splitting, and caching. Otherwise, the result is not correct. #12038
• [BUGFIX] Block-builder-scheduler: Fix bugs in handling of partitions with no commit. #12130

Mixin

• [CHANGE] Remove support for the experimental read-write deployment mode. #11975
• [CHANGE] Alerts: Replace namespace with job label in golang_alerts. #11957
• [FEATURE] Add an alert if the block-builder-scheduler detects that it has skipped data. #12118

Jsonnet

• [CHANGE] Removed support for the experimental read-write deployment mode. #11974
• [ENHANCEMENT] Add assertion to ensure ingester ScaledObject has minimum and maximum replicas set to a value greater than 0. #11979
• [ENHANCEMENT] Add ingest_storage_migration_ignore_ingest_storage_errors and ingest_storage_migration_ingest_storage_max_wait_time configs to control error handling of the partition ingesters during ingest storage migrations. #12105
• [ENHANCEMENT] Add block-builder job processing duration timings and offset-skipped errors to the Block-builder dashboard. #12118

Documentation

• [ENHANCEMENT] Update the MimirIngestedDataTooFarInTheFuture runbook with a note about false positives and the endpoint to flush TSDB blocks by user. #11961

All changes in this release: mimir-2.17.0-rc.0...mimir-2.17.0-rc.1

This release contains 646 PRs from 75 authors, including new contributors Bernaud Vincent, Carrie Edwards, danieleandreatta, David Vávra, Gabija Bruzgaitė, Henrique Lourenço, Innokentii Konstantinov, Jasper Maes, Jeanette Tan, Joobi S B, Lukas Bischofberger, mihaelmiklec, mimir-vendoring[bot], Nasiel, pierremahot, rektabhi, sam clulow, Shay Pletcher, Thor K. Høgås, Toni Cárdenas, Yuran Ou, zhuoyuan-liu. Thank you!

Grafana Mimir version 2.17.0-rc.0 release notes

Grafana Labs is excited to announce version 2.17 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release.
For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

MQE is enabled by default in queriers. MQE provides benefits over the Prometheus engine, inluding reduced memory and CPU consumption and improved performance. To use the Prometheus engine instead of MQE, set -querier.query-engine=prometheus.

Grafana Mimir now supports using the Mimir Query Engine (MQE) in query-frontends in addition to queriers. You can enable MQE in query-frontends by setting the experimental CLI flag -query-frontend.query-engine=mimir or through the corresponding YAML option.

You can export the cortex_ingester_attributed_active_native_histogram_series and cortex_ingester_attributed_active_native_histogram_buckets native histogram cost attribution metrics to a custom Prometheus registry with user-specified labels.

Grafana Mimir supports converting OTel explicit bucket histograms to Prometheus native histograms with custom buckets using the distributor.otel-convert-histograms-to-nhcb flag.

The following experimental features have been removed:

• The max_cost_attribution_labels_per_user cost attribution limit
• Read-write deployment mode in the mixin

Important changes

In Grafana Mimir 2.17, the following behavior has changed:

The following default configuration values now apply to the memberlist KV store:

These values perform better but might cause long-running packets to be dropped in high-latency networks.

The -ruler-storage.cache.rule-group-enabled experimental CLI flag has been removed. Caching rule group contents is now always enabled when a cache is configured for the ruler.

The -ingester.ooo-native-histograms-ingestion-enabled CLI flag and corresponding ooo_native_histograms_ingestion_enabled runtime configuration option have been removed. Out-of-order native histograms are now enabled whenever both native histogram and out-of-order ingestion is enabled.

The -ingester.stream-chunks-when-using-blocks CLI flag and corresponding ingester_stream_chunks_when_using_blocks runtime configuration option have been deprecated and will be removed in a future release.

The cortex_distributor_label_values_with_newlines_total metric has been removed.

In the distributor, memberlist is marked as a stable option for backend storage for the high availability tracker. etcd has been deprecated for this purpose.

Experimental features

Grafana Mimir 2.17 includes some features that are experimental and disabled by default.
Use these features with caution and report any issues that you encounter:

• Prometheus Remote-Write 2.0 protocol.
• Duration expressions in PromQL. These are simple arithmetics on numbers in offset and range specification. For example, rate(http_requests_total[5m * 2]).
• Promoting OTel scope metadata, including name, version, schema URL, and attributes, to metric labels, prefixed with otel_scope_. Enable this feature through the -distributor.otel-promote-scope-metadata flag.
• Allowing primitive delta metrics ingestion through the OTLP endpoint with the -distributor.otel-native-delta-ingestion option.
• Support for sort_by_label and sort_by_label_desc PromQL functions.
• Support for cluster validation in HTTP calls. When enabled, the HTTP server verifies if a request coming from an HTTP client comes from an expected cluster. You can configure this validation with the following options:
  • -server.cluster-validation.label
  • -server.cluster-validation.http.enabled
  • -server.cluster-validation.http.soft-validation
  • -server.cluster-validation.http.exclude-paths

Bug fixes

For a detailed list of bug fixes, refer to the CHANGELOG.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is released independently.
Refer to the Grafana Mimir Helm chart documentation.

Changelog

2.17.0-rc.0

Grafana Mimir

• [CHANGE] Query-frontend: Ensure that cache keys generated from cardinality estimate middleware are less than 250 bytes in length by hashing the tenant IDs that are included in them. This change invalidates all cardinality estimates in the cache. #11568
• [CHANGE] Ruler: Remove experimental CLI flag -ruler-storage.cache.rule-group-enabled to enable or disable caching the contents of rule groups. Caching rule group contents is now always enabled when a cache is configured for the ruler. #10949
• [CHANGE] Ingester: Out-of-order native histograms are now enabled whenever both native histogram and out-of-order ingestion is enabled. The -ingester.ooo-native-histograms-ingestion-enabled CLI flag and corresponding ooo_native_histograms_ingestion_enabled runtime configuration option have been removed. #10956
• [CHANGE] Distributor: removed the cortex_distributor_label_values_with_newlines_total metric. #10977
• [CHANGE] Ingester/Distributor: renamed the experimental max_cost_attribution_cardinality_per_user config to max_cost_attribution_cardinality. #11092
• [CHANGE] Frontend: The subquery spin-off feature is now enabled with -query-frontend.subquery-spin-off-enabled=true instead of -query-frontend.instant-queries-with-subquery-spin-off=.* #11153
• [CHANGE] Overrides-exporter: Don't export per-tenant overrides that are set to their default values. #11173
• [CHANGE] gRPC/HTTP clients: Rename metric cortex_client_request_invalid_cluster_validation_labels_total to cortex_client_invalid_cluster_validation_label_requests_total. #11237
• [CHANGE] Querier: Use Mimir Query Engine (MQE) by default. Set -querier.query-engine=prometheus to continue using Prometheus' engine. #11501
• [CHANGE] Memcached: Ignore initial DNS resolution failure, meaning don't depend on Memcached on startup. #11602
• [CHANGE] Ingester: The -ingester.stream-chunks-when-using-blocks CLI flag and ingester_stream_chunks_when_using_blocks runtime configuration option have been deprecated and will be removed in a future release. #11711
• [CHANGE] Distributor: track cortex_ingest_storage_writer_latency_seconds metric for failed writes too. Added outcome label to distinguish between success and failure. #11770
• [CHANGE] Distributor: renamed few metrics used by experimental ingest storage. #11766
  • Renamed cortex_ingest_storage_writer_produce_requests_total to cortex_ingest_storage_writer_produce_records_enqueued_total
  • Renamed cortex_ingest_storage_writer_produce_failures_total to cortex_ingest_storage_writer_produce_records_failed_total
• [CHANGE] Distributor: moved HA tracker timeout config to limits. #11774
  • Moved distributor.ha_tracker.ha_tracker_update_timeout to limits.ha_tracker_update_timeout.
  • Moved distributor.ha_tracker.ha_tracker_update_timeout_jitter_max to limits.ha_tracker_update_timeout_jitter_max.
  • Moved distributor.ha_tracker.ha_tracker_failover_timeout to limits.ha_tracker_failover_timeout.
• [CHANGE] Distributor: Memberlist marked as stable as an option for backend storage for the HA tracker. #11861
• [CHANGE] Distributor: etcd deprecated as an option for backend storage for the HA tracker. #12047
• [CHANGE] Memberlist: Apply new default configuration values for MemberlistKV. This unlocks using it as backend storage for the HA Tracker. We have observed better performance with these defaults across different production loads. #11874
  • memberlist.packet-dial-timeout: 500ms
  • memberlist.packet-write-timeout: 500ms
  • memberlist.max-concurrent-writes: 5
  • memberlist.acquire-writer-timeout: 1s
    These defaults perform better but may cause long-running packets to be dropped in high-latency networks.
• [CHANGE] Query-frontend: Apply query pruning and check for disabled experimental functions earlier in query processing. #11939
• [FEATURE] Distributor: Experimental support for Prometheus Remote-Write 2.0 protocol. Limitations: Created timestamp is ignored, per series metadata is merged on metric family level automatically, ingestion might fail if client sends ProtoBuf fields out of order. The label version is added to the metric cortex_distributor_requests_in_total with a value of either 1.0 or 2.0 depending on the detected Remote-Write protocol. #11100 #11101 #11192 #11143
• [FEATURE] Query-frontend: expand query-frontend.cache-errors and query-frontend.results-cache-ttl-for-errors configuration options to cache non-transient response failures for instant queries. #11120
• [FEATURE] Query-frontend: Allow use of Mimir Query Engine (MQE) via the experimental CLI flags -query-frontend.query-engine or -query-frontend.enable-query-engine-fallback or corresponding YAML. #11417 #11775
• [FEATURE] Querier, query-frontend, ruler: Enable experimental support for duration expressions in PromQ...

This release contains 14 PRs from 8 authors. Thank you!

Changelog

2.16.1

Grafana Mimir

• [BUGFIX] Update to Go v1.23.9 to address CVE-2025-22871. #11543
• [BUGFIX] Update golang.org/x/net to v0.38.0 to address CVE-2025-22872. #11281
• [BUGFIX] Query-frontend: Fix a panic in monolithic mode caused by a clash in labels of the cortex_client_invalid_cluster_validation_label_requests_total metric definition. #11455

All changes in this release: mimir-2.16.0...mimir-2.16.1

This release contains 10 PRs from 5 authors, including new contributors mimir-github-bot[bot]. Thank you!

Changelog

2.15.3

Grafana Mimir

• [BUGFIX] Update to Go v1.23.9 to address CVE-2025-22871. #11537

Mimirtool

• [BUGFIX] Upgrade Alpine Linux to 3.20.6, fixes CVE-2025-26519. #11530

Mimir Continuous Test

• [BUGFIX] Upgrade Alpine Linux to 3.20.6, fixes CVE-2025-26519. #11530

All changes in this release: mimir-2.15.2...mimir-2.15.3

This release contains 4 PRs from 3 authors. Thank you!

Changelog

2.15.2

Grafana Mimir

• [BUGFIX] Update module golang.org/x/net to v0.36.0 to address CVE-2025-22870. #10875
• [BUGFIX] Update module github.com/golang-jwt/jwt/v5 to v5.2.2 to address CVE-2025-30204. #11045

All changes in this release: mimir-2.15.1...mimir-2.15.2

This release contains 463 PRs from 69 authors, including new contributors Alessandro Verzicco, Alex Greenbank, André Pires, Bjorn Stout, Bruno FERNANDO, Casie Chen, Dustin Wilson, Edwin Tye, Kenny Trytek, Leszek Błażewski, Markus Opolka, Matthew Jacobson, Matt Veitas, mimir-github-bot[bot], Moustafa Baiou, Ryan Brady, TheRealNoob. Thank you!

Grafana Mimir version 2.16.0 release notes

Features and enhancements

In rulers, when rule concurrency is enabled for a rule group, its rules will now be reordered and run in batches based on their dependencies. This increases the number of rules that can potentially run concurrently. Note that the global and tenant-specific limits around the number of rule groups and rules per group still apply.

Using mimirtool to analyze Grafana dashboards now supports bar chart, pie chart, state timeline, status history, histogram, candlestick, canvas, flame graph, geomap, node graph, trend, and XY chart panels.

Important changes

In Grafana Mimir 2.16, the following behavior has changed:

Grafana Mimir only provides container images based on distroless images. Alpine Linux-based container images were deprecated in the 2.12 release and are no longer built.

How experimental PromQL functions are enabled has changed.

• The experimental CLI flags -querier.promql-experimental-function B60C s-enabled and -query-frontend.block-promql-experimental-functions and respective YAML configuration have been removed from query-frontends and queriers.
• Experimental PromQL functions are disabled by default but can be enabled using only the per-tenant setting enabled_promql_experimental_functions.

Support for native histograms and out-of-order native histograms is enabled by default in ingesters.

Distributors discard float and histogram samples with duplicated timestamps from each timeseries in a request before the request is forwarded to ingesters. Discarded samples are tracked by cortex_discarded_samples_total metrics with the reason sample_duplicate_timestamp.

Experimental features

Grafana Mimir 2.16 includes some features that are experimental and disabled by default. Use these features with caution and report any issues that you encounter:

Distributors now include experimental support for the Influx line protocol.

Query-frontends now include experimental support to "spin off" subqueries as actual range queries, so that they benefit from query acceleration techniques such as sharding, splitting, and caching.

Bug fixes

• Distributor: Use a boolean to track changes while merging the ReplicaDesc components, rather than comparing the objects directly. #10185
• Querier: Fix timeout responding to the query-frontend when the response size is within a few hundred bytes of -querier.frontend-client.grpc-max-send-msg-size. #10154
• Query-frontend and querier: Show warning and info annotations in some cases where they were missing (if a lazy querier was used). #10277
• Query-frontend: Fix an issue where transient errors are inadvertently cached. #10537 #10631
• Ruler: Fix indeterminate rules always, instead of never, running concurrently when -ruler.max-independent-rule-evaluation-concurrency is set. prometheus/prometheus#15560 #10258
• PromQL: Fix various UTF-8 bugs related to quoting. prometheus/prometheus#15531 #10258
• Ruler: Fix an issue when using the experimental -ruler.max-independent-rule-evaluation-concurrency feature, where if a rule group was eligible for concurrency, it would flap between running concurrently or not based on the time it took after running concurrently. #9726 #10189
• Mimirtool: remote-read commands now return data. #10286
• PromQL: Fix deriv, predict_linear and double_exponential_smoothing with histograms prometheus/prometheus#15686 #10383
• MQE: Fix deriv with histograms. #10383
• PromQL: Fix <aggr_over_time> functions with histograms. prometheus/prometheus#15711 #10400
• MQE: Fix <aggr_over_time> functions with histograms. #10400
• Distributor: Return HTTP status 415, Unsupported Media Type, instead of 200, Success, for Remote Write 2.0 until we support it. #10423 #10916
• Query-frontend: Add -query-frontend.prom2-range-compat flag and corresponding YAML to rewrite queries with ranges that worked in Prometheus 2 but are invalid in Prometheus 3. #10445 #10461 #10502
• Distributor: Fix edge case at the HA-tracker with memberlist as KVStore, where when a replica in the KVStore is marked as deleted but not yet removed, it fails to update the KVStore. #10443
• Distributor: Fix panics in DurationWithJitter util functions when computed variance is zero. #10507
• Ingester: Fixed a race condition in the PostingsForMatchers cache that may have infrequently returned expired cached postings. #10500
• Distributor: Report partially converted OTLP requests with status 400, Bad Request. #10588
• Ruler: Fix issue where rule evaluations could be missed while shutting down a ruler instance if that instance owns many rule groups. prometheus/prometheus#15804 #10762
• Ingester: Add additional check on reactive limiter queue sizes. #10722
• TSDB: Fix unknown series errors and possible lost data during WAL replay when series are removed from the head due to inactivity and reappear before the next WAL checkpoint. prometheus/prometheus#16060 #10824
• Querier: Fix issue where label_join could incorrectly return multiple series with the same labels rather than failing with vector cannot contain metrics with the same labelset. prometheus/prometheus#15975 #10826
• Querier: Fix issue where counter resets on native histograms could be incorrectly under or over-counted when using subqueries. prometheus/prometheus#15987 #10871
• Ingester: Fix goroutines and memory leak when experimental ingest storage is enabled and a server-side error occurs during metrics ingestion. #10915
• Mimirtool: Fix issue where MIMIR_HTTP_PREFIX environment variable was ignored and the value from MIMIR_MIMIR_HTTP_PREFIX was used instead. #10207

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is released independently.
Refer to the Grafana Mimir Helm chart documentation.

Changelog

2.16.0

Grafana Mimir

• [CHANGE] Querier: pass context to queryable IsApplicable hook. #10451
• [CHANGE] Distributor: OTLP and push handler replace all non-UTF8 characters with the unicode replacement character \uFFFD in error messages before propagating them. #10236
• [CHANGE] Querier: pass query matchers to queryable IsApplicable hook. #10256
• [CHANGE] Build: removed Mimir Alpine Docker image and related CI tests. #10469
• [CHANGE] Query-frontend: Add topic label to cortex_ingest_storage_strong_consistency_requests_total, cortex_ingest_storage_strong_consistency_failures_total, and cortex_ingest_storage_strong_consistency_wait_duration_seconds metrics. #10220
• [CHANGE] Ruler: cap the rate of retries for remote query evaluation to 170/sec. This is configurable via -ruler.query-frontend.max-retries-rate. #10375 #10403
• [CHANGE] Query-frontend: Add topic label to cortex_ingest_storage_reader_last_produced_offset_requests_total, cortex_ingest_storage_reader_last_produced_offset_failures_total, cortex_ingest_storage_reader_last_produced_offset_request_duration_seconds, cortex_ingest_storage_reader_partition_start_offset_requests_total, cortex_ingest_storage_reader_partition_start_offset_failures_total, cortex_ingest_storage_reader_partition_start_offset_request_duration_seconds metrics. #10462
• [CHANGE] Ingester: Set -ingester.ooo-native-histograms-ingestion-enabled to true by default. #10483
• [CHANGE] Ruler: Add user and reason labels to cortex_ruler_write_requests_failed_total and cortex_ruler_queries_failed_total; add user to
  cortex_ruler_write_requests_total and cortex_ruler_queries_total metrics. #10536
• [CHANGE] Querier / Query-frontend: Remove experimental -querier.promql-experimental-functions-enabled and -query-frontend.block-promql-experimental-functions CLI flags and respective YAML configuration options to enable experimental PromQL functions. Instead access to experimental PromQL functions is always blocked. You can enable them using the per-tenant setting enabled_promql_experimental_functions. #10660 #10712
• [CHANGE] Store-gateway: Include posting sampling rate in sparse index headers. When the sampling rate isn't set in a sparse index header, store gateway rebuilds the sparse header with the configured blocks-storage.bucket-store.posting-offsets-in-mem-sampling value. If the sparse header's sampling rate is set but doesn't match the configured rate, store gateway either rebuilds the sparse header or downsamples to the configured sampling rate. #10684 #10878
• [CHANGE] Distributor: Return specific error message when burst size limit is exceeded. #10835
• [CHANGE] Ingester: enable native histograms ingestion by default, meaningingester.native-histograms-ingestion-enabled defaults to true. #10867
• [FEATURE] Ingester/Distributor: Add support for exporting cost attribution metrics (cortex_ingester_attributed_active_series, cortex_distributor_received_attributed_samples_total, and cortex_discarded_attributed_samples_total) with labels specified by customers to a custom Prometheus registry. This feature enables more flexible billing data tracking. #10269 #10702
• [FEATURE] Ruler: Added /ruler/tenants endpoints to list the discovered tenants with rule groups. #10738
• [FEATURE] Distributor: Add experimental Influx handler. #10153
• [ENHANCEMENT]...