Deprecate /metrics & Set /metrics/otel As Primary Endpoint

Alright guys, let's dive into a proposal to streamline our metrics endpoints. Currently, the Telemetry plugin registers both /metrics and /metrics/otel when FEATURE_OTEL=true. However, our documentation actually recommends scraping /metrics/otel to avoid any conflicts with the core-backend metrics formats. So, what's the plan? We're proposing to make /metrics/otel the primary endpoint in the plugin, which will help simplify things and ensure consistency.

Context: Why the Change?

Let’s break down the current situation. When FEATURE_OTEL is enabled, our Telemetry plugin diligently registers two endpoints: /metrics and /metrics/otel. Now, you might be wondering, “Why two?” Well, historically, we've needed to ensure compatibility and avoid disruptions. However, our documentation has evolved, and we now advise users to scrape /metrics/otel specifically. This recommendation stems from the need to sidestep potential clashes with core-backend metrics formats. These formats can sometimes overlap, leading to confusion and inaccurate data. By focusing on /metrics/otel, we create a clear separation, ensuring that the metrics we collect are precise and reliable. This move isn't just about tidiness; it's about ensuring the integrity of our telemetry data. When we have a single, recommended endpoint, it reduces the chances of misconfiguration and makes it easier for our users to get the insights they need. Think of it as streamlining a busy intersection to prevent traffic jams – it makes everything flow more smoothly and efficiently.

Proposal: Streamlining Metrics Endpoints

So, what's the nitty-gritty of this proposal? The core idea is to make /metrics/otel the star of the show. We want it to be the go-to endpoint for all things metrics in the plugin. But don't worry, we're not going to leave anyone in the lurch. To ensure a smooth transition, we'll keep /metrics around as a compatibility alias for one cycle. Think of it as a grace period, giving everyone time to adjust. We'll even include a config flag that allows you to disable the alias early if needed. This gives us flexibility and control over the transition process. As part of this change, we'll also be updating our dashboards and documentation to exclusively use /metrics/otel. This will help to reinforce the new standard and prevent confusion down the line. The goal here is to create a clear, unified approach to metrics collection. By focusing on a single primary endpoint, we simplify the configuration process, reduce the potential for errors, and make it easier for users to leverage the full power of our telemetry plugin. This isn’t just a technical change; it’s a step towards a more user-friendly and efficient system.

Making /metrics/otel the Primary Endpoint: A Phased Approach

To ensure a smooth transition, we're implementing a phased approach to making /metrics/otel the primary endpoint. This approach minimizes disruption and gives everyone time to adapt. First, we'll designate /metrics/otel as the default and recommended endpoint. This means that all new configurations and documentation will point to /metrics/otel. Next, we'll maintain /metrics as a compatibility alias for a defined period, essentially a grace period. During this time, both endpoints will function, but we'll actively encourage users to switch to /metrics/otel. We'll also provide a configuration flag that allows users to disable the /metrics alias early if they choose. This is particularly useful for those who want to ensure strict adherence to the new standard. Finally, after the grace period, we'll officially deprecate /metrics, removing it as a supported endpoint. This phased approach ensures that users have ample time to adjust, update their configurations, and avoid any unexpected disruptions. It’s all about making the transition as seamless and painless as possible, while still moving towards a more streamlined and efficient system.

Acceptance Criteria: Ensuring a Smooth Transition

To make sure this transition goes off without a hitch, we've established some clear acceptance criteria. These criteria cover everything from the code itself to the documentation and testing. On the code side, the plugin should only register /metrics/otel by default. The optional alias for /metrics will be controlled by a flag, specifically FEATURE_OTEL_METRICS_ALIAS=true. This gives us a clean separation and allows for easy control. Documentation is another critical piece. We'll be updating the README and any relevant guides to reflect the change. This ensures that everyone has the information they need to use the new primary endpoint. Finally, we'll be adjusting our local smoke tests and CI tests to validate the new configuration. This will help us catch any issues early on and ensure that the change is working as expected. These acceptance criteria provide a clear roadmap for the transition, ensuring that we've covered all the bases and that the new system is robust and reliable. It’s about more than just making a change; it’s about making sure that change is implemented correctly and effectively.

Code Modifications for Primary Endpoint Designation

The code modifications required to designate /metrics/otel as the primary endpoint involve several key steps. First and foremost, the plugin's default behavior must be altered to register only /metrics/otel. This means adjusting the codebase to ensure that this endpoint is the standard, go-to option. Next, we'll implement the optional alias for /metrics, controlled by the FEATURE_OTEL_METRICS_ALIAS=true flag. This involves adding conditional logic that registers /metrics only when this flag is enabled. This approach provides flexibility, allowing users to maintain the old behavior temporarily if needed, while encouraging the adoption of the new primary endpoint. Furthermore, any existing code that references /metrics will need to be reviewed and updated to use /metrics/otel. This includes internal plugin logic, configuration settings, and any external integrations. Thorough testing will be essential to ensure that these changes are implemented correctly and that no functionality is broken. This meticulous approach to code modification is crucial for a smooth transition, minimizing the risk of bugs and ensuring that the new primary endpoint functions flawlessly. It’s about making sure the foundation is solid, so everything built on top of it can thrive.

Notes: Key Considerations

There are a few important notes to keep in mind. First, there will be no impact while FEATURE_OTEL=false. This is the default setting, so most users won't see any change right away. This is great news because it means we're not forcing a change on anyone who isn't ready for it. Second, we're planning to schedule this change after a staging observation window. This gives us time to monitor the impact of the change in a controlled environment before rolling it out more widely. It’s like a dress rehearsal before the main performance, allowing us to catch any potential issues and address them before they affect a large audience. This cautious approach minimizes risk and ensures a smooth transition for everyone. It demonstrates our commitment to stability and reliability, ensuring that any changes we make are thoroughly vetted and carefully implemented. We want to make sure that this transition is as seamless and painless as possible, and these precautions are a key part of that effort.

Impact Assessment and Mitigation Strategies

Before implementing any significant change, it's crucial to conduct a thorough impact assessment. In this case, we need to consider the potential effects of deprecating /metrics and making /metrics/otel the primary endpoint. One key area to assess is the existing user base. How many users are currently scraping /metrics? What configurations will need to be updated? We'll need to provide clear communication and guidance to these users, ensuring they understand the changes and how to adapt. Another area to consider is any external integrations that rely on /metrics. These integrations will need to be updated to use /metrics/otel, and we'll need to work with the relevant teams to ensure a smooth transition. To mitigate potential issues, we'll implement several strategies. As mentioned earlier, we'll maintain /metrics as a compatibility alias for a period, providing a grace period for users to adapt. We'll also provide detailed documentation and support resources, answering common questions and troubleshooting any issues that arise. Additionally, we'll closely monitor the system after the change is implemented, looking for any unexpected behavior or performance degradation. This proactive approach to impact assessment and mitigation ensures that we're prepared for any potential challenges and that we can minimize disruption to our users. It’s about being responsible stewards of the system, anticipating potential problems and taking steps to address them proactively.

By making /metrics/otel the primary endpoint and carefully managing the transition, we're setting ourselves up for a more streamlined and efficient metrics collection process. Let's make it happen!