OpenTelemetry header propagation to Tyk Pump

Hi!

I’m currently trying to configure OpenTelemetry in Tyk. And so far, it works smoothly in the API Gateway. However, I ran into an issue with Tyk Pump. My goal is to receive traceparent header in the Tyk Pump tags field. Unfortunately, I can’t get it to work. I see the traces from Tyk, and I see that the traceparent header is coming to my service from Tyk Gateway, but I don’t see this header in Tyk Pump records.

I tried adding the header to the API definition tag_headers field but it didn’t help:

tag_headers:
    - traceparent

I would appreciate any help with this issue. Thanks in advance!

@olena Hello and welcome to the community :tada:

This is great to know. If you have any issues with integrating Otel, please drop a message on this topic or you can create a new one as you have done.

I see that you’ve added the tag headers in the API definition. However, did you send a request with the header name and value as shown in our docs?

How did you observe this? Also can you share your gateway version?

Hi @Olu, thank you!

I used this guide. So, I don’t send the traceparent header myself, but Tyk Gateway adds it to the request before forwarding it to my service. In other words, the trace is started in Tyk. That’s probably why it’s not “tagged”. Is it possible to configure it?

I’m using gateway version v5.2.2. To observe this, I logged incoming headers in my service (it’s behind the Tyk Gateway). As I mentioned it receives and correctly uses the traceparent header. I also checked the Tyk Pump backend and the records didn’t have the header, though it had other tags. Additionally, I can see the traces coming from Tyk in my tracing backend.

So, the OpenTelemetry works in Tyk Gateway, but I was wondering if it’s possible to integrate it with Tyk Pump. Hope it helps and thanks again :smiling_face:

I guess so. The transform headers should do the trick.

Can you share the API definition for review? You can strip any sensitive info

Ha! This was the part I needed some clarification on. From my understanding, they are different services. Pump retrieves the analytics from Redis. The analytics record is generated and sent to Redis by the Gateway. However, Otel retrieves the traces directly from the Gateway or however, you’ve setup the input (either from zipkin or jaeger).

We don’t have an Otel-pump. But since otel has receivers like prometheus and kafka, you could simply configure their corresponding pumps and use as input in Otel.

Hope this helps.

Thank you for your response, @Olu.

Could you please share an example? I’m not sure how I can reference the traceparent value generated in Tyk Gateway.

My API definition is similar to the following:

  apps:
    my-api:
      name: "My API"
      api_id: my-api
      active: true
      version_data:
        default_version: v1
        not_versioned: false
        versions:
          v1:
            name: v1
            use_extended_paths: true
            extended_paths:
              white_list:
                - path: "/my-list"
                  method_actions:
                    GET:
                      action: no_action
      proxy:
        listen_path: "/my/api/"
        strip_listen_path: true
        target_url: "https://my-api.com"
      tag_headers:
        - traceparent
      enable_context_vars: true

I think I wasn’t clear enough. We have a couple of pumps configured and I don’t think there is an issue with them. By integration, I meant propagating the traceparent header to the tags field in the pump records.

Let me share an example for the API definition I shared above.

  1. I send a request curl -X GET https://my-tyk.com/my/api/list with no headers.
  2. Tyk Gateway accepts the request, adds the traceparent header, and forwards the call to my service. Now the request looks like this curl -X GET https://my-api.com/list -H "traceparent: 01-XXX-123"
  3. Then the response is returned and probably at this point Tyk Gateway creates an analytic record and sends it to Redis. The record will look similar to the following:
{
  "api_id": "my-api",
  "api_key": "123",
  "api_name": "My API",
  "api_version": "Non Versioned",
  "content_length": 0,
  "host": "my-api.com",
  "ip_address": "11.11.11.11",
  "method": "GET",
  "oauth_id": "",
  "org_id": "default",
  "path": "/list",
  "raw_path": "/list",
  "raw_request": "",
  "raw_response": "",
  "request_time_ms": 123,
  "response_code": 200,
  "tags": [
    "pol-my-policy",
    "key-123",
    "org-default",
    "api-my-api"
  ],
  "timestamp": "2024-04-02T10:10:10.03264713Z"
}

And the result I’m trying to achieve is that the tags field looks like this instead:

"tags": [
    "pol-my-policy",
    "key-123",
    "org-default",
    "api-my-api",
    "traceparent-01-XXX-123"
  ],

Does it make sense? Hope it’s more clear this way.

We have examples of middleware configs in our operator repo. You can use either the global_headers or the transform_headers to achieve what you want.

How is the header traceparent: 01-XXX-123 added? I don’t see a middleware that adds this from Tyk side.

Are you saying you get this at your upstream?

I think so. Could you ensure the header isn’t added during the proxy and add it instead in the request to confirm what happens

I would have to test this on my end as well

It is added by the OpenTelemetry configuration from your guide. It says:

This setting allows you to specify the type of context propagator to use for trace data. It’s essential for ensuring compatibility and data integrity between different services in your architecture. The available options are:

If you follow the link to w3c specification, it mentions that tracing is propagated using traceparent header. Here is Tyk Gateway source code that adds it to the request headers. So, I didn’t add any custom middleware, I just enabled OpenTelemetry according to your tutorial.

Since this header is injected into the request headers in Tyk Gateway, I can’t find a way to access it. I tried using $tyk_context and $tyk_meta, but neither has this header. That’s why I would appreciate an example.

As I mentioned above, I don’t get it from the caller that sends a request to Tyk. Tyk Gateway adds it in the newly added feature - OpenTelemetry support.

I’m sure it comes from Tyk and is not generated elsewhere.

Thank you! Looking forward to the results.

Sorry for the delay in response. I had a very busy week.

I believe I now understand after testing. I am not sure if that’s exactly where the traceparent is generated from the source you shared since I haven’t followed or stepped through but it looks like it’s not added before analytics generation takes place. Or the analytics just completely ignores it.

I tried hacking a workaround with URL rewrite and looping but none seemed to give. I think only forwarding the request to another gateway is what would work (overkill) but I haven’t tested this.

I don’t even know how it’s added so none of those should work.

I will ask internally about the flow and get back to you.

Hi Olu,

I was wondering if you have any updates on the issue. Perhaps there is a plan to investigate or fix it?

Thank you in advance.

Sorry for the lack of updates. I have opened a bug request for this internally. The reference id is TT-11977.

I can see it’s been ranked slightly above medium but there has been no update for about 3 weeks now.

1 Like

Hi Olu!

I hope you’re having an amazing summer.

I was wondering if there are any updates on the issue.

Hi Olena,

I hope you are enjoying your summer as well!

Thank you for reaching out and following up on your issue.

Unfortunately, the issue hasn’t been picked up yet. We are currently addressing priority 1 issues at the moment and the issue has been scoped to priority 2.

Rest assured that we will update you as soon as we have more information or a resolution. We apologize for any inconvenience this delay may have caused.

1 Like