Pump Prometheus Latency Bucket only Has Type Total

Hi Tyk team,

We’re using tyk pump to pump monitoring data to Prometheus. However I found your instruction docs are kinda outdated.

I tried to extract upstream latency from Prometheus. There’s a doc from Tyk Monitor your Apis with Prometheus. And the sample query is histogram_quantile(0.90, sum(rate(tyk_http_latency_bucket{api_name="<api name>"}[1m])) by (le,api_name)). However I believe tyk_http_latency_bucket is changed to tyk_latency_bucket, and api_name is api now. And when I tried to group the data by type, I found there’s only total latency I can choose (there’s no upstream option I can choose from)

How to get upstream latency from Prometheus then? Is that a bug or you decided to only use total type? Thank you!

Hi @byu,

Do you mean that the docs shared and the sample in GitHub do not match?

Or that the docs shared and Prometheus docs on querying does not match?

@sonja or @tombuchaillot Could you help here?

hi @byu ,

you are right, those metrics are not exposed by default. You need to update your Tyk Pump configuration to expose them, see the example at the top of the doc page: Monitor your Apis with Prometheus.

Look for:

"custom_metrics":[
     {
         "name":"tyk_http_requests_total",
         "description":"Total of API requests",
         "metric_type":"counter",
         "labels":["response_code","api_name","method","api_key","alias","path"]
     },
     {
         "name":"tyk_http_latency",
         "description":"Latency of API requests",
         "metric_type":"histogram",
         "labels":["type","response_code","api_name","method","api_key","alias","path"]
     }

Exposing custom Prometheus metrics is a functionality is available since Tyk Pump 1.6, so you also might want to check which version you are using.

The only time that can be exposed at the moment is the upstream time. We do not expose the time spent in the Gateway at the moment (though that is something we are considering for the future). I agree that this is confusing when talking about total, what we mean here is the sum of the upstream time.

I hope this helps, let me know if you have any questions!

Also you might want to look at the demo project on GitHub to see it running. Your feedback about Tyk & Prometheus would be also very appreciated on this thread: SLIs and SLOs with Prometheus and Grafana for your APIs managed by Tyk

Sonja

Hey @Olu @sonja ! Thank you so much for helping out! oh i didn’t know that. i just enabled the pump and wait for it to gather all the data for me lol. Let me try that and I’ll let you know!

Hey @sonja ! Just updated our pump version and tested it. I tried to add tags in the label field. However, pump log told me like:

panic: inconsistent label cardinality: expected 3 label values but got 2 in []string{"200", "/"}

I configured the pump like:

    "prometheus": {
      "type": "prometheus",
      "meta": {
        "listen_address": ":9595",
        "path": "/metrics",
        "custom_metrics": [
          {
            "name": "tyk_http_tags_total",
            "description": "Total of API requests with tags",
            "metric_type": "counter",
            "labels": [
              "response_code",
              "tags",
              "path"
            ]
          }
        ]
      }
    }

I tried to print out all the data using stdout pump and I can find tags there. Do you know why custom_metric can’t get tags data? Many thanks!

I think the issue here is the tags label. I am sure if you remove that, it would work since the 200 and the / are for the response_code and path.

I assume you are trying to get the tags associated with the API definition.

@sonja Is this possible? I am not too familiar with Prometheus but checking pump codebase I don’t see where we map tags

1 Like

Hi @Olu ! Thank you for coming back to me!

It makes sense to me, that the source code doesn’t have tags in the GetLabelsValues function. If Tyk Pump can’t provide the data we need, I can edit the code base and maintain a local image of our own. Or if you don’t have the plan to add tags for custom metrics yet, I can open a Github PR to do it

@olu is right! to confirm: labels are not available yet for the custom prometheus metrics. You will find the list of the field that are supported here: GitHub - TykTechnologies/tyk-pump: Tyk Analytics Pump to move analytics data from Redis to any supported back end (multiple back ends can be written to at once)..

The labels configuration determines the label name and value extracted from the analytic record. The available values are: ["host","method", "path", "response_code", "api_key", "time_stamp", "api_version", "api_name", "api_id", "org_id", "oauth_id", "request_time", "ip_address", "alias"]

I’m curious @byu: how do you use tags in Tyk?