Pump Prometheus Latency Bucket only Has Type Total

Support Operation and Testing
, ,
1

Hi Tyk team,

We’re using tyk pump to pump monitoring data to Prometheus. However I found your instruction docs are kinda outdated.

I tried to extract upstream latency from Prometheus. There’s a doc from Tyk Monitor your APIs with Prometheus. And the sample query is histogram_quantile(0.90, sum(rate(tyk_http_latency_bucket{api_name="<api name>"}[1m])) by (le,api_name)). However I believe tyk_http_latency_bucket is changed to tyk_latency_bucket, and api_name is api now. And when I tried to group the data by type, I found there’s only total latency I can choose (there’s no upstream option I can choose from)

How to get upstream latency from Prometheus then? Is that a bug or you decided to only use total type? Thank you!

2

Hi @byu,

Do you mean that the docs shared and the sample in GitHub do not match?

Or that the docs shared and Prometheus docs on querying does not match?

3

@sonja or @tombuchaillot Could you help here?

4

hi @byu ,

you are right, those metrics are not exposed by default. You need to update your Tyk Pump configuration to expose them, see the example at the top of the doc page: Monitor your APIs with Prometheus.

Look for:

"custom_metrics":[
     {
         "name":"tyk_http_requests_total",
         "description":"Total of API requests",
         "metric_type":"counter",
         "labels":["response_code","api_name","method","api_key","alias","path"]
     },
     {
         "name":"tyk_http_latency",
         "description":"Latency of API requests",
         "metric_type":"histogram",
         "labels":["type","response_code","api_name","method","api_key","alias","path"]
     }

Exposing custom Prometheus metrics is a functionality is available since Tyk Pump 1.6, so you also might want to check which version you are using.

The only time that can be exposed at the moment is the upstream time. We do not expose the time spent in the Gateway at the moment (though that is something we are considering for the future). I agree that this is confusing when talking about total, what we mean here is the sum of the upstream time.

I hope this helps, let me know if you have any questions!

Also you might want to look at the demo project on GitHub to see it running. Your feedback about Tyk & Prometheus would be also very appreciated on this thread: SLIs and SLOs with Prometheus and Grafana for your APIs managed by Tyk

Sonja

5

Hey @Olu @sonja ! Thank you so much for helping out! oh i didn’t know that. i just enabled the pump and wait for it to gather all the data for me lol. Let me try that and I’ll let you know!

6

Hey @sonja ! Just updated our pump version and tested it. I tried to add tags in the label field. However, pump log told me like:

panic: inconsistent label cardinality: expected 3 label values but got 2 in []string{"200", "/"}

I configured the pump like:

    "prometheus": {
      "type": "prometheus",
      "meta": {
        "listen_address": ":9595",
        "path": "/metrics",
        "custom_metrics": [
          {
            "name": "tyk_http_tags_total",
            "description": "Total of API requests with tags",
            "metric_type": "counter",
            "labels": [
              "response_code",
              "tags",
              "path"
            ]
          }
        ]
      }
    }

I tried to print out all the data using stdout pump and I can find tags there. Do you know why custom_metric can’t get tags data? Many thanks!

7

I think the issue here is the tags label. I am sure if you remove that, it would work since the 200 and the / are for the response_code and path.

I assume you are trying to get the tags associated with the API definition.

@sonja Is this possible? I am not too familiar with Prometheus but checking pump codebase I don’t see where we map tags

1 Like
8

Hi @Olu ! Thank you for coming back to me!

It makes sense to me, that the source code doesn’t have tags in the GetLabelsValues function. If Tyk Pump can’t provide the data we need, I can edit the code base and maintain a local image of our own. Or if you don’t have the plan to add tags for custom metrics yet, I can open a Github PR to do it

9

@olu is right! to confirm: labels are not available yet for the custom prometheus metrics. You will find the list of the field that are supported here: GitHub - TykTechnologies/tyk-pump: Tyk Analytics Pump to move analytics data from Redis to any supported back end (multiple back ends can be written to at once)..

The labels configuration determines the label name and value extracted from the analytic record. The available values are: ["host","method", "path", "response_code", "api_key", "time_stamp", "api_version", "api_name", "api_id", "org_id", "oauth_id", "request_time", "ip_address", "alias"]

I’m curious @byu: how do you use tags in Tyk?

10

Hey @sonja ! Thank you for explaining this!

We extracted some user information in auth plugin, and added them in the tags. And wish pump can send the user info for data analysis usage

11

btw, after we updated to v1.6, there’s something weird happened to Tyk Pump.

I was reading the pump logs flow while sending requests to tyk gateway. However sometimes the number shows in the pump log is much less than what I actually sent. For example, I was using hey to send 300 requests all at once. But pump log showed “Purged 34 records…”. Since it was sent to a database, I checked database but the number is still incorrect. I reverted it back to v1.5.1 and found the number was correct

12

Can you share your pump config for us to verify this?

13

Yep!

version: v1.6
I kinda forget how my configuration file looks like, and forget to save it in a commit. but I’ll try my best to recall:

{
  "analytics_storage_type": "redis",
  "analytics_storage_config": {
    "type": "redis",
    "host": "redis",
    "port": 6379,
    "hosts": null,
    "username": "",
    "password": "",
    "database": 0,
    "optimisation_max_idle": 1000,
    "optimisation_max_active": 2000,
    "enable_cluster": false
  },
  "purge_delay": 10,
  "pumps": {
    "prometheus": {
      "type": "prometheus",
      "meta": {
        "listen_address": ":9595",
        "path": "/metrics",
        "custom_metrics": [
          {
            "name": "tyk_http_tags_total",
            "description": "Total of API requests with tags",
            "metric_type": "counter",
            "labels": [
              "response_code",
              "tags",
              "path"
            ]
          }
        ]
      }
    }
  },
  "dont_purge_uptime_data": true
}

And as I said in the previous conversation, I did a small change for pump to get a value from tags:

// prometheus.go file ~line 330
// extract custom tags
	tagMap := extractTags(decoded.Tags)
	for _, label := range pm.Labels {
		if val, ok := mapping[label]; ok {
			values = append(values, fmt.Sprint(val))
		} else {
			// extract custom tags
			if label == "x_apig_developerid" { // label must be set without "dash" since prom doesn't support it
				if val, ok := tagMap["x-apig-developerid"]; ok {
					values = append(values, fmt.Sprint(val))
				} else {
					values = append(values, fmt.Sprint(""))
				}
			}
		}
	}
	return values

Other than that, cause custom metrics doesn’t support pump env variable configure (i assume), I also had another customized pump configured in the values.yaml.tpl in helm-chart. (so it’s like… one pump configured in pump conf file as above, and another configured by env variable)