Prometheus custom metrics in 1.6.0 not working?

Support
, ,
1

We’re trying to expose custom metrics with pump to prometheus. However if we define the custom metrics as described in the Readme we’re gettng an error in the log:

time="Nov 11 15:02:21" level=info msg="## Tyk Analytics Pump, 1.6.0 ##"
time="Nov 11 15:02:21" level=info msg="--> [REDIS] Creating single-node client"
time="Nov 11 15:02:21" level=info msg="Serving health check endpoint at http://localhost:8083/health ..."
time="Nov 11 15:02:22" level=error msg="invalid metric type:"
time="Nov 11 15:02:22" level=error msg="invalid metric type:"
time="Nov 11 15:02:22" level=error msg="invalid metric type:"
time="Nov 11 15:02:22" level=error msg="invalid metric type:"
time="Nov 11 15:02:22" level=error msg="invalid metric type:"
time="Nov 11 15:02:22" level=info msg="Starting prometheus listener on::9090"
time="Nov 11 15:02:22" level=info msg="Prometheus Pump Initialized"
time="Nov 11 15:02:22" level=info msg="Init Pump: PROMETHEUS"

Our Config:

TYK_PMP_PUMPS_PROMETHEUS_TYPE=prometheus
TYK_PMP_PUMPS_PROMETHEUS_META_ADDR=localhost:9090
TYK_PMP_PUMPS_PROMETHEUS_META_PATH=/metrics
TYK_PMP_PUMPS_PROMETHEUS_META_CUSTOMMETRICS='[{"name":"tyk_http_requests_total","description":"Total of API requests","metric_type":"counter","labels":["response_code","api_name"]}]'
TYK_PMP_PUMPS_PROMETHEUS_META_DISABLED_METRICS=[]
2

Hello @markus.wiedmer

Welcome back to the community

On v1.6.0, Prometheus pump was missing the _META postfix in its default environment variable prefix.

A fix for this is included in Pump 1.7.0 set to be released later this month/early Dec.

The custom metrics work fine when configured in the pump.conf though. See a sample configuration below:

"prometheus": {
  "type": "prometheus",
  "meta": {
    "listen_address": ":8092",
    "path": "/metrics",
    "custom_metrics": [
      {
        "name": "tyk_http_requests_total",
        "description": "Total of API requests",
        "metric_type": "counter",
        "labels": ["response_code", "api_name", "method", "api_key","alias", "path"]
      },
      {
        "name": "tyk_http_latency",
        "description": "Latency of API requests",
        "metric_type": "histogram",
        "labels": ["type", "response_code", "api_name", "method", "api_key", "alias", "path"]
      }
    ]
  }
}

Will this be a way forward for you until v1.7 is out?

1 Like
3

Hi Michael,

Switching form env-File to conf was our original intent. however if we set it up that way we can only get the prometheus endpoint to work if we do not activate the mongo-db-pump as well. If we do enable mongo-pump. the prometheus config seems to be entirely ignored. If we disable the mongo-db-pump, the prometheus endpoint works.

{
  "analytics_storage_type": "redis",
  "analytics_storage_config": {
    "type": "redis",
    "host": "<redis-host>",
    "port": 6379,
    "hosts": null,
    "username": "<user>",
    "password": "<pw>",
    "database": 0,
    "optimisation_max_idle": 100,
    "optimisation_max_active": 0,
    "enable_cluster": false
  },
  "purge_delay": 10,
  "pumps": {
    "mongo": {
      "type": "mongo",
      "meta": {
        "collection_name": "tyk_analytics",
        "mongo_url": "mongodb:/<mongo-db-host>:27017/tyk_analytics",
        "collection_cap_max_size_bytes": 1048576,
        "collection_cap_enable": true
      }
    },
    "mongo-pump-aggregate": {
      "type": "mongo-pump-aggregate",
      "meta": {
        "mongo_url": "mongodb:/<mongo-db-host>:27017/tyk_analytics",
        "use_mixed_collection": true
      }
    },
    "prometheus": {
      "type": "prometheus",
      "meta": {
        "listen_address": ":9090",
        "path": "/metrics",
        "custom_metrics": [
          {
            "name": "tyk_http_requests_total",
            "description": "Total of API requests",
            "metric_type": "counter",
            "labels": ["response_code", "api_name", "method", "api_key","alias", "path"]
          },
          {
            "name": "tyk_http_latency",
            "description": "Latency of API requests",
            "metric_type": "histogram",
            "labels": ["type", "response_code", "api_name", "method", "api_key", "alias", "path"]
          }
        ]
      }
    }
  },
  "uptime_pump_config": {
    "collection_name": "tyk_uptime_analytics",
    "mongo_url": "mongodb:/<mongo-db-host>:27017/tyk_analytics",
    "collection_cap_max_size_bytes": 1048576,
    "collection_cap_enable": true
  },
  "dont_purge_uptime_data": false
}
4

Hi @markus.wiedmer,

There’s a known bug with custom metrics for 1.6.0, and a current workaround is modifying the labels like so:

"prometheus": {
    "type": "prometheus",
    "meta": {
      "listen_address": ":8092",
      "path": "/metrics",
      "custom_metrics": [
        {
          "name": "tyk_http_requests_total",
          "description": "Total of API requests",
          "metric_type": "counter",
          "labels": ["response_code", "api_name", "method", "api_key", "path"]
        },
        {
          "name": "tyk_http_latency",
          "description": "Latency of API requests",
          "metric_type": "histogram",
          "labels": ["response_code", "api_name", "method", "api_key", "path"]
        }
      ]
    }

This works fine together with other pumps AFAIK. Try this in your setup.
You can use only one of the custom metrics though… one or the other, not both.

The env_fix, as well as this, is expected in 1.7.0

5

we changed the custom metric to the notation given here, prometheus and mongodb pump now both work, but I stll get the same error (time=“Nov 22 09:13:49” level=error msg=“invalid metric type:”) in the log and the custom metrics don’t appear in the prometheus endpoint. I guess we’ll just wait for 1.7 and try again.

6

Hey @markus.wiedmer

Am happy to let you know Pump v1.7.0 is out.

Give it a try and let us know how you get along.

7

thank you, we upgraded to 1.7.0 now. Custom Metrics are working now, but we had to change the definition from the Readme.md a bit

when we set the property as documented:

TYK_PMP_PUMPS_PROMETHEUS_META_CUSTOMMETRICS='[{"name":"tyk_http_requests_total","description":"Total of API requests","metric_type":"counter","labels":["response_code","api_name"]}]'

we got a syntax-error. we then omitted the enclosing ’ and now it seems to be working well