Reducing cardinality of Tyk Pump analytics

Product FAQ
, , ,
1

I’ve currently got Tyk Pump set up with the Prometheus backend, tracking all paths and have it configured for some custom metrics (just to make the labels more relevant).

An example metric being emitted for scraping is as follows:

tyk_http_requests_total{api_name="my-api",method="GET",path="/v3/resource/123abc/subresource/abc123",response_code="401"} 1

The path information above is valuable, but I’m not interested in the nitty gritty of the specific wildcard resource IDs that are in the URL so I was wondering if there was a way I could remove the concrete ids in favour of producing something like this:

tyk_http_requests_total{api_name="my-api",method="GET",path="/v3/resource/{resourceId}/subresource/{subResourceId}",response_code="401"} 1

I can’t see an obvious way to configure this within Tyk itself but was wondering if anyone else has had any luck with this sort of thing?

2

@DeanBruntThirdfort Welcome to the community.

Off the top of my head, you may be looking for analytics_config.normalise_urls. If I am not wrong one of the

  • normalise_uuids
  • normalise_ulids or
  • custom_patterns

should work for you

1 Like
3

Hi Olu,

I’ve not tested those yet, but they look like they’ll be bang on the money.
I’ll give them a try and get back to you, many thanks!

4

Just to circle back to this, Olu’s suggestion did indeed work
.
In our case, we use XIDs and prefixed XIDs so we leveraged the custom_patterns part of the config.
One gotcha I found that took a moment to work out is when setting the custom patterns via environment variable (TYK_GW_ANALYTICSCONFIG_NORMALISEURLS_CUSTOM), you have to use a comma separated list string rather than a stringified array.

In our case, this meant that the working config was to set the following env vars:

TYK_GW_ANALYTICSCONFIG_NORMALISEURLS_ENABLED=true
TYK_GW_ANALYTICSCONFIG_NORMALISEURLS_CUSTOM=prefix[0-9a-z]{20},[0-9a-z]{20}