Tyk Pump to inject tags column in influx db

Product Feature Requests & Feedback
, ,
1

Hi, we have a requirement of ingesting pump Tags in influx db . The Tags header has a string of values including clients access key and TLS version , we want the whole Tags to be ingested into influx db, but we are unable to do so.
Kindly, find the reference below:

influx": {
“name”: “influx”,
“meta”: {
“database_name”: “connect”,
“address”: “http://metrics-api”,
“username”: “”,
“password”: “”,
“fields”: [
“request_time”
],
“tags”: [
“method”,
“response_code”,
“api_name”,
“tags”
]
}
}
},

2

Hi,
As per your code tyk-pump/influx.go at 3cecf3c56df143290919c70648afe1c3442c17f3 · TykTechnologies/tyk-pump · GitHub

we can see that there are some predefined tags that we can add , can we add custom tags in the influx db from pump

3

Hello @bhavi and welcome to the community.

I don’t think you can custom tags there. The tag names must be the same as the JSON tags of the analytics record fields.

However, I think you can add custom analytics tags using http headers.

4

We have a Tag in our csv generated in tyk-pump we need to ingest that in the influx db.
The Tag includes values such as client-access-key, TLS version, appkeys etc.

5

Can you share your gateway version, pump version and pump config or environment variables for the generated csv pump?

6

TYK Gateway version- 2.5.1
TYK Pump version- 0.7.0

Pump conf file:

“purge_delay”: 1,
“pumps”: {
“csv”: {
“name”: “csv”,
“meta”: {
“csv_dir”: “/csvs/”
}
},
“influx”: {
“name”: “influx”,
“meta”: {
“database_name”: “connect”,
“address”: “http://metrics”,
“username”: “”,
“password”: “”,
“fields”: [
“request_time”
],
“tags”: [“method”, “path”, “response_code”, “api_key”, “time_stamp”, “api_version”, “api_name”, “api_id”, “org_id”, “oauth_id”, “raw_request”, “request_time”, “raw_response”, “ip_address” ]
}
}
},
“dont_purge_uptime_data”: true,

7

I don’t see where you add tags to the csv pump shared. So it appears, I may have failed to understand your earlier statement

You can build a custom pump for your use case. The source of pump is open and available on GitHub, so you can modify it and add functionality based on your needs.

8

I have a TAG in csv pump i had added that in my pump.conf, but the value of TAG column is coming empty in my influx DB, why is that so

9

How did you add the tags client-access-key, TLS version, appkeys for csv?

In your config, I don’t observe a tags section within the csv pump. Maybe you can elaborate more.

About influxdb, are you saying the analytics tags are missing in your influxdb? Or are the custom tags client-access-key, TLS version, appkeys which you wish to add not being populated?

10

We are adding these 3 things in the gateway by using RecordRestHeader policy , then it is being pumped into the tyk-pump csv, now we want it to be ingested into influxDB , we have added that in
“tags”: [“method”, “path”, “response_code”, “api_key”, “time_stamp”, “api_version”, “api_name”, “api_id”, “org_id”, “oauth_id”, “raw_request”, “request_time”, “raw_response”, “ip_address”.“tlsversion”,“clientaccesskey” ]
}

but, these columns in influx db are being populated as null value

11

Thanks for expatiating. I now have a full understanding of the issue.

The CSV pump doesn’t have a tags filter. This means that any tags added to the API request would ultimately end up in storage. However, the influxdb has a filter based on the codebase. That filter ensures that only the default tags from the analytics are sent to your datasink. No custom tags are allowed

I could transform this thread into a feature request for the engineering team. However, decision to implement the said feature still rests with the engineering team.

I would suggest an alternative where you can modify the source code of pump and build a custom version for yourself. The suggestion is based on the fact that any added feature request would be available on the latest version of pump.

Pump should work on any version of the gateway but there have been some recent breaking changes. So considering your version is slightly old one v0.7.0, I would just advise that you edit the source and remove the limitation.

Hope this helps