Tyk-pump cannot pump

Hello,
I’ve a tyk-ce on K8s; enabled pump in values.yaml file; below;

...
  extraEnvs:
    - name: TYK_LOGLEVEL
      value: debug
    - name: TYK_GW_ENABLEHASHEDKEYSLISTING
      value: true
    - name: TYK_GW_ENABLEANALYTICS
      value: true
    - name: TYK_DB_USESHARDEDANALYTICS
      value: true  
    - name: TYK_PMP_PUMPS_PROM_TYPE
      value: prometheus
    - name: TYK_PMP_PUMPS_PROM_TIMEOUT
      value: 10
    - name: TYK_PMP_PUMPS_PROM_META_LISTENADDRESS
      value: "prometheus-server.monitoring.svc.cluster.local:80"
    - name: TYK_PMP_PUMPS_PROM_META_PATH
      value: "/metrics"

    - name: TYK_PMP_PUMPS_PROM_ANALYTICSSTORAGETYPE
      value: redis
    - name: TYK_PMP_PUMPS_PROM_ANALYTICSSTORAGECONFIG_HOST
      value: tyk-redis-master.tyk.svc.cluster.local
    - name: TYK_PMP_PUMPS_PROM_ANALYTICSSTORAGECONFIG_PORT
      value: 6379
    - name: TYK_PMP_PUMPS_PROM_ANALYTICSSTORAGECONFIG_PASSWORD
      value: "veryhiddenpassword"
    # - name: TYK_PMP_PUMPS_PROM_ANALYTICSSTORAGECONFIG_DATABASE
    #   value: 0
    - name: TYK_PMP_PUMPS_PROM_ANALYTICSSTORAGECONFIG_MAXIDLE
      value: 100
    - name: TYK_PMP_PUMPS_PROM_ANALYTICSSTORAGECONFIG_MAXACTIVE
      value: 100
    - name: TYK_PMP_PUMPS_PROM_ANALYTICSSTORAGECONFIG_ENABLECLUSTER
      value: false
    - name: TYK_PMP_PUMPS_PROM_ANALYTICSSTORAGECONFIG_MASTERNAME
      value: tyk-redis-master
    - name: TYK_PMP_ANALYTICSSTORAGECONFIG_REDISUSESSL
      value: false
    - name: TYK_PMP_ANALYTICSSTORAGECONFIG_REDISSSLINSECURESKIPVERIFY
      value: true
  mounts: []

# If pump is enabled the Gateway will create and collect analytics data to send
# to a data store of your choice. These can be set up in the pump config. The
# possible pump configs can be found here:
# https://github.com/TykTechnologies/tyk-pump#configuration
pump:
  # Determines whither or not the pump component should be installed.
  enabled: true

  replicaCount: 1
  image:
    repository: docker.tyk.io/tyk-pump/tyk-pump
    tag: v1.4.0
    pullPolicy: IfNotPresent
...

but there is no any action in logs; looks like pump cannot pump.

k logs -n tyk pump-tyk-ce-tyk-headless-6799f5d555-9hvmz -f
time="Oct  5 21:20:02" level=info msg="## Tyk Analytics Pump, 1.4.0 ##"
time="Oct  5 21:20:02" level=info msg="--> [REDIS] Creating single-node client"
time="Oct  5 21:20:02" level=info msg="Serving health check endpoint at http://localhost:8083/health ..."

also there is nothing at prometheus.

Am I missing something? Please advise…

Regards

Hello @tirelibirefe, since you are gonna configure pump with Prometheus for the first time, setting ENV variables will not work as they are only meant to override the configs set in config file for the Tyk Pump.

The config file for pump is defined in a ConfigMap named “pump-conf-tyk-ce-tyk-headless”. Remove the Env variables which you have set for pump in values.yaml file and edit this configMap with the prometheus config as mentioned here Prometheus pump config as per your environment.

Once the changes are saved in configMap, restart the pump pod to pick up the changes and you should be able to see Prometheus Pump Initialization from pump pod logs.

Hello @Cherry
here is my prometheus ns:

k get svc -n monitoring
NAME                            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
grafana                         ClusterIP   10.100.92.209    <none>        80/TCP     2d7h
prometheus-alertmanager         ClusterIP   10.100.149.234   <none>        80/TCP     2d7h
prometheus-kube-state-metrics   ClusterIP   10.100.155.134   <none>        8080/TCP   2d7h
prometheus-node-exporter        ClusterIP   None             <none>        9100/TCP   2d7h
prometheus-pushgateway          ClusterIP   10.100.250.177   <none>        9091/TCP   2d7h
prometheus-server               ClusterIP   10.100.205.165   <none>        80/TCP     2d7h

Here is my configmap

apiVersion: v1
data:
  pump.conf: |-
    {
      "analytics_storage_type": "redis",
      "analytics_storage_config": {
        "type": "redis",
        "enable_cluster": false,
        "host": "tyk-redis-master.tyk.svc.cluster.local",
        "username": "",
        "password": "",
        "database": 0,
        "optimisation_max_idle": 2000,
        "optimisation_max_active": 4000
      },
      "purge_delay": 4,
      "pumps": {
        "mongo": {
          "name": "mongo",
          "meta": {
            "collection_name": "tyk_analytics_headless",
            "mongo_url": ""
          }
        },
        "mongo-pump-aggregate": {
          "name": "mongo-pump-aggregate",
          "meta": {
            "mongo_url": "",
            "use_mixed_collection": true
          }
        }
      },
        "prometheus": {
          "type": "prometheus",
            "meta": {
                "listen_address": "prometheus-server.monitoring.svc.cluster.local:80",
                "path": "/metrics"
            }
        },
      "uptime_pump_config": {
        "collection_name": "tyk_uptime_analytics_headless",
        "mongo_url": ""
      },
      "dont_purge_uptime_data": false
    }
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: tyk-ce
    meta.helm.sh/release-namespace: tyk
  creationTimestamp: "2021-10-05T21:20:01Z"
  labels:
    app: pump-tyk-ce-tyk-headless
    app.kubernetes.io/managed-by: Helm
    chart: tyk-headless-0.9.3
    heritage: Helm
    release: tyk-ce
  name: pump-conf-tyk-ce-tyk-headless
  namespace: tyk
  resourceVersion: "3033085"
  uid: ebca7276-77a8-4559-9165-07c5fa9e6016

I edit configmap and reapplied it then I deleted the pump pod and it was recreated. Also I’ve removed environment variables from values.yaml file of gw.

…and the result:

k logs -n tyk pump-tyk-ce-tyk-headless-6799f5d555-4f6bb -f
time="Oct  6 07:11:11" level=info msg="## Tyk Analytics Pump, 1.4.0 ##"
time="Oct  6 07:11:11" level=info msg="--> [REDIS] Creating single-node client"
time="Oct  6 07:11:11" level=info msg="Serving health check endpoint at http://localhost:8083/health ..."
time="Oct  6 07:11:12" level=info msg=Init collection_name="tyk_analytics_headless" url=
time="Oct  6 07:11:12" level=info msg="-- No max batch size set, defaulting to 10MB"
time="Oct  6 07:11:12" level=info msg="-- No max document size set, defaulting to 10MB"

Nothing changed, pump pod sits there without pump.

P.S > Do I not need to fill redis auth information? As you know redis has only password which is defined in values.yaml file and gateway pods authenticate by using. Does pump pod read it and use the same information to logon redis?

Do I not need to fill redis auth information?

I don’t think you need the redis auth information in pump. My redis section looks similar to yours so that should be fine.

Regarding your pump config, I observed that your prometheus section is outside the pumps section. It needs to be a child of the pumps section. Also, if you are using CE then you could remove the mongo configurations. I have dropped a snippet below

{
  "analytics_storage_type": "redis",
  "analytics_storage_config": {
	"type": "redis",
	"enable_cluster": false,
	"host": "tyk-redis-master.tyk.svc.cluster.local",
	"username": "",
	"password": "",
	"database": 0,
	"optimisation_max_idle": 2000,
	"optimisation_max_active": 4000
  },
  "log_level":"debug",
  "log_format":"text",
  "purge_delay": 4,
  "pumps": {
	"mongo": {
	  "name": "mongo",
	  "meta": {
		"collection_name": "tyk_analytics_headless",
		"mongo_url": ""
	  }
	},
	"mongo-pump-aggregate": {
	  "name": "mongo-pump-aggregate",
	  "meta": {
		"mongo_url": "",
		"use_mixed_collection": true
	  }
	},
	"prometheus": {
	  "type": "prometheus",
		"meta": {
			"listen_address": "prometheus-server.monitoring.svc.cluster.local:80",
			"path": "/metrics"
		}
	}
  }
  "uptime_pump_config": {
	"collection_name": "tyk_uptime_analytics_headless",
	"mongo_url": ""
  },
  "dont_purge_uptime_data": true
}

I have also enabled debug logging for more information if errors occur. Let us know how it goes.

Hello @tirelibirefe, I was reproducing in my cluster and it turns out that we cannot connect pump to prometheus because pump deployed using helm-charts currently does not expose any service endpoint which can be used by the prometheus to scrape metrics.

Hoping that this would be added in upcoming releases !

Feel free to open a github issue in tyk-helm-chart repo !!!

Thank you @Cherry This last comment is very helpful, I was struggle with it…
Ok, I open an issue to GitHub…