Tyk-pump pumping or not

Support Installation and Deployment
1

Hi
after deploying tyk-gateway CE in a so easy way, I’m facing difficulties about tyk-pump config (and analytics)
I get several test apps working fine on tyk-gateway and I updated gateway config to activate analytics

...
 "enable_analytics": true,
 "analytics_config": {
   "type": "",
   "storage_expiration_time":120,
   "ignored_ips": [],
   "normalise_urls": {
   "enabled": true,
   "normalise_uuids": true,
   "normalise_numbers": true,
   "custom_patterns": []
 }
},

And I install Tyk Pump and setup with same Redis DB

{
  "analytics_storage_type": "redis",
  "analytics_storage_config": {
	"type": "redis",
	"host": "localhost",
	"hosts": null,
	"port": 6379,
	"username": "",
	"password": "XXXXXX",
	"database": 0,
	"optimisation_max_idle": 2000,
	"optimisation_max_active": 4000,
	"enable_cluster": false,
	"redis_use_ssl": false,
	"redis_ssl_insecure_skip_verify": false
  },
  "log_level":"info",
  "log_format":"text",
  "purge_delay": 5,
  "health_check_endpoint_name": "hello",
  "health_check_endpoint_port": 8083,
  "pumps": {
	"syslog": {
	  "name": "syslog",
	  "meta": {
		"transport": "udp",
		"network_addr": "localhost:5140",
		"log_level": 6,
		"tag":"syslog-pump"
	  }
	},
	"elasticsearch": {
	  "type": "elasticsearch",
	  "meta": {
		"index_name": "tyk_analytics",
		"elasticsearch_url": "http://xxxxxxxxxx:9200",
		"enable_sniffing": false,
		"document_type": "tyk_analytics",
		"rolling_index": false,
		"extended_stats": false,
		"version": "6"
	  }
	}
  },
  "dont_purge_uptime_data": false,
  "omit_detailed_recording": false
}

I launch both service a cli with --DEBUG flag to check what’s happing… and nothing, in log, syslog… except in / out trace for proxy on tyk-gateway

I checked Redis DB and keys show that both are well connected to

127.0.0.1:6379> KEYS *
1) tyk-liveness-probe
2) apikey-73bba5f1
3) host-checker:PollerActiveInstanceID
4) redis-test-xxxxxxxxxxxxxxxxx
5) quota-73bba5f1
6) apikey-995f85f8
7) version-check-pump
8) analytics-tyk-system-analytics
9) foo

Key 7 is from pump
Key 8 looks like linked to analytics and disappears (from Gateway)

But pump seems not finding in DB something to do nor display attempt to pump something in DB
is it gateway not writing analytics or pump not finding the right key?

thanks for help

S.

2

Hi @ECL_DSI and welcome to o community.

You have your pump set up to move the analytics to syslog and elasticsearch. can you confirm you are not receiving the analytics in those locations?

Thanks, G.

3

HI @Gregor
not sure to understand the question
I provide pump.conf in ticket and both destination are setup
But as pump not seems detect any element from Redis DB, it generates none export to pump destination, it’s what I’m guessing.
I generated lots of requests from clients through Tyk Gateway and Pump in debug mode doesnt show any event about “pumping” something, so I guess it consider there isn’t something to do.
It’s why I checked Redis DB content to see if something was written by gateway into, and found a key “analytics-tyk-system-analytics”. But nothing in doc explain which one is in survey process by pump, so I’m not sure it’s the right one.

4

I’m progressing
It appears a param should be true

"dont_purge_uptime_data": true,

Once done, the pump process is “pumping”

 [Jun 11 15:33:05]  INFO elasticsearch-pump: Purged 3 records...
 [Jun 11 15:33:05]  INFO syslog-pump: Purged 3 records...
 [Jun 11 15:38:40]  INFO elasticsearch-pump: Purged 1 records...
 [Jun 11 15:38:40]  INFO syslog-pump: Purged 1 records...
 [Jun 11 15:56:05]  INFO elasticsearch-pump: Purged 1 records...
 [Jun 11 15:56:05]  INFO syslog-pump: Purged 1 records...

But nothing yet in Syslog with “sudo tail -f /var/log/syslog”

To continue

5

About syslog, basic error from my side
rsyslogd service wasn’t installed with listener for UDP/TCP activated to collect pump sending and populate syslog file.
Now requests are displayed in syslog :+1:

Next ElasticSearch

To continue

6

Issue with ElasticSearch is now solved
It appears the GIT doc is wrong if you are using last release of ElasticSearch (v7+)
dont use “document_type” as “tyk_analytics” which is not recognized by ElasticSearch v7+, use generic doc “_doc”.

"elasticsearch": {
  "type": "elasticsearch",
  "meta": {
    "index_name": "tyk_analytics_staging",
    "elasticsearch_url": "http://<SERVER_URL>:9200",
    "enable_sniffing": false,
    "document_type": "_doc", <====== dont use "tyk_analytics"
    "rolling_index": false,
    "extended_stats": true,
    "version": "6",
    "bulk_config":{
      "workers": 2,
      "flush_interval": 60
    }