Hi guys,
I am getting crazy with this problem. Trying to set up tyk gateway in Kubernetes. I already set up redis cluster that runs in 6 instances (3 master and 3 slave), and everything seems OK.
On the other side, I am using tyk gateway image tykio/tyk-gateway and deploying it based on tyk-oss-k8s-deployment github yamls. I set up storage as type redis and cluster enabled. Problem is that I am seeing this messages continously:
[Sep 21 14:34:20] DEBUG Creating new Redis connection pool
[Sep 21 14:34:20] INFO --> [REDIS] Creating cluster client
[Sep 21 14:34:29] ERROR main: Redis health check failed error=storage: Redis is either down or was not configured liveness-check=true
[Sep 21 14:34:29] DEBUG host-check-mgr: No Primary instance found, assuming control
[Sep 21 14:34:29] ERROR cannot set key in pollerCacheKey error=storage: Redis is either down or was not configured
[Sep 21 14:34:29] DEBUG HOST CHECKER: Host list reset
[Sep 21 14:34:29] ERROR pub-sub: Connection to Redis failed, reconnect in 10s error=storage: Redis is either down or was not configured
[Sep 21 14:34:39] ERROR main: Redis health check failed error=storage: Redis is either down or was not configured liveness-check=true
[Sep 21 14:34:39] DEBUG host-check-mgr: No Primary instance found, assuming control
[Sep 21 14:34:39] ERROR cannot set key in pollerCacheKey error=storage: Redis is either down or was not configured
[Sep 21 14:34:39] ERROR pub-sub: Connection to Redis failed, reconnect in 10s error=storage: Redis is either down or was not configured
[Sep 21 14:34:49] ERROR main: Redis health check failed error=storage: Redis is either down or was not configured liveness-check=true
[Sep 21 14:34:49] DEBUG host-check-mgr: No Primary instance found, assuming control
[Sep 21 14:34:49] ERROR cannot set key in pollerCacheKey error=storage: Redis is either down or was not configured
[Sep 21 14:34:49] ERROR pub-sub: Connection to Redis failed, reconnect in 10s error=storage: Redis is either down or was not configured
[Sep 21 14:34:59] DEBUG host-check-mgr: No Primary instance found, assuming control
My tyk.conf (maybe a little messed up yet), looks like this
{
"listen_address": "",
"listen_port": 8081,
"secret": "352d20ee67be67f6340b4c0605b044b7",
"template_path": "/opt/tyk/templates",
"tyk_js_path": "/opt/tyk/js/tyk.js",
"middleware_path": "/opt/tyk/middleware",
"use_db_app_configs": false,
"db_app_conf_options": {
"connection_string": "http://tyk-dashboard.tyk.svc.cluster.local:3000",
"node_is_segmented": false,
"tags": ["test2"]
},
"app_path": "/opt/tyk/apps/",
"storage": {
"type": "redis",
"enable_cluster": true,
"addrs": [
"redis-cluster-0.redis-cluster.redis.svc.cluster.local:6379",
"redis-cluster-3.redis-cluster.redis.svc.cluster.local:6379",
"redis-cluster-1.redis-cluster.redis.svc.cluster.local:6379",
"redis-cluster-4.redis-cluster.redis.svc.cluster.local:6379",
"redis-cluster-2.redis-cluster.redis.svc.cluster.local:6379",
"redis-cluster-5.redis-cluster.redis.svc.cluster.local:6379"]
,
"database": 0,
"optimisation_max_idle": 2000,
"optimisation_max_active": 4000,
"username": "",
"password": "",
"use_ssl": false
},
"enable_analytics": false,
"analytics_config": {
"type": "mongo",
"csv_dir": "/tmp",
"mongo_url": "",
"mongo_db_name": "",
"mongo_collection": "",
"purge_delay": -1,
"ignored_ips": []
},
"optimisations_use_async_session_write": true,
"enable_non_transactional_rate_limiter": true,
"enable_sentinel_rate_limiter": false,
"enable_redis_rolling_limiter": false,
"allow_master_keys": false,
"hash_keys": true,
"close_connections": true,
"http_server_options": {
"enable_websockets": true
},
"allow_insecure_configs": true,
"coprocess_options": {
"enable_coprocess": false,
"coprocess_grpc_server": ""
},
"enable_bundle_downloader": true,
"bundle_base_url": "",
"global_session_lifetime": 100,
"force_global_session_lifetime": false,
"max_idle_connections_per_host": 100,
"use_ssl": false,
"redis_addrs": [
"redis-cluster-0.redis-cluster.redis.svc.cluster.local:6379",
"redis-cluster-3.redis-cluster.redis.svc.cluster.local:6379",
"redis-cluster-1.redis-cluster.redis.svc.cluster.local:6379",
"redis-cluster-4.redis-cluster.redis.svc.cluster.local:6379",
"redis-cluster-2.redis-cluster.redis.svc.cluster.local:6379",
"redis-cluster-5.redis-cluster.redis.svc.cluster.local:6379"
]
}
I think I tryed everything right now.
Tryed with different versions with same results, but actually Iâm on tyk gateway 4.2 and redis 6.0.16.
Tryed âuse_sslâ: false and true
Visibility seems ok between pods. From inside tyk pod:
root@tyk-gateway-65df8fdb9c-w5ls4:~# nc -vz redis-cluster-0.redis-cluster.redis.svc.cluster.local 6379
Connection to redis-cluster-0.redis-cluster.redis.svc.cluster.local (10.244.2.194) 6379 port [tcp/redis] succeeded!
root@tyk-gateway-65df8fdb9c-w5ls4:~# nc -vz redis-cluster-1.redis-cluster.redis.svc.cluster.local 6379
Connection to redis-cluster-1.redis-cluster.redis.svc.cluster.local (10.244.4.207) 6379 port [tcp/redis] succeeded!
root@tyk-gateway-65df8fdb9c-w5ls4:~# nc -vz redis-cluster-2.redis-cluster.redis.svc.cluster.local 6379
Connection to redis-cluster-2.redis-cluster.redis.svc.cluster.local (10.244.3.139) 6379 port [tcp/redis] succeeded!
root@tyk-gateway-65df8fdb9c-w5ls4:~# nc -vz redis-cluster-3.redis-cluster.redis.svc.cluster.local 6379
Connection to redis-cluster-3.redis-cluster.redis.svc.cluster.local (10.244.4.209) 6379 port [tcp/redis] succeeded!
root@tyk-gateway-65df8fdb9c-w5ls4:~# nc -vz redis-cluster-4.redis-cluster.redis.svc.cluster.local 6379
Connection to redis-cluster-4.redis-cluster.redis.svc.cluster.local (10.244.3.140) 6379 port [tcp/redis] succeeded!
root@tyk-gateway-65df8fdb9c-w5ls4:~# nc -vz redis-cluster-5.redis-cluster.redis.svc.cluster.local 6379
Connection to redis-cluster-5.redis-cluster.redis.svc.cluster.local (10.244.2.195) 6379 port [tcp/redis] succeeded!
Also installed redis-cli on gateway pod, and played with some commands just to check I could access without problems.
Please guys, could you give some clues?
Thank you
EDIT - some more info:
root@tyk-gateway-65df8fdb9c-w5ls4:~# netstat -tunlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:33763 0.0.0.0:* LISTEN 1/tyk
tcp6 0 0 :::8081 :::* LISTEN 1/tyk
root@redis-cluster-0:~# netstat -plnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:6379 0.0.0.0:* LISTEN 13/redis-server *:6
tcp 0 0 0.0.0.0:16379 0.0.0.0:* LISTEN 13/redis-server *:6
tcp6 0 0 :::6379 :::* LISTEN 13/redis-server *:6
tcp6 0 0 :::16379 :::* LISTEN 13/redis-server *:6
Also comment that redis and tyk pods are in different namespaces, but that shouldnât be a problem right?
EDIT2: Tryed with redis and tyk gateway on same namespace. The problem persists.
I attach complete log:
time="Sep 23 06:41:07" level=debug msg="Using /opt/tyk-gateway/tyk.conf for configuration" prefix=main
time="Sep 23 06:41:07" level=info msg="Tyk API Gateway 4.2.0" prefix=main
time="Sep 23 06:41:07" level=warning msg="Insecure configuration allowed" config.allow_insecure_configs=true prefix=checkup
time="Sep 23 06:41:07" level=warning msg="Default secret should be changed for production." config.secret=352d20ee67be67f6340b4c0605b044b7 prefix=checkup
time="Sep 23 06:41:07" level=error msg="Could not set version in versionStore" error="storage: Redis is either down or was not configured" prefix=main
time="Sep 23 06:41:07" level=debug msg="No Primary instance found, assuming control" prefix=host-check-mgr
time="Sep 23 06:41:07" level=error msg="cannot set key in pollerCacheKey" error="storage: Redis is either down or was not configured"
time="Sep 23 06:41:07" level=info msg="Starting Poller" prefix=host-check-mgr
time="Sep 23 06:41:07" level=debug msg="---> Initialising checker" prefix=host-check-mgr
time="Sep 23 06:41:07" level=debug msg="[HOST CHECKER] Config:TriggerLimit: 3"
time="Sep 23 06:41:07" level=debug msg="[HOST CHECKER] Config:Timeout: ~10"
time="Sep 23 06:41:07" level=debug msg="[HOST CHECKER] Config:WorkerPool: 2"
time="Sep 23 06:41:07" level=info msg="Rich plugins are disabled" prefix=coprocess
time="Sep 23 06:41:07" level=debug msg="Notifier will not work in hybrid mode" prefix=main
time="Sep 23 06:41:07" level=debug msg="[HOST CHECKER] Init complete"
time="Sep 23 06:41:07" level=debug msg="---> Starting checker" prefix=host-check-mgr
time="Sep 23 06:41:07" level=debug msg="[HOST CHECKER] Starting..."
time="Sep 23 06:41:07" level=debug msg="[HOST CHECKER] Check loop started..."
time="Sep 23 06:41:07" level=debug msg="[HOST CHECKER] Host reporter started..."
time="Sep 23 06:41:07" level=debug msg="---> Checker started." prefix=host-check-mgr
time="Sep 23 06:41:07" level=info msg="PIDFile location set to: /var/run/tyk/tyk-gateway.pid" prefix=main
time="Sep 23 06:41:07" level=warning msg="The control_api_port should be changed for production" prefix=main
time="Sep 23 06:41:07" level=debug msg="Initialising default org store" prefix=main
time="Sep 23 06:41:07" level=info msg="Initialising Tyk REST API Endpoints" prefix=main
time="Sep 23 06:41:07" level=error msg="Connection to Redis failed, reconnect in 10s" error="storage: Redis is either down or was not configured" prefix=pub-sub
time="Sep 23 06:41:07" level=debug msg="Creating new Redis connection pool"
time="Sep 23 06:41:07" level=info msg="--> [REDIS] Creating cluster client"
time="Sep 23 06:41:07" level=debug msg="Loaded API Endpoints" prefix=main
time="Sep 23 06:41:07" level=info msg="--> Standard listener (http)" port=":8081" prefix=main
time="Sep 23 06:41:07" level=warning msg="Starting HTTP server on:[::]:8081" prefix=main
time="Sep 23 06:41:07" level=info msg="Initialising distributed rate limiter" prefix=main
time="Sep 23 06:41:07" level=debug msg="DRL: Setting node ID: solo-dad9e877-b232-44fe-961e-1e3b5d451c2b|tyk-gateway-65df8fdb9c-6znfx"
time="Sep 23 06:41:07" level=info msg="Tyk Gateway started (4.2.0)" prefix=main
time="Sep 23 06:41:07" level=info msg="--> Listening on address: (open interface)" prefix=main
time="Sep 23 06:41:07" level=info msg="--> Listening on port: 8081" prefix=main
time="Sep 23 06:41:07" level=info msg="--> PID: 1" prefix=main
time="Sep 23 06:41:07" level=info msg="Loading policies" prefix=main
time="Sep 23 06:41:07" level=debug msg="No policy record name defined, skipping..." prefix=main
time="Sep 23 06:41:07" level=info msg="Loading API Specification from /opt/tyk/apps/app_sample.json"
time="Sep 23 06:41:07" level=info msg="Starting gateway rate limiter notifications..."
time="Sep 23 06:41:07" level=debug msg="Checking for transform paths..."
time="Sep 23 06:41:07" level=debug msg="Checking for transform paths..."
time="Sep 23 06:41:07" level=info msg="Detected 1 APIs" prefix=main
time="Sep 23 06:41:07" level=info msg="Loading API configurations." prefix=main
time="Sep 23 06:41:07" level=info msg="Tracking hostname" api_name="Tyk Test API" domain="(no host)" prefix=main
time="Sep 23 06:41:07" level=info msg="Initialising Tyk REST API Endpoints" prefix=main
time="Sep 23 06:41:07" level=debug msg="Loaded API Endpoints" prefix=main
time="Sep 23 06:41:07" level=info msg="API bind on custom port:0" prefix=main
time="Sep 23 06:41:07" level=debug msg="Initializing API" api_id=1 api_name="Tyk Test API" org_id=default
time="Sep 23 06:41:07" level=debug msg="Batch requests enabled for API" prefix=main
time="Sep 23 06:41:07" level=debug msg=Init api_id=1 api_name="Tyk Test API" mw=VersionCheck org_id=default
time="Sep 23 06:41:07" level=debug msg=Init api_id=1 api_name="Tyk Test API" mw=RateCheckMW org_id=default
time="Sep 23 06:41:07" level=info msg="Checking security policy: Token" api_id=1 api_name="Tyk Test API" org_id=default
time="Sep 23 06:41:07" level=debug msg=Init api_id=1 api_name="Tyk Test API" mw=AuthKey org_id=default
time="Sep 23 06:41:07" level=debug msg=Init api_id=1 api_name="Tyk Test API" mw=KeyExpired org_id=default
time="Sep 23 06:41:07" level=debug msg=Init api_id=1 api_name="Tyk Test API" mw=AccessRightsCheck org_id=default
time="Sep 23 06:41:07" level=debug msg=Init api_id=1 api_name="Tyk Test API" mw=GranularAccessMiddleware org_id=default
time="Sep 23 06:41:07" level=debug msg=Init api_id=1 api_name="Tyk Test API" mw=RateLimitAndQuotaCheck org_id=default
time="Sep 23 06:41:07" level=debug msg=Init api_id=1 api_name="Tyk Test API" mw=VersionCheck org_id=default
time="Sep 23 06:41:07" level=debug msg=Init api_id=1 api_name="Tyk Test API" mw=KeyExpired org_id=default
time="Sep 23 06:41:07" level=debug msg=Init api_id=1 api_name="Tyk Test API" mw=AccessRightsCheck org_id=default
time="Sep 23 06:41:07" level=debug msg="Rate limit endpoint is: /tyk-api-test/tyk/rate-limits" api_id=1 api_name="Tyk Test API" org_id=default
time="Sep 23 06:41:07" level=debug msg="Setting Listen Path: /tyk-api-test/" api_id=1 api_name="Tyk Test API" org_id=default
time="Sep 23 06:41:07" level=info msg="API Loaded" api_id=1 api_name="Tyk Test API" org_id=default prefix=gateway server_name=-- user_id=-- user_ip=--
time="Sep 23 06:41:07" level=debug msg="Checker host list" prefix=main
time="Sep 23 06:41:07" level=info msg="Loading uptime tests..." prefix=host-check-mgr
time="Sep 23 06:41:07" level=debug msg="--- Setting tracking list up" prefix=host-check-mgr
time="Sep 23 06:41:07" level=debug msg="Reset initiated" prefix=host-check-mgr
time="Sep 23 06:41:07" level=debug msg="[HOST CHECKER] Checker reset queued!"
time="Sep 23 06:41:07" level=debug msg="Checker host Done" prefix=main
time="Sep 23 06:41:07" level=info msg="Initialised API Definitions" prefix=main
time="Sep 23 06:41:07" level=info msg="API reload complete" prefix=main
time="Sep 23 06:41:08" level=debug msg="Creating new Redis connection pool"
time="Sep 23 06:41:08" level=info msg="--> [REDIS] Creating cluster client"
time="Sep 23 06:41:08" level=debug msg="Creating new Redis connection pool"
time="Sep 23 06:41:08" level=info msg="--> [REDIS] Creating cluster client"
time="Sep 23 06:41:17" level=error msg="Redis health check failed" error="storage: Redis is either down or was not configured" liveness-check=true prefix=main
time="Sep 23 06:41:17" level=debug msg="No Primary instance found, assuming control" prefix=host-check-mgr
time="Sep 23 06:41:17" level=error msg="cannot set key in pollerCacheKey" error="storage: Redis is either down or was not configured"
time="Sep 23 06:41:17" level=error msg="Connection to Redis failed, reconnect in 10s" error="storage: Redis is either down or was not configured" prefix=pub-sub
time="Sep 23 06:41:18" level=debug msg="[HOST CHECKER] Host list reset"
time="Sep 23 06:41:27" level=debug msg="No Primary instance found, assuming control" prefix=host-check-mgr
time="Sep 23 06:41:27" level=error msg="cannot set key in pollerCacheKey" error="storage: Redis is either down or was not configured"
time="Sep 23 06:41:27" level=error msg="Redis health check failed" error="storage: Redis is either down or was not configured" liveness-check=true prefix=main
time="Sep 23 06:41:27" level=error msg="Connection to Redis failed, reconnect in 10s" error="storage: Redis is either down or was not configured" prefix=pub-sub
time="Sep 23 06:41:37" level=debug msg="No Primary instance found, assuming control" prefix=host-check-mgr
time="Sep 23 06:41:37" level=error msg="cannot set key in pollerCacheKey" error="storage: Redis is either down or was not configured"
time="Sep 23 06:41:37" level=error msg="Redis health check failed" error="storage: Redis is either down or was not configured" liveness-check=true prefix=main
time="Sep 23 06:41:37" level=error msg="Connection to Redis failed, reconnect in 10s" error="storage: Redis is either down or was not configured" prefix=pub-sub
time="Sep 23 06:41:47" level=error msg="Redis health check failed" error="storage: Redis is either down or was not configured" liveness-check=true prefix=main
time="Sep 23 06:41:47" level=debug msg="No Primary instance found, assuming control" prefix=host-check-mgr
time="Sep 23 06:41:47" level=error msg="cannot set key in pollerCacheKey" error="storage: Redis is either down or was not configured"
time="Sep 23 06:41:47" level=error msg="Connection to Redis failed, reconnect in 10s" error="storage: Redis is either down or was not configured" prefix=pub-sub
time="Sep 23 06:41:57" level=debug msg="No Primary instance found, assuming control" prefix=host-check-mgr
time="Sep 23 06:41:57" level=error msg="cannot set key in pollerCacheKey" error="storage: Redis is either down or was not configured"
time="Sep 23 06:41:57" level=error msg="Redis health check failed" error="storage: Redis is either down or was not configured" liveness-check=true prefix=main
time="Sep 23 06:41:57" level=error msg="Connection to Redis failed, reconnect in 10s" error="storage: Redis is either down or was not configured" prefix=pub-sub
time="Sep 23 06:42:07" level=debug msg="No Primary instance found, assuming control" prefix=host-check-mgr
time="Sep 23 06:42:07" level=error msg="cannot set key in pollerCacheKey" error="storage: Redis is either down or was not configured"
time="Sep 23 06:42:07" level=error msg="Redis health check failed" error="storage: Redis is either down or was not configured" liveness-check=true prefix=main
time="Sep 23 06:42:07" level=error msg="Connection to Redis failed, reconnect in 10s" error="storage: Redis is either down or was not configured" prefix=pub-sub