Redis clustering issue - "Reconnecting storage: Redis is either down or was not configured"

What I have been trying:
I am using Tyk open source gateway v4.0
I have successfully set up a gateway with a single redis server at back and a couple of APIs.

Now I am trying redis clustering.

What I am getting now:

tyk-gateway_1 | time=“Mar 10 06:32:34” level=debug msg=“No Primary instance found, assuming control” prefix=host-check-mgr
tyk-gateway_1 | time=“Mar 10 06:32:34” level=warning msg=“Reconnecting storage: Redis is either down or was not configured” prefix=pub-sub
tyk-gateway_1 | time=“Mar 10 06:32:44” level=error msg=“Redis health check failed” error=“storage: Redis is either down or was not configured” liveness-check=true prefix=main

I have enabled option to enable the clustering for each of the nodes, and necessary options in my docker-compose, tyk.standalone.conf as well.

Won’t Tyk look after clustering the redis nodes if I set the enable_cluster to true and list the host:port in addrs: []. Is there anything more I should do? P.S: I am running the stuff with docker.

Can somebody please help me solve this? I have already spent a good amount of time over this.

Tyk will not set up the redis cluster for you. You need to ensure that the cluster is properly set up before pointing tyk at it.
You can check the cluster status by running these commands in one of the redis containers.

redis-cli cluster info
# and
redis-cli cluster nodes

Is it possible to run Tyk CE gateway in docker compose with “network_mode: host” instead of other networks. For instance:

networks:

  • tyk

which was there by default when I cloned the git repo.

My requirement is that I need to run a Redis cluster at it’s back and Redis cluster does not seem to support networks other than the host network. Everything had been working fine when I used a single Redis instance. Now I have setup a Redis cluster but I am getting issues with running gateway.

I am using docker compose and when I run docker compose up, it starts running, loading the APIs, policies etc then after some time the gateway instance stops with an error:

tyk-gateway_1 | time=“Mar 12 10:43:55” level=debug msg=“Stopping uptime tests” prefix=host-check-mgr
tyk-gateway_1 | panic: runtime error: invalid memory address or nil pointer dereference
tyk-gateway_1 | [signal SIGSEGV: segmentation violation code=0x1 addr=0x260 pc=0x14607cc]
tyk-gateway_1 |
tyk-gateway_1 | goroutine 42 [running]:
tyk-gateway_1 | github.com/TykTechnologies/tyk/storage.(*RedisCluster).GetKey(0xc000324280, 0x210ca15, 0x16, 0x0, 0x0, 0x0, 0x0)
tyk-gateway_1 | /go/src/github.com/TykTechnologies/tyk/storage/redis_cluster.go:323 +0x10c
tyk-gateway_1 | github.com/TykTechnologies/tyk/gateway.(*HostCheckerManager).AmIPolling(0x2f91f60, 0x0)
tyk-gateway_1 | /go/src/github.com/TykTechnologies/tyk/gateway/host_checker_manager.go:152 +0x213
tyk-gateway_1 | github.com/TykTechnologies/tyk/gateway.(*HostCheckerManager).checkPollerLoop(0x2f91f60, 0x23b47c0, 0xc0004f4f40)
tyk-gateway_1 | /go/src/github.com/TykTechnologies/tyk/gateway/host_checker_manager.go:120 +0x5b
tyk-gateway_1 | github.com/TykTechnologies/tyk/gateway.(*HostCheckerManager).CheckActivePollerLoop(0x2f91f60, 0x23b47c0, 0xc0004f4f40)
tyk-gateway_1 | /go/src/github.com/TykTechnologies/tyk/gateway/host_checker_manager.go:113 +0xd0
tyk-gateway_1 | created by github.com/TykTechnologies/tyk/gateway.(*HostCheckerManager).Start
tyk-gateway_1 | /go/src/github.com/TykTechnologies/tyk/gateway/host_checker_manager.go:87 +0x65
tyk-gateway_1 exited with code 2

Then when I again restart gateway(which failed just now) using Docker desktop UI it runs without any complaints but when I am trying to reach the gateway at localhost:8080 were it is set up, I am getting:

Could not send request
Error: connect ECONNREFUSED 127.0.0.1:8080

Otherwise I would be seeing another error message when the cluster is set properly and I am able to access the redis cluster via redis-cli:

time=“Mar 12 16:12:08” level=warning msg=“Reconnecting storage: Redis is either down or was not configured” prefix=pub-sub
time=“Mar 12 16:12:18” level=error msg=“Redis health check failed” error=“storage: Redis is either down or was not configured” liveness-check=true prefix=main

Can anybody please help me with the correct steps and precautions to be taken while setting up a Tyk gateway with a redis cluster. Any help will be highly appreciated.
PS: My next attempt will be to set up gateway node cluster.

Hi @sailor, have you had a look at our documentation regrading redis cluster configuration?

2 Likes