I am setting up a Tyk cluster in AWS using Elasticache for Redis. Tyk is running and the basics seem to be working, but in my logs I am getting this error:
"Could not EXPIRE key: MOVED 3242 172.31.46.168:6379"
Seems to log out every ten seconds. I haven’t been able to find a lot of info about it thus far. Anyone have a hint for me?
Hi
Do you have any other logs that might help pin this down? This error normally occurs if trying to update or create a key in redis and the key cannot be located in the Redis location you are connected to.
Thanks
Josh
The error I posted is the only thing in the logs besides the normal startup stuff which I will post below. I have not attempted to create any keys yet. I’ve just spun up the Tyk instances.
time="Dec 18 18:36:07" level=info msg="Starting Poller"
time="Dec 18 18:36:07" level=info msg="Initialising Tyk REST API Endpoints"
time="Dec 18 18:36:07" level=info msg="--> Standard listener (http)" port=":8080"
time="Dec 18 18:36:07" level=info msg="Setting up Server"
time="Dec 18 18:36:07" level=info msg="Initialising distributed rate limiter"
time="Dec 18 18:36:07" level=info msg="Starting gateway rate limiter notifications..."
time="Dec 18 18:36:07" level=info msg="Loading API Specification from /opt/tyk-gateway/apps/app_sample.json"
time="Dec 18 18:36:07" level=info msg="Loading API Specification from /opt/tyk-gateway/apps/loyalty.json"
time="Dec 18 18:36:07" level=info msg="Loading API Specification from /opt/tyk-gateway/apps/tpg.json"
time="Dec 18 18:36:07" level=info msg="Health Check initialised" type=log-msg
time="Dec 18 18:36:07" level=info msg="Detected 3 APIs"
time="Dec 18 18:36:07" level=info msg="Loading API configurations."
time="Dec 18 18:36:07" level=info msg="Tracking hostname" api_name="Tyk Test API" domain="(no host)"
time="Dec 18 18:36:07" level=info msg="Tracking hostname" api_name="/v1/loyalty" domain="(no host)"
time="Dec 18 18:36:07" level=info msg="Tracking hostname" api_name="TPG Health Check API" domain="(no host)"
time="Dec 18 18:36:07" level=info msg="Loading API" api_name="TPG Health Check API"
time="Dec 18 18:36:07" level=warning msg="The Health Checker is deprecated and we do no longer recommend its use."
time="Dec 18 18:36:07" level=info msg="Loading API" api_name="Tyk Test API"
time="Dec 18 18:36:07" level=info msg="Loading API" api_name="/v1/loyalty"
time="Dec 18 18:36:07" level=info msg="Checking security policy: Token" api_name="Tyk Test API"
time="Dec 18 18:36:08" level=info msg="Checking security policy: Open" api_name="/v1/loyalty"
time="Dec 18 18:36:08" level=info msg="Checking security policy: Token" api_name="TPG Health Check API"
time="Dec 18 18:36:08" level=info msg="Processed and listening on: /tyk-api-test/{rest:.*}"
time="Dec 18 18:36:08" level=info msg="Processed and listening on: /loyalty/v1/{rest:.*}"
time="Dec 18 18:36:08" level=info msg="Processed and listening on: /tpg/{rest:.*}"
time="Dec 18 18:36:08" level=info msg="Loading uptime tests..."
time="Dec 18 18:36:08" level=info msg="Initialised API Definitions"
time="Dec 18 18:36:08" level=info msg="Loading policies"
time="Dec 18 18:36:08" level=error msg="Couldn't unmarshal policies: EOF"
time="Dec 18 18:36:08" level=info msg="Policies found (0 total):"
time="Dec 18 18:36:08" level=info msg="Gateway started (v2.4.1)"
time="Dec 18 18:36:08" level=info msg="--> Listening on address: (open interface)"
time="Dec 18 18:36:08" level=info msg="--> Listening on port: 8080"
time="Dec 18 18:36:08" level=info msg="--> PID: 8"
time="Dec 18 18:36:17" level=error msg="Could not EXPIRE key: MOVED 3242 172.31.46.168:6379"
time="Dec 18 18:36:27" level=error msg="Could not EXPIRE key: MOVED 3242 172.31.46.168:6379"
time="Dec 18 18:36:37" level=error msg="Could not EXPIRE key: MOVED 3242 172.31.46.168:6379"This error is usually down to an incorrect redis cluster setup - if you are running redis in clustered mode then you need to ensure Tyk can see all elements in the redis cluster.
I think with elasticache in cluster mode there is only one endpoint, perhaps you are using the wrong endpoint?
maybe this will help:
Thanks for the help. The elasticache cluster is operating normally, Tyk is configured with the correct endpoint, and I’ve just double checked security groups/connectivity.
Is there something else in my config I can look at or additional logging I can turn on?
Hi,
Do yu have tyk configured with enable_cluster?
Many thanks,
Martin
Hi Martin,
Sorry for the late reply, been out of the office. Hope you had a great holiday season.
To answer the question, yes, Tyk is configured with "enable_cluster": true
Any other ideas?
It could be just that, you may want to disable that if you only have one redis endpoint. Since the endpoint will be handling the slot hashing for you.
AWS Elasticache has two ways of running Redis, a redis DB server with slaves and multi-AZ failover, and redis in cluster mode, which is a proper redis cluster as is described byt he redis documentation.
The error is being generated from Redis and basically says “The key you are looking for is not on this server”, but if you have only one redis endpoint configured in Tyk, it has no choice but to access that server.
Have you tried that?
I think I’ve got it figured out. enable_cluster=true in tyk.conf doesn’t seem to work in docker for some reason? Or maybe it isn’t supposed to?
When I set the environment variable to enable cluster storage, as well as the storage hosts everything seems to be working as expected.
That is odd… were you mounting the tyk.conf in?
No, no mounts for anything. Going for immutability. Redeploy for config changes.
Once I wrap this project up (next few days) I will attempt to get it back in that same state and send you configs + cloudformation.