Connection to Redis failed, reconnect in 10s

Support Installation and Deployment
,
1

Hi,
I wana setup tyk-gateway using helm, but after tyk-gateway up, it keeps logging “Connection to Redis failed, reconnect in 10s”.
I have tested in tyk-gateway pod that redis cluster is accessable, I have no idea what went wrong

Tyk-Gateway Log

Defaulted container "tyk-gateway" out of: tyk-gateway, setup-directories (init)
time="Aug 01 02:51:24" level=info msg="Tyk API Gateway 4.1.0-rc16" prefix=main
time="Aug 01 02:51:24" level=warning msg="Insecure configuration allowed" config.allow_insecure_configs=true prefix=checkup
time="Aug 01 02:51:24" level=warning msg="AnalyticsConfig.PoolSize unset. Defaulting to number of available CPUs" prefix=checkup runtime.NumCPU=4
time="Aug 01 02:51:24" level=warning msg="AnalyticsConfig.RecordsBufferSize < minimum - Overriding" minRecordsBufferSize=1000 prefix=checkup
time="Aug 01 02:51:24" level=error msg="cannot set key in pollerCacheKey" error="storage: Redis is either down or was not configured"
time="Aug 01 02:51:24" level=info msg="Starting Poller" prefix=host-check-mgr
time="Aug 01 02:51:24" level=error msg="Could not set version in versionStore" error="storage: Redis is either down or was not configured" prefix=main
time="Aug 01 02:51:24" level=info msg="PIDFile location set to: /mnt/tyk-gateway/tyk.pid" prefix=main
time="Aug 01 02:51:24" level=warning msg="The control_api_port should be changed for production" prefix=main
time="Aug 01 02:51:24" level=info msg="Initialising Tyk REST API Endpoints" prefix=main
time="Aug 01 02:51:24" level=error msg="Connection to Redis failed, reconnect in 10s" error="storage: Redis is either down or was not configured" prefix=pub-sub
time="Aug 01 02:51:24" level=info msg="--> [REDIS] Creating cluster client"
time="Aug 01 02:51:24" level=info msg="--> Standard listener (http)" port=":8080" prefix=main
time="Aug 01 02:51:24" level=warning msg="Starting HTTP server on:[::]:8080" prefix=main
time="Aug 01 02:51:24" level=info msg="Initialising distributed rate limiter" prefix=main
time="Aug 01 02:51:24" level=info msg="Tyk Gateway started (4.1.0-rc16)" prefix=main
time="Aug 01 02:51:24" level=info msg="--> Listening on address: (open interface)" prefix=main
time="Aug 01 02:51:24" level=info msg="--> Listening on port: 8080" prefix=main
time="Aug 01 02:51:24" level=info msg="--> PID: 1" prefix=main
time="Aug 01 02:51:24" level=info msg="Starting gateway rate limiter notifications..."
time="Aug 01 02:51:24" level=info msg="Loading policies" prefix=main
time="Aug 01 02:51:24" level=info msg="Policies found (3 total):" prefix=main
time="Aug 01 02:51:24" level=info msg="Loading API Specification from /mnt/tyk-gateway/apis/web.json"
time="Aug 01 02:51:24" level=info msg="Detected 1 APIs" prefix=main
time="Aug 01 02:51:24" level=info msg="Loading API configurations." prefix=main
time="Aug 01 02:51:24" level=info msg="Tracking hostname" api_name=Web domain=developer-api.nftgo.dev prefix=main
time="Aug 01 02:51:24" level=info msg="Initialising Tyk REST API Endpoints" prefix=main
time="Aug 01 02:51:24" level=info msg="API bind on custom port:0" prefix=main
time="Aug 01 02:51:24" level=info msg="API hostname set: developer-api.nftgo.dev" prefix=main
time="Aug 01 02:51:24" level=info msg="Checking security policy: Open" api_id=web api_name=Web org_id=
time="Aug 01 02:51:24" level=info msg="API Loaded" api_id=web api_name=Web org_id= prefix=gateway server_name=-- user_id=-- user_ip=--
time="Aug 01 02:51:24" level=info msg="Loading uptime tests..." prefix=host-check-mgr
time="Aug 01 02:51:24" level=info msg="Initialised API Definitions" prefix=main
time="Aug 01 02:51:24" level=info msg="API reload complete" prefix=main
time="Aug 01 02:51:25" level=info msg="--> [REDIS] Creating cluster client"
time="Aug 01 02:51:25" level=info msg="--> [REDIS] Creating cluster client"
time="Aug 01 02:51:34" level=error msg="cannot set key in pollerCacheKey" error="storage: Redis is either down or was not configured"
time="Aug 01 02:51:34" level=error msg="Redis health check failed" error="storage: Redis is either down or was not configured" liveness-check=true prefix=main
time="Aug 01 02:51:34" level=error msg="Connection to Redis failed, reconnect in 10s" error="storage: Redis is either down or was not configured" prefix=pub-sub
time="Aug 01 02:51:44" level=error msg="cannot set key in pollerCacheKey" error="storage: Redis is either down or was not configured"
time="Aug 01 02:51:44" level=error msg="Redis health check failed" error="storage: Redis is either down or was not configured" liveness-check=true prefix=main
time="Aug 01 02:51:44" level=error msg="Connection to Redis failed, reconnect in 10s" error="storage: Redis is either down or was not configured" prefix=pub-sub

Tyk-Gateway Deployment file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tyk-gateway
  labels:
    app: tyk-gateway
spec:
  replicas: 1
  minReadySeconds: 5
  selector:
    matchLabels:
      app: tyk-gateway
  template:
    metadata:
      labels:
        app: tyk-gateway
    spec:
      initContainers:
      - name: "setup-directories"
        image: busybox:1.32
        command: ['sh','-c','mkdir -p apps middleware policies && touch policies/policies.json']
        workingDir: /mnt/tyk-gateway
        volumeMounts:
          - name: tyk-scratch
            mountPath: /mnt/tyk-gateway
      containers:
      - name: tyk-gateway
        image: docker.tyk.io/tyk-gateway/tyk-gateway:v4.1.0-rc16
        imagePullPolicy: "IfNotPresent"
        env:
          # Lagacy support for Redis Cluster driver. Driver dropped in v3.0.0.
          - name: REDIGOCLUSTER_SHARDCOUNT
            value: "128"
          - name: TYK_GW_SECRET
            value: "1234"
          - name: TYK_GW_NODESECRET
            value: "1234"
        command: ["/opt/tyk-gateway/tyk", "--conf=/etc/tyk-gateway/tyk.conf"]
        workingDir: /opt/tyk-gateway
        ports:
          - containerPort: 8080
        volumeMounts:
          - name: tyk-mgmt-gateway-conf
            mountPath: /etc/tyk-gateway
          - name: tyk-scratch
            mountPath: /mnt/tyk-gateway
          - name: tyk-gateway-policies
            mountPath: /mnt/tyk-gateway/policies
          - name: tyk-gateway-apis
            mountPath: /mnt/tyk-gateway/apis
      volumes:
        - name: tyk-scratch
          emptyDir: {}
        - name: tyk-gateway-policies
          configMap:
            name: tyk-gateway-policies
        - name: tyk-gateway-apis
          configMap:
            name: tyk-gateway-apis
        - name: tyk-mgmt-gateway-conf
          configMap:
            name: tyk-gateway-config
            items:
              - key: tyk-gateway-config.json
                path: tyk.conf

Tyk-Gateway Config

apiVersion: v1
kind: ConfigMap
metadata:
  name: tyk-gateway-config
  labels:
    app: tyk-gateway-config
data:
  tyk-gateway-config.json: |
    {
      "enable_hashed_keys_listing": true,
      "listen_port": 8080,
      "secret": "1234",
      "node_secret": "1234",
      "template_path": "/opt/tyk-gateway/templates",
      "tyk_js_path": "/opt/tyk-gateway/js/tyk.js",
      "middleware_path": "/mnt/tyk-gateway/middleware",
      "use_db_app_configs": false,
      "db_app_conf_options": {
        "connection_string": "",
        "node_is_segmented": false,
        "tags": []
      },
      "app_path": "/mnt/tyk-gateway/apis",
      "storage": {
        "type": "redis",
        "enable_cluster": true,
        "host": "redis-cluster.tyk",
        "port": 6379,
        "username": "",
        "password": "Yc2Mv5f1pC",
        "database": 0,
        "optimisation_max_idle": 1000
      },
      "enable_analytics": true,
      "analytics_config": {
        "type": "mongo",
        "enable_detailed_recording": false,
        "storage_expiration_time": 60,
        "enable_multiple_analytics_keys": true,
        "purge_interval": 10,
        "ignored_ips": []
      },
      "health_check": {
        "enable_health_checks": false,
        "health_check_value_timeouts": 60
      },
      "optimisations_use_async_session_write": true,
      "enable_non_transactional_rate_limiter": true,
      "enable_sentinel_rate_limiter": false,
      "allow_master_keys": false,
      "policies": {
        "policy_source": "file",
        "policy_record_name": "/mnt/tyk-gateway/policies/policies.json",
        "allow_explicit_policy_id": true
      },
      "hash_keys": false,
      "hash_key_function": "",
      "close_connections": false,
      "http_server_options": {
        "enable_websockets": true,
        "use_ssl": false,
        "server_name": "*",
        "min_version": 771,
        "certificates": [
          {
            "domain_name": "*",
            "cert_file": "/etc/certs/cert.pem",
            "key_file": "/etc/certs/key.pem"
          }
        ]
      },
      "allow_insecure_configs": true,
      "coprocess_options": {
        "enable_coprocess": true,
        "coprocess_grpc_server": ""
      },
      "enable_bundle_downloader": false,
      "bundle_base_url": "",
      "global_session_lifetime": 100,
      "force_global_session_lifetime": false,
      "max_idle_connections_per_host": 500,
      "enable_custom_domains": true,
      "pid_file_location": "/mnt/tyk-gateway/tyk.pid"
    }

Redis Cluster

I installed redis cluster by helm

helm install redis-cluster bitnami/redis-cluster --version 8.1.0  --namespace tyk --set image.tag=7.0.3-debian-11-r1
1 Like
2

From the gateway logs, it looks like there is a connection issue with your Redis. You said you tested in the gateway pod that Redis cluster is accessible. Can you share how you tested?

Also can you crosscheck the redis properties used in the gateway:

  • name/host
  • port
  • username
  • password
  • enable_cluster

If cluster is enabled then please, follow our guide on Configuring Redis Cluster

3

For how I test redis conneciton in gateway.
actually I just installed redis-cli in gateway pod, and use redis-cli -c to connect redis cluster everything is fine

I still can’t figure out what went wrong (I tried you suggested and all I could think of)
I finally decided to configure gateway useing standalone redis, that is ok
Anyway, thank you too much for helping :slight_smile:

4

Did you try to change your redis version from 7.0.3 to Redis version 2.8.x to 6.0.x.From the documentation it seems that those are the supported redis versions.

1 Like