Couldn't initialize python dispatcher

Support Installation and Deployment
, ,
1

We are testing tyk in EKS. Everything was working fine but started breaking once we started testing how to use custom python plugins. I am sure redis is fine and gateway was able to connect to redis cluster.

time="Oct 27 09:16:59" level=debug msg="[HOST CHECKER] Config:WorkerPool: 64"
time="Oct 27 09:16:59" level=debug msg="[HOST CHECKER] Init complete"
time="Oct 27 09:16:59" level=debug msg="---> Starting checker" prefix=host-check-mgr
time="Oct 27 09:16:59" level=debug msg="[HOST CHECKER] Starting..."
time="Oct 27 09:16:59" level=debug msg="[HOST CHECKER] Check loop started..."
time="Oct 27 09:16:59" level=debug msg="[HOST CHECKER] Host reporter started..."
time="Oct 27 09:16:59" level=debug msg="---> Checker started." prefix=host-check-mgr
time="Oct 27 09:16:59" level=info msg="PIDFile location set to: /mnt/tyk-gateway/tyk.pid" prefix=main
time="Oct 27 09:16:59" level=warning msg="The control_api_port should be changed for production" prefix=main
time="Oct 27 09:16:59" level=debug msg="Initialising default org store" prefix=main
time="Oct 27 09:16:59" level=info msg="Initialising Tyk REST API Endpoints" prefix=main
time="Oct 27 09:16:59" level=error msg="Connection to Redis failed, reconnect in 10s" error="storage: Redis is either down or was not configured" prefix=pub-sub
time="Oct 27 09:16:59" level=debug msg="Creating new Redis connection pool"
time="Oct 27 09:16:59" level=info msg="--> [REDIS] Creating cluster client"
time="Oct 27 09:16:59" level=debug msg="Loaded API Endpoints" prefix=main
time="Oct 27 09:16:59" level=info msg="--> Standard listener (http)" port=":8080" prefix=main
time="Oct 27 09:16:59" level=warning msg="Starting HTTP server on:[::]:8080" prefix=main
time="Oct 27 09:16:59" level=info msg="Initialising distributed rate limiter" prefix=main
time="Oct 27 09:16:59" level=debug msg="DRL: Setting node ID: solo-5c7221cd-c0d2-40c7-92e2-79e672f249be|gateway-tyk-gateway-tyk-headless-69976b84b7-sk8rd"
time="Oct 27 09:16:59" level=info msg="Tyk Gateway started (5.1.0)" prefix=main
time="Oct 27 09:16:59" level=info msg="--> Listening on address: (open interface)" prefix=main
time="Oct 27 09:16:59" level=info msg="--> Listening on port: 8080" prefix=main
time="Oct 27 09:16:59" level=info msg="--> PID: 1" prefix=main
time="Oct 27 09:16:59" level=info msg="Loading policies" prefix=main
time="Oct 27 09:16:59" level=info msg="Loading policy from dir /mnt/tyk-gateway/policies/policies.json"
time="Oct 27 09:16:59" level=error msg="Couldn't unmarshal policy configuration from dir: /mnt/tyk-gateway/policies/policies.json : EOF"
time="Oct 27 09:16:59" level=info msg="Policies found (1 total):" prefix=main
time="Oct 27 09:16:59" level=debug msg=" - " prefix=main
time="Oct 27 09:16:59" level=info msg="Loading API Specification from /mnt/tyk-gateway/apps/ZHN0L2FkdC1jYXJnby1tb3ZlbWVudHMtYXBpa2V5.json"
time="Oct 27 09:16:59" level=info msg="Starting gateway rate limiter notifications..."
time="Oct 27 09:16:59" level=info msg="Loading existing bundle: bundle.zip" prefix=main
time="Oct 27 09:16:59" level=info msg="----> Loading bundle: bundle.zip" prefix=main
time="Oct 27 09:16:59" level=info msg="----> Using bundle: bundle.zip" prefix=main
time="Oct 27 09:16:59" level=debug msg="Requested python version: \"\"" prefix=dlpython
time="Oct 27 09:16:59" level=debug msg="Found python-config binary: 3.9 (/usr/bin/python3.9-config)" prefix=dlpython
time="Oct 27 09:16:59" level=debug msg="Using latest Python version" prefix=dlpython
time="Oct 27 09:16:59" level=debug msg="Selected Python configuration path: /usr/bin/python3.9-config" prefix=dlpython
time="Oct 27 09:16:59" level=debug msg="Selected Python library path: /usr/lib/python3.9/config-3.9-x86_64-linux-gnu/libpython3.9.so" prefix=dlpython
time="Oct 27 09:16:59" level=debug msg="Creating new Redis connection pool"
time="Oct 27 09:16:59" level=info msg="--> [REDIS] Creating cluster client"
time="Oct 27 09:16:59" level=info msg="Python version '3.9' loaded" prefix=coprocess
time="Oct 27 09:16:59" level=fatal msg="Couldn't initialize Python dispatcher" prefix=coprocess

Here is the middleware.py

from tyk.decorators import *
from gateway import TykGateway as tyk

@Hook
def RewriteUpstreamMiddleware(request, session, metadata, spec):
    # Extract the host from the incoming request
    tyk.log("I'm logged!", "info")
    tyk.log("Request body" + request.object.body, "info")
    tyk.log("API config_data" + spec['config_data'], "info")
    new_host = "seeing-if-it-works"
    request.object.url = "http://" + new_host + "private.com"
    return request, session, metadata
2

Hi @utkarsh079,

Please share your manifest.json and your bundling process.

3

This is the manifest.json

  {
    "file_list": [
      "rewrite_upstream.py"
    ],
    "custom_middleware": {
      "driver": "python",
      "post": [
        {
          "name": "RewriteUpstreamMiddleware",
          "require_session": false
        }
      ]
    }
  }

bundling process:

docker run \  
  --rm -w "/tmp" -v $(pwd):/tmp \
  --entrypoint "/bin/sh" -it \
  tykio/tyk-gateway:$IMAGETAG \
  -c '/opt/tyk-gateway/tyk bundle build -o "Bundle.zip" -y'

IMAGETAG=v5.1.0

4

Hi @utkarsh079,

I don’t experience this error in my environment. It loads fine.

Are you able to reproduce it outside of EKS?

5

I have not tried this outside EKS. I can try in another cluster but I will have to resolve this error as I have to run tyk in this cluster only.

What I am not understanding is why I am getting “redis is down” error (that to be without “retry” message) after the APIDefinition is deployed. Before that everything is working fine.

I have tested with gateway 5.2.2 and still failing. Would really appreciate some help here.

6

Hi,

This error always happens once during startup. The redis connection is managed asynchronously from a background thread which hasn’t yet established the session. Another thread tries to use the connection and finds it down so the error occurs. A few seconds later it tries again and succeeds.

Are you using the tyk built images to do this? I suspect not because there are mentions of /mnt/tyk-gateway in the logs. It would be helpful to mention this with future questions.

coprocess_options.python_path_prefix must be set to the base directory that tyk is installed to so that the custom python modules referred to in your plugin code can be loaded.

Please set TYK_GW_COPROCESSOPTIONS_PYTHONPATHPREFIX=/mnt/tyk-gateway and try again.

Cheers,
Pete

7

Thanks for your reply @Pete
Yes, I am using Tyk built images. There has been no modification to the image.

8

I finally go it resolved. I was following this article https://tyk.io/blog/how-to-deploy-python-plugins-in-tyk-running-on-kubernetes/

The quotes here are some different quotes and copy pasted some of values.