LUA + PYTHON PLUGINS: panic: runtime error: invalid memory address or nil pointer dereference

Your configuration looks good (tyk.conf and API definition).

Did you use tyk-cli to build the bundle?

Yes:

/opt/tyk-gateway/utils/tyk-cli bundle build

also tested:

/opt/tyk-gateway/utils/tyk-cli bundle build --output bundle.python.zip

Could you share your full gateway log, when starting tyk-gateway-python and tyk-gateway-lua?

I guess it’s a problem with the libraries (not being loaded).

Here is the log from the python:

time=“Feb 24 08:40:21” level=info msg=“Connection dropped, connecting…”
time=“Feb 24 08:40:21” level=info msg=“Setting PYTHONPATH to ‘/opt/tyk-gateway/coprocess/python:/opt/tyk-gateway/middleware/python:/opt/tyk-gateway/event_handlers:/opt/tyk-gateway/coprocess/python/proto:middleware/bundles/7ff770fbe75d4a2144d6938a41216d4d-bundle.python.zip’”
time=“Feb 24 08:40:21” level=info msg=“Initializing interpreter, Py_Initialize()”
time=“Feb 24 08:40:25” level=error msg=“Can’t initialize a dispatcher”
time=“Feb 24 08:40:29” level=info msg=“Setting up analytics normaliser”
time=“Feb 24 08:40:29” level=info msg=“PIDFile location set to: ./tyk-gateway.pid”
time=“Feb 24 08:40:29” level=error msg=“Instrumentation is enabled, but no connectionstring set for statsd”
time=“Feb 24 08:40:29” level=info msg=“Initialising Tyk REST API Endpoints”
time=“Feb 24 08:40:29” level=info msg="–> Standard listener (http)"
time=“Feb 24 08:40:29” level=info msg=“Setting up Server”
time=“Feb 24 08:40:29” level=info msg=“Waiting for zeroconf signal…”
time=“Feb 24 08:40:31” level=info msg=“Starting Poller”
time=“Feb 24 08:40:31” level=warning msg=“Insecure configuration detected (allowing)!”
time=“Feb 24 08:40:31” level=info msg=“Hostname set with dashboard zeroconf signal”
time=“Feb 24 08:40:32” level=info msg=“Registering node.”
time=“Feb 24 08:40:32” level=info msg=“Node registered” id=d027fa3b-d85f-4024-6b32-ac07d91dc565
time=“Feb 24 08:40:32” level=info msg=“Initialising distributed rate limiter”
time=“Feb 24 08:40:32” level=info msg=“Starting gateway rate imiter notifications…”
time=“Feb 24 08:40:32” level=info msg=“Detected 8 APIs”
time=“Feb 24 08:40:32” level=info msg=“Loading API configurations.”

(…)

time=“Feb 24 08:40:32” level=info msg=“Tracking hostname” api_name=“Szymon API” domain="(no host)"

(…)

time=“Feb 24 08:40:32” level=info msg=“Loading API” api_name=“Szymon API”
time=“Feb 24 08:40:32” level=info msg=“Loading existing bundle: bundle.python.zip”
time=“Feb 24 08:40:32” level=info msg="----> Loading bundle: bundle.python.zip"
time=“Feb 24 08:40:32” level=info msg="----> Using bundle: bundle.python.zip"
time=“Feb 24 08:40:32” level=info msg=“Checking security policy: Open” api_name=“Szymon API”
time=“Feb 24 08:40:32” level=info msg=“Loaded: Szymon API” api_id=7ff770fbe75d4a2144d6938a41216d4d org_id=5885dc5bafa02d0de73efb11 server_name=-- user_id=-- user_ip=–

(…)

time=“Feb 24 08:40:33” level=info msg=“Processed and listening on: /szymonk/{rest:.*}”
time=“Feb 24 08:40:33” level=info msg=“Loading uptime tests…”
time=“Feb 24 08:40:33” level=info msg=“Initialised API Definitions”
time=“Feb 24 08:40:33” level=info msg=“Loading policies”
time=“Feb 24 08:40:33” level=info msg=“Using Policies from Dashboard Service”
time=“Feb 24 08:40:33” level=info msg=“Mutex lock acquired… calling”
time=“Feb 24 08:40:33” level=info msg=“Calling dashboard service for policy list”
time=“Feb 24 08:40:33” level=info msg=“Processing policy list”
time=“Feb 24 08:40:33” level=info msg="–> Processing policy ID: 588600d2afa02d0717189033"
time=“Feb 24 08:40:33” level=info msg="–> Processing policy ID: 5888abafafa02d0e3aca2863"
time=“Feb 24 08:40:33” level=info msg="–> Processing policy ID: 5889b01cafa02d0e3aca286a"
time=“Feb 24 08:40:33” level=info msg="–> Processing policy ID: 58aebf38afa02d394b95abae"
time=“Feb 24 08:40:33” level=info msg=“Gateway started (v2.3.2)”
time=“Feb 24 08:40:33” level=info msg="–> Listening on address: (open interface)"
time=“Feb 24 08:40:33” level=info msg="–> Listening on port: 8080"
time=“Feb 24 08:40:33” level=info msg="–> PID: 26414"
time=“Feb 24 08:40:35” level=info msg=“Using /opt/tyk-gateway/tyk.conf for configuration”

and the log from lua:

time=“Feb 24 08:48:08” level=info msg=“Connection dropped, connecting…”
time=“Feb 24 08:48:08” level=info msg=“Starting Poller”
time=“Feb 24 08:48:08” level=info msg=“Loading Tyk/Lua modules.”
time=“Feb 24 08:48:08” level=info msg=“Setting up analytics normaliser”
time=“Feb 24 08:48:08” level=info msg=“PIDFile location set to: ./tyk-gateway.pid”
time=“Feb 24 08:48:08” level=error msg=“Instrumentation is enabled, but no connectionstring set for statsd”
time=“Feb 24 08:48:08” level=info msg=“Initialising Tyk REST API Endpoints”
time=“Feb 24 08:48:08” level=info msg="–> Standard listener (http)"
time=“Feb 24 08:48:08” level=info msg=“Setting up Server”
time=“Feb 24 08:48:08” level=info msg=“Waiting for zeroconf signal…”
time=“Feb 24 08:48:09” level=warning msg=“Insecure configuration detected (allowing)!”
time=“Feb 24 08:48:09” level=info msg=“Hostname set with dashboard zeroconf signal”
time=“Feb 24 08:48:09” level=info msg=“Registering node.”
time=“Feb 24 08:48:09” level=info msg=“Node registered” id=d027fa3b-d85f-4024-6b32-ac07d91dc565
time=“Feb 24 08:48:09” level=info msg=“Initialising distributed rate limiter”
time=“Feb 24 08:48:09” level=info msg=“Starting gateway rate imiter notifications…”
time=“Feb 24 08:48:09” level=info msg=“Detected 8 APIs”
time=“Feb 24 08:48:09” level=info msg=“Loading API configurations.”

(…)

time=“Feb 24 08:48:09” level=info msg=“Tracking hostname” api_name=“Szymon API” domain="(no host)"

(…)

time=“Feb 24 08:48:09” level=info msg=“Loading API” api_name=“Szymon API”
time=“Feb 24 08:48:09” level=info msg=“Loading existing bundle: bundle.lua.zip”
time=“Feb 24 08:48:09” level=info msg=“Loaded: Mock API” api_id=904c9096b5df472458cf7b2860150954 org_id=5885dc5bafa02d0de73efb11 server_name=-- user_id=-- user_ip=–
time=“Feb 24 08:48:09” level=info msg="----> Loading bundle: bundle.lua.zip"
time=“Feb 24 08:48:09” level=info msg="----> Using bundle: bundle.lua.zip"
time=“Feb 24 08:48:09” level=info msg=“Checking security policy: Open” api_name=“Szymon API”
time=“Feb 24 08:48:09” level=info msg=“Loaded: Szymon API” api_id=7ff770fbe75d4a2144d6938a41216d4d org_id=5885dc5bafa02d0de73efb11 server_name=-- user_id=-- user_ip=–

(…)

time=“Feb 24 08:48:10” level=info msg=“Processed and listening on: /szymonk/{rest:.*}”
time=“Feb 24 08:48:10” level=info msg=“Loading uptime tests…”
time=“Feb 24 08:48:10” level=info msg=“Initialised API Definitions”
time=“Feb 24 08:48:10” level=info msg=“Loading policies”
time=“Feb 24 08:48:10” level=info msg=“Using Policies from Dashboard Service”
time=“Feb 24 08:48:10” level=info msg=“Mutex lock acquired… calling”
time=“Feb 24 08:48:10” level=info msg=“Calling dashboard service for policy list”
time=“Feb 24 08:48:10” level=info msg=“Processing policy list”
time=“Feb 24 08:48:10” level=info msg="–> Processing policy ID: 588600d2afa02d0717189033"
time=“Feb 24 08:48:10” level=info msg="–> Processing policy ID: 5888abafafa02d0e3aca2863"
time=“Feb 24 08:48:10” level=info msg="–> Processing policy ID: 5889b01cafa02d0e3aca286a"
time=“Feb 24 08:48:10” level=info msg="–> Processing policy ID: 58aebf38afa02d394b95abae"
time=“Feb 24 08:48:10” level=info msg=“Gateway started (v2.3.2)”
time=“Feb 24 08:48:10” level=info msg="–> Listening on address: (open interface)"
time=“Feb 24 08:48:10” level=info msg="–> Listening on port: 8080"
time=“Feb 24 08:48:10” level=info msg="–> PID: 26474"
time=“Feb 24 08:48:12” level=info msg=“Using /opt/tyk-gateway/tyk.conf for configuration”

I see you’re getting:

time="Feb 24 08:40:21" level=info msg="Initializing interpreter, Py_Initialize()" 
time="Feb 24 08:40:25" level=error msg="Can't initialize a dispatcher" 

It may indicate a path problem (the gateway isn’t able to find its modules) or the Python version isn’t compatible.

How to verify the path problem?

Note that I’m getting panic for lua version as well. What about lua issue?

I’ve got something in:
/opt/tyk-gateway/coprocess/python

[root@tyk python]# ls
README.md init.py pycache binding.h cythonize dispatcher.h dispatcher.py proto tyk

Are we talking about these files?

How to set the path and where?

The path is set automatically, based on the current directory of the tyk binary.
Yes, those files!

Ok, so any ideas how to force it to work?

Why lua version is not working?

Regards

Anyone?

Regards

Szymon

Which versions are you using?

Lua requirements

Python requirements

Hey

I’m also seeing this issue using docker image tykio/tyk-gateway:v2.3.6

My bundles are stored in an s3 bucket.

time=“Jul 13 16:39:42” level=info msg=“Connection dropped, connecting…”
time=“Jul 13 16:39:42” level=info msg=“Setting PYTHONPATH to ‘/opt/tyk-gateway/coprocess/python:/opt/tyk-gateway/middleware/python:/opt/tyk-gateway/event_handlers:/opt/tyk-gateway/coprocess/python/proto’”
time=“Jul 13 16:39:42” level=info msg=“Initializing interpreter, Py_Initialize()”
time=“Jul 13 16:39:43” level=info msg=“Initializing dispatcher”
time=“Jul 13 16:39:43” level=info msg=“PIDFile location set to: /var/run/tyk-gateway.pid”
time=“Jul 13 16:39:43” level=error msg=“Instrumentation is enabled, but no connectionstring set for statsd”
time=“Jul 13 16:39:43” level=info msg=“Initialising Tyk REST API Endpoints”
time=“Jul 13 16:39:43” level=info msg="–> Standard listener (http)"
time=“Jul 13 16:39:43” level=info msg=“Setting up Server”
time=“Jul 13 16:39:43” level=info msg=“Initialising distributed rate limiter”
time=“Jul 13 16:39:43” level=info msg=“Loading API Specification from /opt/tyk-gateway/apps/abc123.json”
time=“Jul 13 16:39:43” level=info msg=“Starting gateway rate imiter notifications…”
time=“Jul 13 16:39:43” level=info msg=“Loading API Specification from /opt/tyk-gateway/apps/abc321.json”
time=“Jul 13 16:39:43” level=info msg=“Loading API Specification from /opt/tyk-gateway/apps/abc789.json”
time=“Jul 13 16:39:43” level=info msg=“Detected 3 APIs”
time=“Jul 13 16:39:43” level=info msg=“Loading API configurations.”
time=“Jul 13 16:39:43” level=info msg=“Tracking hostname” api_name=“APIONE” domain="(no host)"
time=“Jul 13 16:39:43” level=info msg=“Tracking hostname” api_name=APITWO domain="(no host)"
time=“Jul 13 16:39:43” level=info msg=“Tracking hostname” api_name=“APITHREE” domain="(no host)"
time=“Jul 13 16:39:43” level=info msg=“Loading API” api_name=“API One”
time=“Jul 13 16:39:43” level=info msg=“Checking security policy: Open” api_name=“API One”
time=“Jul 13 16:39:43” level=info msg=“Loading API” api_name=“API Two”
time=“Jul 13 16:39:43” level=info msg=“Checking security policy: Open” api_name=“API Two”
time=“Jul 13 16:39:43” level=info msg=“Loading API” api_name=API Three
time=“Jul 13 16:39:43” level=info msg="----> Fetching Bundle: authentication"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x446b45]

goroutine 288 [running]:
panic(0xcc4280, 0xc420012080)
/usr/local/go/src/runtime/panic.go:500 +0x1a1
main.fetchBundle(0xc42022c8c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, …)
/src/github.com/TykTechnologies/tyk/coprocess_bundle.go:218 +0x185
main.loadBundle(0xc42022c8c0)
/src/github.com/TykTechnologies/tyk/coprocess_bundle.go:331 +0x44a
main.processSpec(0xc42022c8c0, 0xc4203f2320, 0x1, 0xc420488f20, 0xc420488f40, 0xc420488f60, 0xc4203388a0, 0xc420338960, 0xc4203f2320, 0x0)
/src/github.com/TykTechnologies/tyk/api_loader.go:232 +0x91ee
main.loadApps.func1(0xc4204626f0, 0xc4200260e0, 0xc420488f20, 0xc420488f40, 0xc420488f60, 0xc4203388a0, 0xc420338960, 0xc4203389c0, 0xc42022c8c0, 0x1, …)
/src/github.com/TykTechnologies/tyk/api_loader.go:643 +0x10a
created by main.loadApps
/src/github.com/TykTechnologies/tyk/api_loader.go:646 +0x4eb

Hi, is it a HTTPS URL?

You are good at this! :wink:

Just switched to HTTP and it is now working. All within the VPC so not to worried about TLS anyway but any reason why HTTPS does not work?

Thanks!

The scheme was not supported, I’m preparing a patch for it right now.

Ahh, interesting that this did not pick it up and return an error.

https://github.com/TykTechnologies/tyk/blob/master/coprocess_bundle.go#L189

Although I am now getting this error with HTTP. Can wget the file from the containers fine though.

time=“Jul 13 18:46:23” level=error msg=“Couldn’t fetch bundle: HTTP Error” api_id=THEAPPID org_id=THEORGID path=- server_name=“THESERVERNAME” user_id=- user_ip=-

I had it working with a simple nginx server locally so maybe it just doesn’t like s3. :confused:

The “HTTP Error” message indicates it’s not getting HTTP 200 from the target (in this case S3), do you see anything in your logs?

1 Like

I am only seeing that line in the gateway logs unfortunately.

cURL request from the same container returns:

root@gateway-879090359-mzbsn:/opt/tyk-gateway# curl -I http://s3-eu-west-1.amazonaws.com/my-bucket/authentication.zip
HTTP/1.1 200 OK
x-amz-id-2: jGZ3J++3pbbPQQLooLDlPjcPHAMYoa35gCvhdOk8aOsZ1TAlztYu9ZbXDoDBxo34e+IxAyjK5rw=
x-amz-request-id: 98EE7E3F01C051CA
Date: Thu, 13 Jul 2017 19:49:46 GMT
Last-Modified: Thu, 13 Jul 2017 16:37:00 GMT
ETag: "4a2e6937d68ec30b31de6f176d6d6636"
Accept-Ranges: bytes
Content-Type: application/zip
Content-Length: 787
Server: AmazonS3

I’m using env variables to configure the gateway:

TYK_GW_BUNDLEBASEURL=http://s3-eu-west-1.amazonaws.com/my-bucket/

And the apis custom_middleware_bundle property is “authentication”

Are there any other gateway logs that I could check to help?

The custom_middleware_bundle should contain the .zip at the end, we use this internally to identify the file format as we may support other formats (in the future).

The .zip extension was the problem but it was actually because s3 needed it. Without it s3 returns a 403 not authorized error as it does not match the file name in the bucket.

Used tcpdump to view the outgoing http requests and noticed what was wrong. :rolling_eyes:

Good to know that the gateway also need the extension. May be worth updating the docs so the example has the extension as I actually left it off because of the example.

https://tyk.io/docs/customise-tyk/plugins/rich-plugins/python/tutorial-add-demo-plugin-api/

Now on to my next error :laughing:

level=error msg="Couldn't create bundle directory: mkdir middleware/bundles/API_ID-authorization.zip: no such file or directory

Hopefully this will be an easier one for me to sort.

Thanks Matías!

Looks like it is assumed the bundle directory exists within the middleware directory.

https://github.com/TykTechnologies/tyk/blob/master/coprocess_bundle.go#L27

Can’t see anything anything that actually creates the directory if it does not exist. The bundle directory does not exist in the docker image. And I guess it will also cause problems when I actually mount a volume instead of using ephemeral storage.