Ah, great!
It seems your API definition uses bundle, you will need to use bundle.zip.
Also, if the right version of Tyk is being loaded, the following message will show up in the log:
Initializing interpreter, Py_Initialize()
Hmm, I used bundle.zip as well with same result. I tried this demo plugin as well (tyk-plugin-demo-python/manifest.json at master · TykTechnologies/tyk-plugin-demo-python · GitHub). What else could I be missing? I dont see the initializing line in tyk-gateway.log.
How do you start the gateway?
sudo service tyk-gateway start
Try this:
sudo service tyk-gateway stop
Then:
sudo service tyk-gateway-python start
Issue still persists. I have pasted full output of tyk-gateway-python.log below
vagrant@vagrant-ubuntu-trusty-64:/vagrant$ sudo cat /var/log/upstart/tyk-gateway-python.log
[Dec 20 14:34:46] INFO Connection dropped, connecting…
[Dec 20 14:34:46] INFO host-check-mgr: Starting Poller
[Dec 20 14:34:46] INFO main: Setting up analytics normaliser
[Dec 20 14:34:46] INFO main: PIDFile location set to: ./tyk-gateway.pid
[Dec 20 14:34:46] INFO main: Initialising Tyk REST API Endpoints
[Dec 20 14:34:46] INFO main: → Standard listener (http)
[Dec 20 14:34:46] INFO main: Setting up Server
[Dec 20 14:34:46] INFO main: Waiting for zeroconf signal…
[Dec 20 14:34:48] WARN pub-sub: Insecure configuration detected (allowing)!
[Dec 20 14:34:48] INFO pub-sub: Hostname set with dashboard zeroconf signal
[Dec 20 14:34:48] INFO main: Registering node.
[Dec 20 14:34:48] INFO dashboard: Node registered id=67d7fb9b-ff56-4672-734b-8d6579a99c74
[Dec 20 14:34:48] INFO main: Initialising distributed rate limiter
[Dec 20 14:34:48] INFO Starting gateway rate imiter notifications…
[Dec 20 14:34:48] INFO main: Detected 3 APIs
[Dec 20 14:34:48] INFO main: Loading API configurations.
[Dec 20 14:34:48] INFO main: Tracking hostname api_name=Test API2 domain=(no host)
[Dec 20 14:34:48] INFO main: Tracking hostname api_name=Test API3 domain=(no host)
[Dec 20 14:34:48] INFO main: Tracking hostname api_name=Test API domain=(no host)
[Dec 20 14:34:48] INFO main: Loading API api_name=Test API2
[Dec 20 14:34:48] INFO main: Checking security policy: Token api_name=Test API2
[Dec 20 14:34:48] INFO gateway: Loaded: Test API2 api_id=d7516f793fcc4c8e5e4f61bdbd800a06 org_id=585880030640fd19584db29e server_name=-- user_id=-- user_ip=–
[Dec 20 14:34:48] INFO main: Loading API api_name=Test API3
[Dec 20 14:34:48] INFO main: Checking security policy: Token api_name=Test API3
[Dec 20 14:34:48] INFO gateway: Loaded: Test API3 api_id=d2b339ac83544e246443bec559a37aa2 org_id=585880030640fd19584db29e server_name=-- user_id=-- user_ip=–
[Dec 20 14:34:48] INFO main: Loading API api_name=Test API
[Dec 20 14:34:48] INFO main: Checking security policy: Token api_name=Test API
[Dec 20 14:34:48] INFO gateway: Loaded: Test API api_id=538b1f22a3f145715135261dab32e030 org_id=585880030640fd19584db29e server_name=-- user_id=-- user_ip=–
[Dec 20 14:34:49] INFO main: Processed and listening on: /test-api2/{rest:.}
[Dec 20 14:34:49] INFO main: Processed and listening on: /test-api3/{rest:.}
[Dec 20 14:34:49] INFO main: Processed and listening on: /test-api/{rest:.*}
[Dec 20 14:34:49] INFO host-check-mgr: Loading uptime tests…
[Dec 20 14:34:49] INFO main: Initialised API Definitions
[Dec 20 14:34:49] INFO main: Loading policies
[Dec 20 14:34:49] INFO main: Using Policies from Dashboard Service
[Dec 20 14:34:49] INFO policy: Mutex lock acquired… calling
[Dec 20 14:34:49] INFO policy: Calling dashboard service for policy list
[Dec 20 14:34:49] INFO policy: Processing policy list
[Dec 20 14:34:49] INFO main: Gateway started (v2.3.0.33)
[Dec 20 14:34:49] INFO main: → Listening on address: (open interface)
[Dec 20 14:34:49] INFO main: → Listening on port: 8080
[Dec 20 14:34:49] INFO main: → PID: 16683
[Dec 20 14:34:51] INFO pub-sub: Using /opt/tyk-gateway/tyk.conf for configuration
[Dec 20 14:34:51] INFO pub-sub: Configuration request responded.
Have you modified your tyk.conf to enable rich plugins?
The most important setting is:
"coprocess_options": {
"enable_coprocess": true,
}
Cool that knob made it progress a bit more. Now I see below error.
[Dec 20 14:49:05] ERROR main: Couldn’t create bundle directory: mkdir middleware/bundles/d7516f793fcc4c8e5e4f61bdbd800a06-bundle.zip: no such file or directory api_id=d7516f793fcc4c8e5e4f61bdbd800a06 org_id=585880030640fd19584db29e path=- server_name=http://my-tyk-instance.com:9090/ user_id=- user_ip=-
[Dec 20 14:49:05] INFO main: Checking security policy: Token api_name=Test API2
[Dec 20 14:49:05] INFO gateway: Loaded: Test API2 api_id=d7516f793fcc4c8e5e4f61bdbd800a06 org_id=585880030640fd19584db29e server_name=-- user_id=-- user_ip=–
Initialize the bundle directory using:
mkdir -p /opt/tyk-gateway/middleware/bundles
And then restart Tyk.
We’re currently working on updating the docs for this feature.
Best.
Cool that worked.
Question - Does the python plugin execute as a separate process? If so is there an ipc between gateway and plugin during an api call?
1 Like
Awesome.
The Python interpreter runs as part of the Tyk process, in the main thread. It’s basically an embedded Python interpreter.
During an API call, the GIL is acquired and released as soon as the Python plugin returns.
Another question. The current way of enabling plugin seems to be at granularity of single api level. Is there a way to enable plugins at a coarser granularity like group of apis or all apis with exclude list etc?
That’s not possible at this time, we may consider it for future releases.
If you’re looking to share Python code, you could introduce a library as part of your Python installation, and load/call it from different bundles.
The thread here helped me enormously - I was stuck getting my python middleware bundle to load. I was missing the mkdir and tyk-python steps. Now when tyk starts I see it getting the bundle.zip from the web server. However I am now stuck at the next step…
tyk_1 | time=“Jan 17 16:21:44” level=info msg="----> Loading bundle: bundle.zip"
tyk_1 | time=“Jan 17 16:21:44” level=info msg="----> Verifying bundle: bundle.zip"
tyk_1 | time=“Jan 17 16:21:44” level=info msg="----> Bundle is valid, adding to spec: bundle.zip"
tyk_1 | Fatal Python error: drop_gil: GIL is not locked
tyk_1 | SIGABRT: abort
tyk_1 | PC=0x7fce23ee3c37 m=3
tyk_1 | signal arrived during cgo execution
I am using the code exactly as in the github sample
from tyk.decorators import *
from gateway import TykGateway as tyk
@Hook
def MyPreMiddleware(request, session, spec):
request.add_header('myheader', 'myvalue')
return request, sessionYou’re using the Ubuntu version, right?
Here’s my Dockerfile.
FROM ubuntu:14.04
RUN apt-get update
RUN apt-get install -y wget curl ca-certificates apt-transport-https curl
RUN curl https://packagecloud.io/gpg.key | apt-key add -
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10
RUN apt-get update && apt-get install -y ca-certificates
RUN apt-get install -y redis-server
RUN apt-get install -y nginx
RUN apt-get install -y wget
RUN apt-get install -y build-essential
RUN apt-get install -y libluajit-5.1-2
RUN apt-get install -y luarocks
RUN luarocks install lua-cjson
RUN wget https://github.com/google/protobuf/releases/download/v3.1.0/protobuf-python-3.1.0.tar.gz
RUN tar -xvzf protobuf-python-3.1.0.tar.gz
RUN cd protobuf-3.1.0/ && ./configure -prefix=/usr && make && make install
RUN apt-get install -y python3-setuptools
RUN apt-get install -y python3-dev
RUN cd protobuf-3.1.0/python && python3 setup.py build --cpp_implementation && python3 setup.py install --cpp_implementation
RUN apt-get install -y libpython3.4
RUN apt-get install -y python3-pip
RUN pip3 install grpcio
RUN echo "deb https://packagecloud.io/tyk/tyk-gateway/ubuntu/ trusty main" | sudo tee /etc/apt/sources.list.d/tyk_tyk-gateway.list
RUN echo "deb-src https://packagecloud.io/tyk/tyk-gateway/ubuntu/ trusty main" | sudo tee -a /etc/apt/sources.list.d/tyk_tyk-gateway.list
RUN apt-get update
RUN apt-get install -y tyk-gateway=2.3.1.2
RUN wget https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64 \
&& mv jq-linux64 /usr/local/bin/jq \
&& chmod +x /usr/local/bin/jq
COPY ./tyk.conf /opt/tyk-gateway/tyk.conf
COPY ./run_filter.jq /opt/tyk-gateway/run_filter.jq
COPY ./run.sh /opt/tyk-gateway/run.sh
WORKDIR /opt/tyk-gateway
RUN mkdir -p /opt/tyk-gateway/middleware/bundles
RUN chmod +x run.sh
CMD ["./run.sh"]
EXPOSE 8080
if I exec in to the container and run python3 I note that I cant import gateway at all
root@cd8f23b428ea:/opt/tyk-gateway/coprocess/python/tyk# python3
Python 3.4.3 (default, Nov 17 2016, 01:08:31)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from gateway import TykGateway as tyk
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'gateway'
gateway.pyx/c/h are in /opt/tyk-gateway/coprocess/python/tyk
should I be building tyk instead?
go build -tags 'coprocess python'.
Thanks for your help
Chris
1 Like
Do you get the GIL error everytime?
The import error is ok, when Tyk starts, the embedded Python interpreter sets all the paths.
It happened everytime before I posted. Now it doesn’t happen at all. I am not at all sure I know what I did different. I am going to drop it unless I see it again. Thanks so much for your responsiveness
I ran into a similar problem while running from the docker images. I added this - /opt/tyk-gateway/middleware/bundles to the tyk_gateway section in the docker_compose file. That seemed to resolve the problem. Just thought of letting you know , so that you can update your docker_compose files in the next iteration
2 Likes