Can't make Tyk working with python

Hello!

Could you please help me configuring Tyk to make it working with custom python middleware. My problem is that Tyk does not deal with python or bundles at all: I see no entries in log that could relate to python. I’ve learned multiple topics on this forum with no luck.

Here’s my confguration:

docker-compose.yml (I’m using docker compose)

version: '3.3'
services:
  tyk-gateway:
    image: docker.tyk.io/tyk-gateway/tyk-gateway:v4.0.0
    ports:
      - 8081:8080
    networks:
      - tyk
    volumes:
      - ./tyk.standalone.conf:/opt/tyk-gateway/tyk.conf
      - ./apps:/opt/tyk-gateway/apps
      - ./middleware:/opt/tyk-gateway/middleware
      - ./certs:/opt/tyk-gateway/certs
      - ./policies:/opt/tyk-gateway/policies
      - ./bundles:/opt/tyk-gateway/middleware/bundles
    environment:
      - TYK_GW_SECRET=foo
    depends_on:
      - tyk-redis
    extra_hosts:
      - "host.docker.internal:host-gateway"
  tyk-redis:
    image: redis:5.0-alpine
    networks:
      - tyk
    ports:
      - 6379:6379

networks:
  tyk:

tyk.standalone.conf (I’m using a bundle host from someone else’s topic):

{
  "listen_port": 8080,
  "secret": "352d20ee67be67f6340b4c0605b044b7",
  "template_path": "/opt/tyk-gateway/templates",
  "tyk_js_path": "/opt/tyk-gateway/js/tyk.js",
  "middleware_path": "/opt/tyk-gateway/middleware",
  "use_db_app_configs": false,
  "app_path": "/opt/tyk-gateway/apps/",
  "storage": {
    "type": "redis",
    "host": "tyk-redis",
    "port": 6379,
    "username": "",
    "password": "",
    "database": 0,
    "optimisation_max_idle": 2000,
    "optimisation_max_active": 4000
  },
  "enable_analytics": false,
  "analytics_config": {
    "type": "csv",
    "csv_dir": "/tmp",
    "mongo_url": "",
    "mongo_db_name": "",
    "mongo_collection": "",
    "purge_delay": -1,
    "ignored_ips": []
  },
  "health_check": {
    "enable_health_checks": false,
    "health_check_value_timeouts": 60
  },
  "optimisations_use_async_session_write": false,
  "enable_non_transactional_rate_limiter": true,
  "enable_sentinel_rate_limiter": false,
  "enable_redis_rolling_limiter": false,
  "allow_master_keys": false,
  "policies": {
    "policy_source": "file",
    "policy_record_name": "/opt/tyk-gateway/policies/policies.json"
  },
  "hash_keys": true,
  "close_connections": false,
  "http_server_options": {
    "enable_websockets": true
  },
  "allow_insecure_configs": true,
  "coprocess_options": {
    "enable_coprocess": true,
    "python_path_prefix": "/opt/tyk-gateway"
  },
  "enable_bundle_downloader": true,
  "bundle_base_url": "https://storage.cloud.google.com/mlt-sandbox/tyk-plugins/",
  "global_session_lifetime": 100,
  "force_global_session_lifetime": false,
  "max_idle_connections_per_host": 500,
  "enable_jsvm": true,
  "enable_hashed_keys_listing": true,
  "log_level": "debug"
}

API definition:

{
  "name": "My API",
  "slug": "my-api",
  "api_id": "100",
  "org_id": "",
  "auth_configs": {
    "authToken": {
      "auth_header_name": "Authorization"
    }
  },
  "definition": {
    "location": "header",
    "key": "x-api-version"
  },
  "version_data": {
    "not_versioned": true,
    "versions": {
      "Default": {
        "name": "Default",
        "use_extended_paths": true
    }
   }
  },
  "proxy": {
    "listen_path": "/graphql/",
    "target_url": "http://host.docker.internal:8080/graphql",
    "strip_listen_path": true
  },
  "active": true,
  "use_keyless": false,
  "enable_jwt": false,
  "graphql": {
    "enabled": true,
    "execution_mode": "proxyOnly",
    "schema": "type Query { getString: String }",
    "playground": {
      "enabled": false
    },
    "engine": {
        "field_configs": null,
        "data_sources": null
    }
  },
    "custom_middleware": {
        "pre": [
            {
                "name": "PreHook"
            }
        ],
        "post": [
            {
                "name": "PostHook"
            }
        ],
        "post_key_auth": [
            {
                "name": "PostKeyAuth"
            }
        ],
        "response": [
            {
                "name": "ResponseHook"
            }
        ],
        "auth_check": {
            "name": "AuthCheck"
        },
        "driver": "python",
        "id_extractor": {
            "extract_from": "header",
            "extract_with": "value",
            "extractor_config": {
                "header_name": "Authorization"
            }
        }
    },
  "custom_middleware_bundle": "bundle.zip"
}

manifest.json (it is located in ./middleware):

{
    "file_list": [
        "middleware.py"
    ],
    "custom_middleware": {
        "pre": [
            {
                "name": "PreHook"
            }
        ],
        "post": [
            {
                "name": "PostHook"
            }
        ],
        "post_key_auth": [
            {
                "name": "PostKeyAuth"
            }
        ],
        "response": [
            {
                "name": "ResponseHook"
            }
        ],
        "auth_check": {
            "name": "AuthCheck"
        },
        "driver": "python",
        "id_extractor": {
            "extract_from": "header",
            "extract_with": "value",
            "extractor_config": {
                "header_name": "Authorization"
            }
        }
    }
}

I omitted middleware.py as to make the topic shorter, this file does not seem to be used at all.

Thank you for the greate project!

Here’s the log:

 ⠿ Container tyk-gateway-docker-tyk-redis-1    Created                                                                                                                          0.0s
 ⠿ Container tyk-gateway-docker-tyk-gateway-1  Created                                                                                                                          0.0s
Attaching to tyk-gateway-docker-tyk-gateway-1, tyk-gateway-docker-tyk-redis-1
tyk-gateway-docker-tyk-redis-1    | 1:C 29 Mar 2022 17:25:24.633 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
tyk-gateway-docker-tyk-redis-1    | 1:C 29 Mar 2022 17:25:24.633 # Redis version=5.0.14, bits=64, commit=00000000, modified=0, pid=1, just started
tyk-gateway-docker-tyk-redis-1    | 1:C 29 Mar 2022 17:25:24.633 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
tyk-gateway-docker-tyk-redis-1    | 1:M 29 Mar 2022 17:25:24.634 * Running mode=standalone, port=6379.
tyk-gateway-docker-tyk-redis-1    | 1:M 29 Mar 2022 17:25:24.634 # Server initialized
tyk-gateway-docker-tyk-redis-1    | 1:M 29 Mar 2022 17:25:24.634 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
tyk-gateway-docker-tyk-redis-1    | 1:M 29 Mar 2022 17:25:24.634 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
tyk-gateway-docker-tyk-redis-1    | 1:M 29 Mar 2022 17:25:24.634 * DB loaded from disk: 0.000 seconds
tyk-gateway-docker-tyk-redis-1    | 1:M 29 Mar 2022 17:25:24.634 * Ready to accept connections
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=info msg="Tyk API Gateway v4.0.0" prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=warning msg="Insecure configuration allowed" config.allow_insecure_configs=true prefix=checkup
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="No Primary instance found, assuming control" prefix=host-check-mgr
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=info msg="Starting Poller" prefix=host-check-mgr
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="---> Initialising checker" prefix=host-check-mgr
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="[HOST CHECKER] Config:TriggerLimit: 3"
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="[HOST CHECKER] Config:Timeout: ~10"
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="[HOST CHECKER] Config:WorkerPool: 16"
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="Notifier will not work in hybrid mode" prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="[HOST CHECKER] Init complete"
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="---> Starting checker" prefix=host-check-mgr
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="[HOST CHECKER] Starting..."
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="[HOST CHECKER] Check loop started..."
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="[HOST CHECKER] Host reporter started..."
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="---> Checker started." prefix=host-check-mgr
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=info msg="PIDFile location set to: /var/run/tyk/tyk-gateway.pid" prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=warning msg="The control_api_port should be changed for production" prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="Initialising default org store" prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=info msg="Initialising Tyk REST API Endpoints" prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="Connecting to redis cluster"
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="Creating new Redis connection pool"
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=info msg="--> [REDIS] Creating single-node client"
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="Loaded API Endpoints" prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=info msg="--> Standard listener (http)" port=":8080" prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=warning msg="Starting HTTP server on:[::]:8080" prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=info msg="Initialising distributed rate limiter" prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="DRL: Setting node ID: solo-077ed64f-bc5f-4bb0-94d4-9f0966106624|78a115ff9d9a"
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=info msg="Tyk Gateway started (v4.0.0)" prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=info msg="--> Listening on address: (open interface)" prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=info msg="--> Listening on port: 8080" prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=info msg="--> PID: 1" prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=info msg="Starting gateway rate limiter notifications..."
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="Connecting to redis cluster"
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="Creating new Redis connection pool"
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=info msg="--> [REDIS] Creating single-node client"
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="Connecting to redis cluster"
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="Creating new Redis connection pool"
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=info msg="--> [REDIS] Creating single-node client"
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="Default JSVM timeout used: 5s" prefix=jsvm
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=info msg="Loading policies" prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=info msg="Policies found (2 total):" prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg=" - default" prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg=" - test" prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=info msg="Loading API Specification from /opt/tyk-gateway/apps/task-manager-service.json"
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="Checking for transform paths..."
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="Checking for transform paths..."
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=info msg="Detected 1 APIs" prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=info msg="Loading API configurations." prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=info msg="Tracking hostname" api_name="Task Manager API" domain="(no host)" prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=info msg="Initialising Tyk REST API Endpoints" prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="Loaded API Endpoints" prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=info msg="API bind on custom port:0" prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="Initializing API" api_id=100 api_name="Task Manager API" org_id=
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg=Init api_id=100 api_name="Task Manager API" mw=VersionCheck org_id=
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg=Init api_id=100 api_name="Task Manager API" mw=RateCheckMW org_id=
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=info msg="Checking security policy: Token" api_id=100 api_name="Task Manager API" org_id=
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg=Init api_id=100 api_name="Task Manager API" mw=AuthKey org_id=
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg=Init api_id=100 api_name="Task Manager API" mw=KeyExpired org_id=
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg=Init api_id=100 api_name="Task Manager API" mw=AccessRightsCheck org_id=
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg=Init api_id=100 api_name="Task Manager API" mw=GranularAccessMiddleware org_id=
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg=Init api_id=100 api_name="Task Manager API" mw=RateLimitAndQuotaCheck org_id=
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg=Init api_id=100 api_name="Task Manager API" mw=GraphQLMiddleware org_id=
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg=Init api_id=100 api_name="Task Manager API" mw=GraphQLComplexityMiddleware org_id=
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg=Init api_id=100 api_name="Task Manager API" mw=GraphQLGranularAccessMiddleware org_id=
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg=Init api_id=100 api_name="Task Manager API" mw=VersionCheck org_id=
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg=Init api_id=100 api_name="Task Manager API" mw=KeyExpired org_id=
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg=Init api_id=100 api_name="Task Manager API" mw=AccessRightsCheck org_id=
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="Rate limit endpoint is: /graphql/tyk/rate-limits" api_id=100 api_name="Task Manager API" org_id=
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="Setting Listen Path: /graphql/" api_id=100 api_name="Task Manager API" org_id=
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=info msg="API Loaded" api_id=100 api_name="Task Manager API" org_id= prefix=gateway server_name=-- user_id=-- user_ip=--
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="Checker host list" prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=info msg="Loading uptime tests..." prefix=host-check-mgr
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="--- Setting tracking list up" prefix=host-check-mgr
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="Reset initiated" prefix=host-check-mgr
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="[HOST CHECKER] Checker reset queued!"
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=debug msg="Checker host Done" prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=info msg="Initialised API Definitions" prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:25" level=info msg="API reload complete" prefix=main
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:32" level=debug msg="[HOST CHECKER] Host list reset"
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:35" level=debug msg="Error trying to get value:redis: nil"
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:35" level=debug msg="No Primary instance found, assuming control" prefix=host-check-mgr
tyk-gateway-docker-tyk-gateway-1  | time="Mar 29 17:25:35" level=warning msg="Reconnecting storage: Redis is either down or was not configured" prefix=pub-sub

Hi,

If I copy and paste the info you’ve given I don’t get the same debug output you do.

Mine looks more like this

tyk-redis_1    | 1:C 30 Mar 2022 10:51:51.451 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
tyk-redis_1    | 1:C 30 Mar 2022 10:51:51.451 # Redis version=5.0.14, bits=64, commit=00000000, modified=0, pid=1, just started
tyk-redis_1    | 1:C 30 Mar 2022 10:51:51.451 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
tyk-redis_1    | 1:M 30 Mar 2022 10:51:51.452 * Running mode=standalone, port=6379.
tyk-redis_1    | 1:M 30 Mar 2022 10:51:51.452 # Server initialized
tyk-redis_1    | 1:M 30 Mar 2022 10:51:51.452 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
tyk-redis_1    | 1:M 30 Mar 2022 10:51:51.452 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
tyk-redis_1    | 1:M 30 Mar 2022 10:51:51.452 * DB loaded from disk: 0.000 seconds
tyk-redis_1    | 1:M 30 Mar 2022 10:51:51.452 * Ready to accept connections
tyk-gateway_1  | time="Mar 30 10:51:51" level=info msg="Tyk API Gateway v4.0.0" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:51" level=warning msg="Insecure configuration allowed" config.allow_insecure_configs=true prefix=checkup
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="No Primary instance found, assuming control" prefix=host-check-mgr
tyk-gateway_1  | time="Mar 30 10:51:51" level=info msg="Starting Poller" prefix=host-check-mgr
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="---> Initialising checker" prefix=host-check-mgr
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="[HOST CHECKER] Config:TriggerLimit: 3"
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="[HOST CHECKER] Config:Timeout: ~10"
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="[HOST CHECKER] Config:WorkerPool: 12"
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="Notifier will not work in hybrid mode" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="[HOST CHECKER] Init complete"
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="---> Starting checker" prefix=host-check-mgr
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="[HOST CHECKER] Starting..."
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="[HOST CHECKER] Check loop started..."
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="[HOST CHECKER] Host reporter started..."
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="---> Checker started." prefix=host-check-mgr
tyk-gateway_1  | time="Mar 30 10:51:51" level=info msg="PIDFile location set to: /var/run/tyk/tyk-gateway.pid" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:51" level=warning msg="The control_api_port should be changed for production" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="Initialising default org store" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:51" level=info msg="Initialising Tyk REST API Endpoints" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="Connecting to redis cluster"
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="Creating new Redis connection pool"
tyk-gateway_1  | time="Mar 30 10:51:51" level=info msg="--> [REDIS] Creating single-node client"
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="Loaded API Endpoints" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:51" level=info msg="--> Standard listener (http)" port=":8080" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:51" level=warning msg="Starting HTTP server on:[::]:8080" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:51" level=info msg="Initialising distributed rate limiter" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="DRL: Setting node ID: solo-aaaa2242-261d-4e91-a1bc-45e8ae041ac0|2721469e629d"
tyk-gateway_1  | time="Mar 30 10:51:51" level=info msg="Tyk Gateway started (v4.0.0)" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:51" level=info msg="--> Listening on address: (open interface)" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:51" level=info msg="--> Listening on port: 8080" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:51" level=info msg="--> PID: 1" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:51" level=info msg="Starting gateway rate limiter notifications..."
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="Connecting to redis cluster"
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="Creating new Redis connection pool"
tyk-gateway_1  | time="Mar 30 10:51:51" level=info msg="--> [REDIS] Creating single-node client"
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="Connecting to redis cluster"
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="Creating new Redis connection pool"
tyk-gateway_1  | time="Mar 30 10:51:51" level=info msg="--> [REDIS] Creating single-node client"
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="Default JSVM timeout used: 5s" prefix=jsvm
tyk-gateway_1  | time="Mar 30 10:51:51" level=info msg="Loading policies" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:51" level=error msg="Couldn't open policy file: open /opt/tyk-gateway/policies/policies.json: no such file or directory" prefix=policy
tyk-gateway_1  | time="Mar 30 10:51:51" level=info msg="Policies found (0 total):" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:51" level=info msg="Loading API Specification from /opt/tyk-gateway/apps/api1.json"
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="Loading custom PRE-PROCESSOR middleware: PreHook" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="Loading custom POST-PROCESSOR middleware: PostHook" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="Default JSVM timeout used: 5s" prefix=jsvm
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="Checking for transform paths..."
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="Checking for transform paths..."
tyk-gateway_1  | time="Mar 30 10:51:51" level=info msg="Detected 1 APIs" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:51" level=info msg="Loading API configurations." prefix=main
tyk-gateway_1  | time="Mar 30 10:51:51" level=info msg="Tracking hostname" api_name="My API" domain="(no host)" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:51" level=info msg="Initialising Tyk REST API Endpoints" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:51" level=debug msg="Loaded API Endpoints" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:51" level=info msg="API bind on custom port:0" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:51" level=info msg="----> Fetching Bundle: bundle.zip" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:52" level=debug msg="----> Saving Bundle: bundle.zip" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:52" level=info msg="----> Loading bundle: bundle.zip" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:52" level=error msg="Couldn't load bundle: open /opt/tyk-gateway/middleware/bundles/100_c343271e0935000c0ea41f8d9822015c/manifest.json: no such file or directory" api_id=100 org_id= path=- prefix=main server_name="http://host.docker.internal:8080/graphql" user_id=- user_ip=-
tyk-gateway_1  | time="Mar 30 10:51:52" level=debug msg="Initializing API" api_id=100 api_name="My API" org_id=
tyk-gateway_1  | time="Mar 30 10:51:52" level=debug msg="Loading custom PRE-PROCESSOR middleware: PreHook" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:52" level=debug msg="Loading custom POST-PROCESSOR middleware: PostHook" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:52" level=debug msg="Registering coprocess middleware, hook name: PreHookhook type: Pre, driver: python" api_id=100 api_name="My API" org_id= prefix=coprocess
tyk-gateway_1  | time="Mar 30 10:51:52" level=error msg="Driver 'python' isn't loaded" prefix=coprocess
tyk-gateway_1  | time="Mar 30 10:51:52" level=debug msg=Init api_id=100 api_name="My API" mw=VersionCheck org_id=
tyk-gateway_1  | time="Mar 30 10:51:52" level=debug msg=Init api_id=100 api_name="My API" mw=RateCheckMW org_id=
tyk-gateway_1  | time="Mar 30 10:51:52" level=info msg="Checking security policy: Token" api_id=100 api_name="My API" org_id=
tyk-gateway_1  | time="Mar 30 10:51:52" level=debug msg=Init api_id=100 api_name="My API" mw=AuthKey org_id=
tyk-gateway_1  | time="Mar 30 10:51:52" level=debug msg="Registering coprocess middleware, hook name: PostKeyAuthhook type: Pre, driver: python" api_id=100 api_name="My API" org_id= prefix=coprocess
tyk-gateway_1  | time="Mar 30 10:51:52" level=error msg="Driver 'python' isn't loaded" prefix=coprocess
tyk-gateway_1  | time="Mar 30 10:51:52" level=debug msg=Init api_id=100 api_name="My API" mw=KeyExpired org_id=
tyk-gateway_1  | time="Mar 30 10:51:52" level=debug msg=Init api_id=100 api_name="My API" mw=AccessRightsCheck org_id=
tyk-gateway_1  | time="Mar 30 10:51:52" level=debug msg=Init api_id=100 api_name="My API" mw=GranularAccessMiddleware org_id=
tyk-gateway_1  | time="Mar 30 10:51:52" level=debug msg=Init api_id=100 api_name="My API" mw=RateLimitAndQuotaCheck org_id=
tyk-gateway_1  | time="Mar 30 10:51:52" level=debug msg=Init api_id=100 api_name="My API" mw=GraphQLMiddleware org_id=
tyk-gateway_1  | time="Mar 30 10:51:52" level=debug msg=Init api_id=100 api_name="My API" mw=GraphQLComplexityMiddleware org_id=
tyk-gateway_1  | time="Mar 30 10:51:52" level=debug msg=Init api_id=100 api_name="My API" mw=GraphQLGranularAccessMiddleware org_id=
tyk-gateway_1  | time="Mar 30 10:51:52" level=debug msg="Registering coprocess middleware, hook name: PostHookhook type: Post, driver: python" api_id=100 api_name="My API" org_id= prefix=coprocess
tyk-gateway_1  | time="Mar 30 10:51:52" level=error msg="Driver 'python' isn't loaded" prefix=coprocess
tyk-gateway_1  | time="Mar 30 10:51:52" level=debug msg=Init api_id=100 api_name="My API" mw=VersionCheck org_id=
tyk-gateway_1  | time="Mar 30 10:51:52" level=debug msg=Init api_id=100 api_name="My API" mw=KeyExpired org_id=
tyk-gateway_1  | time="Mar 30 10:51:52" level=debug msg=Init api_id=100 api_name="My API" mw=AccessRightsCheck org_id=
tyk-gateway_1  | time="Mar 30 10:51:52" level=debug msg="Rate limit endpoint is: /graphql/tyk/rate-limits" api_id=100 api_name="My API" org_id=
tyk-gateway_1  | time="Mar 30 10:51:52" level=debug msg="Setting Listen Path: /graphql/" api_id=100 api_name="My API" org_id=
tyk-gateway_1  | time="Mar 30 10:51:52" level=info msg="API Loaded" api_id=100 api_name="My API" org_id= prefix=gateway server_name=-- user_id=-- user_ip=--
tyk-gateway_1  | time="Mar 30 10:51:52" level=debug msg="Checker host list" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:52" level=info msg="Loading uptime tests..." prefix=host-check-mgr
tyk-gateway_1  | time="Mar 30 10:51:52" level=debug msg="--- Setting tracking list up" prefix=host-check-mgr
tyk-gateway_1  | time="Mar 30 10:51:52" level=debug msg="Reset initiated" prefix=host-check-mgr
tyk-gateway_1  | time="Mar 30 10:51:52" level=debug msg="[HOST CHECKER] Checker reset queued!"
tyk-gateway_1  | time="Mar 30 10:51:52" level=debug msg="Checker host Done" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:52" level=info msg="Initialised API Definitions" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:52" level=info msg="API reload complete" prefix=main
tyk-gateway_1  | time="Mar 30 10:51:58" level=debug msg="[HOST CHECKER] Host list reset"
tyk-gateway_1  | time="Mar 30 10:52:01" level=debug msg="Error trying to get value:redis: nil"
tyk-gateway_1  | time="Mar 30 10:52:01" level=debug msg="No Primary instance found, assuming control" prefix=host-check-mgr
tyk-gateway_1  | time="Mar 30 10:52:01" level=warning msg="Reconnecting storage: Redis is either down or was not configured" prefix=pub-sub

I’m very confused by the difference between them. I would expect to see the python errors in your output that are present in mine. Are you sure you’re using exactly what you’ve copied into this thread?

Please check and get back to us, but there are a couple of issues with your config which need to be addressed as well.

Tyk needs to be able to download the zipped bundle from the URL specified in bundle_base_url. That isn’t the case for "https://storage.cloud.google.com/mlt-sandbox/tyk-plugins/"
What I did was run a python webserver on my local machine and specified that in my tyk.standalone.conf. Exactly how that is done depends on what you’re running locally but for linux and python 3.9 I put the bundle into ~/plugins and ran python3 -m http.server --bind <my IP address> --directory $HOME/plugins/ 8000

Then I used http://<my IP address>:8000 as my bundle_base_url. Then when I ran docker-compose up I got this: (I’ve skipped some messages for brevity)

tyk-gateway_1  | time="Mar 30 11:00:20" level=debug msg="Checking for transform paths..."
tyk-gateway_1  | time="Mar 30 11:00:20" level=info msg="Detected 1 APIs" prefix=main
tyk-gateway_1  | time="Mar 30 11:00:20" level=info msg="Loading API configurations." prefix=main
tyk-gateway_1  | time="Mar 30 11:00:20" level=info msg="Tracking hostname" api_name="My API" domain="(no host)" prefix=main
tyk-gateway_1  | time="Mar 30 11:00:20" level=info msg="Initialising Tyk REST API Endpoints" prefix=main
tyk-gateway_1  | time="Mar 30 11:00:20" level=debug msg="Loaded API Endpoints" prefix=main
tyk-gateway_1  | time="Mar 30 11:00:20" level=info msg="API bind on custom port:0" prefix=main
tyk-gateway_1  | time="Mar 30 11:00:20" level=info msg="----> Fetching Bundle: python.zip" prefix=main
tyk-gateway_1  | time="Mar 30 11:00:20" level=debug msg="----> Saving Bundle: python.zip" prefix=main
tyk-gateway_1  | time="Mar 30 11:00:20" level=info msg="----> Loading bundle: python.zip" prefix=main
tyk-gateway_1  | time="Mar 30 11:00:20" level=info msg="----> Verifying bundle: python.zip" prefix=main
tyk-gateway_1  | time="Mar 30 11:00:20" level=info msg="----> Bundle is valid, adding to spec: python.zip" prefix=main
tyk-gateway_1  | time="Mar 30 11:00:20" level=debug msg="Found python-config binary: 3.7 (/usr/bin/python3.7-config)" prefix=dlpython
tyk-gateway_1  | time="Mar 30 11:00:20" level=debug msg="Using latest Python version" prefix=dlpython
tyk-gateway_1  | time="Mar 30 11:00:20" level=debug msg="Selected Python configuration path: /usr/bin/python3.7-config" prefix=dlpython
tyk-gateway_1  | time="Mar 30 11:00:20" level=debug msg="Selected Python library path: /usr/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.so" prefix=dlpython
tyk-gateway_1  | time="Mar 30 11:00:20" level=info msg="Python version '3.7' loaded" prefix=coprocess
tyk-gateway_1  | time="Mar 30 11:00:20" level=info msg="Initializing dispatcher" prefix=python
tyk-gateway_1  | time="Mar 30 11:00:20" level=info msg="Python dispatcher was initialized" prefix=coprocess
tyk-gateway_1  | time="Mar 30 11:00:20" level=info msg="Loading module: '100_b72acdf7c0a858927299fccd62fb0d0d'" prefix=python
tyk-gateway_1  | time="Mar 30 11:00:20" level=debug msg="Loading hook 'Peek' (100_b72acdf7c0a858927299fccd62fb0d0d)" prefix=python
tyk-gateway_1  | time="Mar 30 11:00:20" level=debug msg="Initializing API" api_id=100 api_name="My API" org_id=
tyk-gateway_1  | time="Mar 30 11:00:20" level=debug msg="Loading custom PRE-PROCESSOR middleware: Peek" prefix=main

Note that my bundle is called python.zip rather than bundle.zip but that’s not important. This shows the bundle is downloaded and python is initialised. I am still using your configs at this stage with only the two changes. One being the bundle_base_url and the other being the name of the bundle in the api definition.

Have you had a look at the tutorial on creating a python plugin at Custom Authentication With a Python Plugin? It steps you through this process with some helpful details.

Cheers,
Pete

1 Like

Hello @Pete, thanks for the response.

I accurately reproduced all the steps from the article you have mentioned. I created bundle.zip file, created a trivial python-powered HTTP server hosting it and then tried again.

I don’t know exactly what was wrong in me prev. setup, but now my Python middleware is loading and working!

Thank you, team, for your helpful project.