Custom Python Plugin Response Hook Programming

Support
, ,
1

I’m trying to modify the response from upstream using Python Plugin Bundle. The bundle is working and the response hook as defined in the bundle is working.

But I’m facing difficulty in understanding the various Hook(s) definition e.g. object type in the parameters, even by using type() in Python, I do not know what is inside the class. Im searching for the library documentation but can’t seems to find it anywhere.

Can someone help me to enlighten me where can I find the Tyk Python libraries documentation or how to setup any IDE for easier development. Thanks.

2

Hi,

I’m not sure what you’ve seen and what you haven’t so forgive me if I point out links that you’ve already looked at.

The python plugin development documentation starts at Python

And the source for the coprocess all lives under tyk/coprocess at master · TykTechnologies/tyk · GitHub

You will find python details under both tyk/coprocess/bindings/python at master · TykTechnologies/tyk · GitHub and tyk/coprocess/python at master · TykTechnologies/tyk · GitHub

These source directories are also present in your install under /opt/tyk-gateway/coprocess.

I hope this helps,
Pete

3

Thanks @Pete.

I had successfully setup the python rich plugin and was able to read and manipulate the response.body value from upstream.

Following the online documentation, I was able to inject the response header but having difficulty changing the response body.

I tried to write the changes back to the response.body but the API caller is still getting the non-modified body. Tried to write to both response.body and response.raw_body but ended up the API caller did not receive any body.

So question is, how do I actually overwrite the body? To response.body or response.raw_body or somewhere else? What is correct way to modify the response body so that the API caller gets the modified response body.

Bundle Manifest:

{
    "file_list": [
        "middleware.py",
        "fieldMasking.py"
    ],
    "custom_middleware": {
        "response": [
            {
                "name": "PIIMasking"
            }
        ],
        "driver": "python"
    }
}

API Defintion:

{
  "name": "Get 5 PII Data",
  "slug": "get-5-pii-data",
  "api_id": "50",
  "org_id": "1",
  "use_keyless": true,
  "custom_middleware_bundle": "bundle.zip",
  "cusom_middleware": {
     "response": [
         {
            "name": "PIIMasking"
         }
     ],
     "driver" : "python"
  },
  "auth_configs": {
     "authToken": {
        "auth_header_name": "Authorization"
     }
  },
  "definition": {
    "location": "header",
    "key": "x-api-version"
  },
  "version_data": {
    "not_versioned": true,
    "versions": {
       "Default": {
         "name": "Default",
         "use_extended_paths" :true
       }
     }
  },
  "proxy": {
    "listen_path": "/api/get5PIIData",
    "target_url": "http://host.docker.internal:5000/api/get5PIIData",
    "strip_listen_path": true
  },
  "active": true
}

Plugin Code:

from tyk.decorators import *
from gateway import TykGateway as tyk
from time import time
import json
from fieldMasking import FieldMasking

@Hook
def PIIMasking(request, response, session, metadata, spec):
    tyk.log("ResponseHook is called", "info")
    # In this hook we have access to the response object, to inspect it, uncomment the following line:
    print(response)
    tyk.log("PIIMasking: upstream returned body {0}".format(response), "info")
    tyk.log("ResponseHook: upstream returned {0}".format(response.status_code), "info")
    response.headers["injectedkey"] = "injectedvalue"

    tyk.log("PIIMasking: Printing response...{0} ".format(dir(response)), "info")
    tyk.log("PIIMasking: body type {0}.".format(type(response.body)),"info")
    tyk.log("PIIMasking: body {0}.".format(response.body),"info")
    tyk.log("PIIMasking: raw body {0}.".format(response.raw_body),"info")

    json_body = json.loads(response.body)
    tyk.log("PIIMasking: json_body type {0}".format(type(json_body)),"info")
    tyk.log("PIIMasking: json_body value {0}".format(json_body),"info")

    fieldMasking = FieldMasking()

    addressLine1FieldStr = "addressLine1"

    for key in json_body:
      tyk.log("Key is: {0}".format(key),"info")
      if key == addressLine1FieldStr:
         json_body[key] = fieldMasking.maskAddress(json_body[key])
         tyk.log("PIIMasking: json_body={0}".format(json_body[key]),"info")

    response.body = json.dumps(json_body, indent=2)
    #response.raw_body = bytes(response.body,encoding='utf-8')

    tyk.log("PIIMasking: response.body- {0}.".format(response.body),"info")
    tyk.log("PIIMasking: response.rawbody- {0}.".format(response.raw_body),"info")
    return response

Logs:

tyk-gateway-docker-tyk-gateway-1  | time="Dec 15 02:58:31" level=debug msg="Outbound request URL: http://host.docker.internal:5000/api/get5PIIData" api_id=50 api_name="Get 5 PII Data" mw=ReverseProxy org_id=1
tyk-gateway-docker-tyk-gateway-1  | time="Dec 15 02:58:31" level=debug msg="Response hook 'CoProcessMiddleware' is called" prefix=coprocess
tyk-gateway-docker-tyk-gateway-1  | time="Dec 15 02:58:31" level=info msg="ResponseHook is called" prefix=python
tyk-gateway-docker-tyk-gateway-1  | time="Dec 15 02:58:31" level=info msg="PIIMasking: upstream returned body status_code: 200
tyk-gateway-docker-tyk-gateway-1  | raw_body: "{\n  \"addressLine1\": \"888 Huat Huat Place\", \n  \"advisorID\": \"31652154752894192\", \n  \"creditCard\": \"2222222222222222\", \n  \"customerID\": \"111111111111111111\", \n  \"mobile\": \"(292)92929292\"\n}\n"
tyk-gateway-docker-tyk-gateway-1  | body: "{\n  \"addressLine1\": \"888 Huat Huat Place\", \n  \"advisorID\": \"31652154752894192\", \n  \"creditCard\": \"2222222222222222\", \n  \"customerID\": \"111111111111111111\", \n  \"mobile\": \"(292)92929292\"\n}\n"
tyk-gateway-docker-tyk-gateway-1  | headers {
tyk-gateway-docker-tyk-gateway-1  |   key: "Content-Length"
tyk-gateway-docker-tyk-gateway-1  |   value: "191"
tyk-gateway-docker-tyk-gateway-1  | }
tyk-gateway-docker-tyk-gateway-1  | headers {
tyk-gateway-docker-tyk-gateway-1  |   key: "Content-Type"
tyk-gateway-docker-tyk-gateway-1  |   value: "application/json"
tyk-gateway-docker-tyk-gateway-1  | }
tyk-gateway-docker-tyk-gateway-1  | headers {
tyk-gateway-docker-tyk-gateway-1  |   key: "Date"
tyk-gateway-docker-tyk-gateway-1  |   value: "Wed, 15 Dec 2021 02:58:31 GMT"
tyk-gateway-docker-tyk-gateway-1  | }
tyk-gateway-docker-tyk-gateway-1  | headers {
tyk-gateway-docker-tyk-gateway-1  |   key: "Server"
tyk-gateway-docker-tyk-gateway-1  |   value: "Werkzeug/2.0.2 Python/3.7.4"
tyk-gateway-docker-tyk-gateway-1  | }
tyk-gateway-docker-tyk-gateway-1  | " prefix=python
tyk-gateway-docker-tyk-gateway-1  | time="Dec 15 02:58:31" level=info msg="ResponseHook: upstream returned 200" prefix=python
tyk-gateway-docker-tyk-gateway-1  | time="Dec 15 02:58:31" level=info msg="PIIMasking: Printing response...['ByteSize', 'Clear', 'ClearExtension', 'ClearField', 'CopyFrom', 'DESCRIPTOR', 'DiscardUnknownFields', 'Extensions', 'FindInitializationErrors', 'FromString', 'HasExtension', 'HasField', 'HeadersEntry', 'IsInitialized', 'ListFields', 'MergeFrom', 'MergeFromString', 'ParseFromString', 'RegisterExtension', 'SerializePartialToString', 'SerializeToString', 'SetInParent', 'UnknownFields', 'WhichOneof', '_CheckCalledFromGeneratedFile', '_SetListener', '__class__', '__deepcopy__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__slots__', '__str__', '__subclasshook__', '__unicode__', '_extensions_by_name', '_extensions_by_number', 'body', 'headers', 'raw_body', 'status_code'] " prefix=python
tyk-gateway-docker-tyk-gateway-1  | time="Dec 15 02:58:31" level=info msg="PIIMasking: body type <class 'str'>." prefix=python
tyk-gateway-docker-tyk-gateway-1  | time="Dec 15 02:58:31" level=info msg="PIIMasking: body {
tyk-gateway-docker-tyk-gateway-1  |   "addressLine1": "888 Huat Huat Place", 
tyk-gateway-docker-tyk-gateway-1  |   "advisorID": "31652154752894192", 
tyk-gateway-docker-tyk-gateway-1  |   "creditCard": "2222222222222222", 
tyk-gateway-docker-tyk-gateway-1  |   "customerID": "111111111111111111", 
tyk-gateway-docker-tyk-gateway-1  |   "mobile": "(292)92929292"
tyk-gateway-docker-tyk-gateway-1  | }
tyk-gateway-docker-tyk-gateway-1  | ." prefix=python
tyk-gateway-docker-tyk-gateway-1  | time="Dec 15 02:58:31" level=info msg="PIIMasking: raw body b'{\n  "addressLine1": "888 Huat Huat Place", \n  "advisorID": "31652154752894192", \n  "creditCard": "2222222222222222", \n  "customerID": "111111111111111111", \n  "mobile": "(292)92929292"\n}\n'." prefix=python
tyk-gateway-docker-tyk-gateway-1  | time="Dec 15 02:58:31" level=info msg="PIIMasking: json_body type <class 'dict'>" prefix=python
tyk-gateway-docker-tyk-gateway-1  | time="Dec 15 02:58:31" level=info msg="PIIMasking: json_body value {'addressLine1': '888 Huat Huat Place', 'advisorDSID': '31652154752894192', 'creditCard': '2222222222222222', 'customerDSID': '111111111111111111', 'mobile': '(292)92929292'}" prefix=python
tyk-gateway-docker-tyk-gateway-1  | time="Dec 15 02:58:31" level=info msg="Key is: addressLine1" prefix=python
tyk-gateway-docker-tyk-gateway-1  | time="Dec 15 02:58:31" level=info msg="FieldMasking:maskAddress 888 Huat Huat Place." prefix=python
tyk-gateway-docker-tyk-gateway-1  | time="Dec 15 02:58:31" level=info msg="FieldMasking:maskAddress after mask *** **** **** *****." prefix=python
tyk-gateway-docker-tyk-gateway-1  | time="Dec 15 02:58:31" level=info msg="PIIMasking: json_body=*** **** **** *****" prefix=python
tyk-gateway-docker-tyk-gateway-1  | time="Dec 15 02:58:31" level=info msg="Key is: advisorID" prefix=python
tyk-gateway-docker-tyk-gateway-1  | time="Dec 15 02:58:31" level=info msg="Key is: creditCard" prefix=python
tyk-gateway-docker-tyk-gateway-1  | time="Dec 15 02:58:31" level=info msg="Key is: customerID" prefix=python
tyk-gateway-docker-tyk-gateway-1  | time="Dec 15 02:58:31" level=info msg="Key is: mobile" prefix=python
tyk-gateway-docker-tyk-gateway-1  | time="Dec 15 02:58:31" level=info msg="PIIMasking: response.body- {
tyk-gateway-docker-tyk-gateway-1  |   "addressLine1": "*** **** **** *****",
tyk-gateway-docker-tyk-gateway-1  |   "advisorID": "31652154752894192",
tyk-gateway-docker-tyk-gateway-1  |   "creditCard": "2222222222222222",
tyk-gateway-docker-tyk-gateway-1  |   "customerID": "111111111111111111",
tyk-gateway-docker-tyk-gateway-1  |   "mobile": "(292)92929292"
tyk-gateway-docker-tyk-gateway-1  | }." prefix=python
tyk-gateway-docker-tyk-gateway-1  | time="Dec 15 02:58:31" level=info msg="PIIMasking: response.rawbody- b'{\n  "addressLine1": "888 Huat Huat Place", \n  "advisorID": "31652154752894192", \n  "creditCard": "2222222222222222", \n  "customerID": "111111111111111111", \n  "mobile": "(292)92929292"\n}\n'." prefix=python

Thanks.

4

Hi @kamikazane ,

Could you let us know what version of the gateway you’re using? I’ve tested the code below (using a simple rest API pointed at httpbin.org and it gives the right results with gateway 3.0.8

from tyk.decorators import *
from gateway import TykGateway as tyk

@Hook
def ChangeResponseBody(request, response, session, metadata, spec):
        tyk.log("ResponseHook is called", "info")
        tyk.log("ResponseHook: upstream returned {0}".format(response.status_code), "info")
        tyk.log(f"response object is: {response}", "info")
        response.headers["injectedkey"] = "injectedvalue"
        response.raw_body = b"{Pete Woz Here}"
        response.headers["Content-Length"] = str(len(response.raw_body))
        return response

This was my manifest:

{
  "file_list": [
    "responsBody.py"
  ],
  "custom_middleware": {
    "driver": "python",
    "response": [
      {
        "name": "ChangeResponseBody"
      }
    ]
  }
}

An example run

$ curl -i https://10.0.0.21:5001/response/
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Length: 15
Content-Type: application/json
Date: Thu, 16 Dec 2021 14:46:18 GMT
Injectedkey: injectedvalue
Server: gunicorn/19.9.0
X-Ratelimit-Limit: 0
X-Ratelimit-Remaining: 0
X-Ratelimit-Reset: 0

{Pete Woz Here}
5

Hi Pete,

 I'm using v3.2.1. I manage to get it to work after digging in the forum and found out that I have to write to response.raw_body instead of response.body. 

 But thanks to your code sample, I solve another error where I'm not able to show the returned json payload in browser and kept getting dropped connection error.

response.headers["Content-Length"] = str(len(response.raw_body))

I have some questions which I hope u can help answer:

  1. if I use a python plugin bundle in 3 x API, does it means the bundle is copied for all the 3 x API during initial loading? Cos the log seems to suggest so.

  2. I have different response hook and each API Definition may use different ones. By using manifest.json it seems that all hook define in the manifest will be used by the API, what is correct way to have different hook for different API with the same bundle? Or we just have to add in multiple bundle into the API definition?

    Thanks.

6

Hi @kamikazane ,

if I use a python plugin bundle in 3 x API, does it means the bundle is copied for all the 3 x API during initial loading? Cos the log seems to suggest so.

I don’t think the bundle is downloaded for each API that uses it. The bundle is stored by each gateway and loaded into the APIs that refer to it using the bundlid.

I have different response hook and each API Definition may use different ones. By using manifest.json it seems that all hook define in the manifest will be used by the API, what is correct way to have different hook for different API with the same bundle? Or we just have to add in multiple bundle into the API definition?

If you need a different manifest, it will need to be in a different bundle.

Cheers,
Pete