Tyk taking too much time for gRPC call

Hi,
We initially integrated tyk with go plugin.
Due to this below issue we integrated with python plugin.
https://github.com/TykTechnologies/tyk/issues/5141

Once we did this we ran perf to achieve 1000 RPS which we were able to achieve with go.

During the perf run for python RPS drastically came down to 100 RPS.

after that we observed that tyk is spending too much time for gRPC calls.

In plugin code we are calling another endpoint for authorization header token validation.
We observed that service log and it is working as expected and it is responding back very quick.

this is our API definition file

{
    "name": "v1 post delay-19 #komodo",
    "slug": "v1-post-delay-19",
    "listen_port": 0,
    "org_id": "64b68c280255e10001cafc69",
    "api_id": "64b68c280255e10001cafc081",
    "use_keyless": false,
    "use_oauth2": false,
    "external_oauth": {
        "enabled": false,
        "providers": []
    },
    "use_go_plugin_auth": false,
    "enable_coprocess_auth": true,
    "custom_plugin_auth_enabled": false,
    "version_data": {
        "not_versioned": true,
        "default_version": "Default",
        "versions": {
            "Default": {
                "name": "Default",
                "expires": "",
                "paths": {
                    "ignored": [],
                    "white_list": [],
                    "black_list": []
                },
                "use_extended_paths": true,
                "extended_paths": {
                    "hard_timeouts": [
                        {
                            "disabled": false,
                            "path": "/",
                            "method": "POST",
                            "timeout": 5
                        }
                    ],
                    "url_rewrites": [],
                    "persist_graphql": []
                }
            }
        }
    },
    "proxy": {
        "preserve_host_header": false,
        "listen_path": "/v1/perf/19",
        "target_url": "http://test-application-stage.test.svc.cluster.local:9150/v1/perf/19",
        "disable_strip_slash": true,
        "strip_listen_path": true,
        "enable_load_balancing": false,
        "target_list": [],
        "check_host_against_uptime_tests": false,
        "transport": {
            "ssl_insecure_skip_verify": false,
            "ssl_ciphers": [],
            "ssl_min_version": 0,
            "ssl_max_version": 0,
            "ssl_force_common_name_check": false,
            "proxy_url": ""
        }
    },
    "disable_rate_limit": true,
    "disable_quota": true,
    "custom_middleware": {},
    "custom_middleware_bundle": "bundle.zip",
    "custom_middleware_bundle_disabled": false,
    "session_lifetime": 0,
    "active": true,
    "internal": false,
    "strip_auth_data": false,
    "enable_detailed_recording": true
}

logs are attached:

time="Sep 24 16:04:55" level=info msg="Call to phoenix successful" prefix=python
time="Sep 24 16:04:55" level=debug msg="gRPC request processing took" api_id=64b68c280255e10001cafc0819 api_name="v1 post delay-19 #komodo" ms=14307.503753 mw=CoProcessMiddleware org_id=64b68c280255e10001cafc69 origin=10.250.3.114 path=/v1/perf/19
time="Sep 24 16:04:55" level=debug msg=Finished api_id=64b68c280255e10001cafc0819 api_name="v1 post delay-19 #komodo" code=200 mw=CoProcessMiddleware ns=14309777645 org_id=64b68c280255e10001cafc69 origin=10.250.3.114 path=/v1/perf/19
time="Sep 24 16:04:55" level=debug msg=Started api_id=64b68c280255e10001cafc0819 api_name="v1 post delay-19 #komodo" key=-- mw=KeyExpired org_id=64b68c280255e10001cafc69 origin=10.250.3.114 path=/v1/perf/19 ts=1695571495167727385
time="Sep 24 16:04:55" level=debug msg=Finished api_id=64b68c280255e10001cafc0819 api_name="v1 post delay-19 #komodo" code=200 key=-- mw=KeyExpired ns=21079 org_id=64b68c280255e10001cafc69 origin=10.250.3.114 path=/v1/perf/19
time="Sep 24 16:04:55" level=debug msg=Started api_id=64b68c280255e10001cafc0819 api_name="v1 post delay-19 #komodo" key=-- mw=AccessRightsCheck org_id=64b68c280255e10001cafc69 origin=10.250.3.114 path=/v1/perf/19 ts=1695571495167769559
time="Sep 24 16:04:55" level=debug msg=Finished api_id=64b68c280255e10001cafc0819 api_name="v1 post delay-19 #komodo" code=200 key=-- mw=AccessRightsCheck ns=10099 org_id=64b68c280255e10001cafc69 origin=10.250.3.114 path=/v1/perf/19
time="Sep 24 16:04:55" level=debug msg=Started api_id=64b68c280255e10001cafc0819 api_name="v1 post delay-19 #komodo" key=-- mw=GranularAccessMiddleware org_id=64b68c280255e10001cafc69 origin=10.250.3.114 path=/v1/perf/19 ts=1695571495167796358
time="Sep 24 16:04:55" level=debug msg=Finished api_id=64b68c280255e10001cafc0819 api_name="v1 post delay-19 #komodo" code=200 key=-- mw=GranularAccessMiddleware ns=9519 org_id=64b68c280255e10001cafc69 origin=10.250.3.114 path=/v1/perf/19
time="Sep 24 16:04:55" level=debug msg="Started proxy"
time="Sep 24 16:04:55" level=debug msg="Stripping proxy listen path: /v1/perf/19"
time="Sep 24 16:04:55" level=debug msg="Upstream path is: /"
time="Sep 24 16:04:55" level=debug msg=Started api_id=64b68c280255e10001cafc0819 api_name="v1 post delay-19 #komodo" mw=ReverseProxy org_id=64b68c280255e10001cafc69 ts=1695571495167842290
time="Sep 24 16:04:55" level=debug msg="Upstream request URL: /?delay=1000" api_id=64b68c280255e10001cafc0819 api_name="v1 post delay-19 #komodo" mw=ReverseProxy org_id=64b68c280255e10001cafc69
time="Sep 24 16:04:55" level=debug msg="Outbound request URL: http://test-application-stage.test.svc.cluster.local:9150/v1/perf/19?delay=1000" api_id=64b68c280255e10001cafc0819 api_name="v1 post delay-19 #komodo" mw=ReverseProxy org_id=64b68c280255e10001cafc69
time="Sep 24 16:04:55" level=debug msg=Finished api_id=64b68c280255e10001cafc0842 api_name="v1 post delay-42 #komodo" mw=ReverseProxy ns=1004748340 org_id=64b68c280255e10001cafc69
time="Sep 24 16:04:55" level=debug msg="Upstream request took (ms): 1004.79966"
time="Sep 24 16:04:55" level=debug msg="Done proxy"

Is there anything we are missing while integrating plugin with python.

Thanks in advance.

Hi @karthikbhat003,

Welcome to the community :partying_face:

Any chance this has to do with latency of the gRPC call?
Or the processing time at the gRPC server?

Hi @Ubong ,
Apologies for the late reply.

The issue reported will be fixed in next release cycle that should solve the issue for our use case.

With respect to python plugin issue,
we couldn’t find more info with respect to the that gRPC latency log.

Do you need any other configuration or any log to look into it further.?