gRPC Max Msg size limit hit

Support
1

Hi,

We’re using TYK GW CE 2.7.4. It appears in this version, the grpc message size limit is set to the default value at 4MB.

Is it possible to configure TYK GW to up this limit? If so, how?

We’re getting the following from tyk-gateway error log:

time=“Apr 10 05:04:45” level=error msg=“rpc error: code = 8 desc = grpc: received message larger than max (5097416 vs. 4194304)”

time=“Apr 10 05:04:45” level=error msg=“Dispatch error” error=“rpc error: code = 8 desc = grpc: received message larger than max (5097416 vs. 4194304)”

2

Hi Jeffrey, I’m investigating this issue.

Will keep you updated.

Regards.

3

@matiasb Thanks for the prompt response!

4

@matiasb One follow up question:
we want to capture this condition (payload exceed size limit) and provide a more user-friendly error message in the API response. Any suggestion on how can we do that w/o incurring extra API response time?

5

I’ve created a patch that makes this parameter configurable, we’ll test it a bit and keep you updated about its availability.

About payload exceeding size limit, could you check the HTTP request body length or it your use case requires it to be integrated with gRPC in some other way?

Regards.

6

Thanks for the quick turn-around.

  • Is there an installable patch for ver 2.7.4 which we’re using?
  • Is there a plan to back port the fix into existing releases (e.g. 2.7) or will it only be in the up-coming release?

Thanks for the tip about checking the size. I think we could check the request body length.

7

I’m having some issue with the logic to check msg size.

Currently, I have a working API with a “Pre” hook attached to it to do some validation. I added this new logic to check for message size there but it looks like for this case, the “Dispatch error” happens before the “Pre” hook gets executed.

Any ideas where to insert this new check for msg size?
Thanks!

8

BTW, I filed this new bug: gRPC: can only send message no larger than 2MB eventhough grpc limit is 4MB · Issue #2208 · TykTechnologies/tyk · GitHub
It’s different from gRPC fails when sending large messages · Issue #2203 · TykTechnologies/tyk · GitHub

9

@matiasb and team,

Do we have the complete fix for https://github.com/TykTechnologies/tyk/pull/2204 in the Tyk Gateway CE listed under el/7/tyk-gateway-2.9.4.1-1.x86_64.rpm - tyk/tyk-gateway · packagecloud?

I deployed this version and found that the custom limits defined in tyk.conf are not enforced correctly.

For example, I have an API call which results in a grpc payload of size 9850644. Without the custom grpc limit defined in tyk.conf, I get the following error in tyk-gateway_stderr.log:

time=“May 06 21:17:09” level=error msg=“Dispatch error” api_id=“abc” api_name=“Health Insights api” error=“rpc error: code = ResourceExhausted desc = grpc: received message larger than max (9850644 vs. 4194304)” mw=CoProcessMiddleware org_id=1 origin=127.0.0.1 path=“/a/b/c”

With the following config in tyk.conf,
“coprocess_options”: {
“enable_coprocess”: true,
“grpc_recv_max_size”: 10000,
“grpc_send_max_size”: 10000,
“coprocess_grpc_server”: “tcp://localhost:5555”
},

I got:
time=“May 06 21:17:09” level=error msg=“Dispatch error” api_id=“abc” api_name=“Health Insights api” error=“rpc error: code = ResourceExhausted desc = grpc: received message larger than max (9850644 vs. 10000)” mw=CoProcessMiddleware org_id=1 origin=127.0.0.1 path=“/a/b/c”

However, if I increase the limits beyond 9850644 (which is the size of the message being tested), I’m still getting the same error and the max size mentioned in the error message is set at 4194304, which is the default message size limit, like so:

"coprocess_options": {
    "enable_coprocess": true,    
    "grpc_recv_max_size": 99999999,
    "grpc_send_max_size": 90000000,
    "coprocess_grpc_server": "tcp://localhost:5555"
},

corresponding error message:
time=“May 06 21:17:09” level=error msg=“Dispatch error” api_id=“abc” api_name=“Health Insights api” error=“rpc error: code = ResourceExhausted desc = grpc: received message larger than max (9850644 vs. 4194304)” mw=CoProcessMiddleware org_id=1 origin=127.0.0.1 path=“/a/b/c”

10

It turns out in my grpc server (the Tyk plugin), I instantiated the GRPC server instance without the size limits. After setting some large hard-coded limits, the above error went away.

Additional question: what is the recommended way to read these custom grpc limits defined in the tyk.conf from the Grpc plugin server (Go based)? I tried the code below but they return 0 for both params.

recvSize := config.Global().CoProcessOptions.GRPCRecvMaxSize
sendSize := config.Global().CoProcessOptions.GRPCSendMaxSize