We see these a fair amount “http: proxy error: context canceled”
This is a golang net lib error, appears to be due to the client prematurely severing the connection prior to the tyk proxy being complete in proxying the request?
Could this be related to a simple timeout?
Any settings in tyk we could tweak to possibly reduce these?
if “proxy_default_timeout” is NOT set, then is this infinite by default?
We’re just seeing a fair amount of these during certain times and cannot track down what it is.
a) client makes call to simple service via Tyk. This backend service who when called directly and not via tyk responds consistently in < 100ms
b) the plugin calls TykGetKeyData (to lookup a session), if session is valid (in this case it is) it proceeds to proxy the request.
c) all of the log statements generated by the plugins for this request all share “Jul 19 09:09:26” (1 second). So I assume the plugins are executing in < 1 second
d) yet the last log statement (from tyk) shows the client timing out… .(the client has a 30s timeout) “http: proxy error: context canceled”. The timestamp on this tyk log statement is the same “Jul 19 09:09:26”
e) again the backend service is responding fine.
A bit lost as to what is causing this, its definitely something in tyk but not sure where to look as the logs are not very helpful
I can reproduce this in a manual call, its like one in every 30 requests has this issue. The backend responds fine, but when I go through tyk this happens every so often
anyone have any thoughts on this?
We are using JSVM plugins.
again for the failed request, you can see the plugin executing (we have logs) in < 1s, and the “context cancelled” statement occurs after executing and within the same time period.
any tips on how to debug this further its becoming a concern
Hi TYK Team,
We upgraded our gateway & dashboard to the latest TYK versions (gateway to 2.7.0 and dashboard to 1.7.0) last week.
We are also experiencing similar error for some of the API calls.
Could you please let us know the root cause and solution to this issue ?
Thanks
Hi TYK Team,
Any updates on this ?
We are getting these frequently in our production systems.
Regards,
Shabeer
Hello Tyk Team,
I also have the same issue in my gateway log when there are high volume of requests.
Any new updates?
Thank you
getting same issue too. Mostly happening when Tyk is under enormous load
In general, the error itself just means that the connection was closed unexpectedly.
It can happen for various reasons, and in some cases it is totally fine: for example client can have unstable mobile internet.
When it happens on the high load, it can be a lot of different reasons.
For example your OS is running out of system limits, like number of opened sockets, and to validate it, you need to try your system limits. See this guide Planning for Production.
Additionally, it can be CPU bottleneck: you can’t process more than your machine can do. And note that it is not only about the actual utilization %, it is also about context switches it has to do. E.g. having one job which consume 100% of your CPU/cores, or having a few thousands jobs, causing CPU constantly switch between them. Such problems cause internal request processing queues, which cause latency growth (highly recommend measure it). And in some cases latency can grow so big, that some clients can just disconnect/timeout because of it.
Additionally, highly recommend read the following blog post Performance-tuning your Tyk API Gateway - Tyk API Gateway. For example, you can trade memory for performance, and context switch reduction by tuning garbage collector to run less frequently: see Tuning Tyk’s Garbage Collector section.
Also note that it is not Tyk or Golang specific. The problem I described will happen with any webserver on high scale. So in general if you see a lot of “context” errors on high load, use it as a sign that the process is really struggling with the given load, and you need scale it up, either vertically or horizontally.
Hope it makes sense!