Tyk runs out of memory and crashes when uploading a large file with http post

Hi All,

It seems like tyk cant handle large requests without using a lot of memory. When I try to do a post request with a large file, tyk runs out of memory and crashes. Is there some optimization that can be done to prevent this other than just adding more memory ?

Im using tyk gateway v2.3.8. I have included the stack trace in case it could provide some clues.

The tyk gateway server reports 1.8G of free memory before doing the request

free -h total used free shared buff/cache available Mem: 2.0G 45M 1.8G 796K 102M 1.8G Swap: 1.0G 145M 874M

Example curl request:

curl -k -H 'Tyk-Token: some_api_key' -F "file=@test_file_947M" -XPOST http://haproxy_in_front_of_api_gateway:port/some_api/v1/files?token=some_token
* Trying ip...
* Connected to haproxy_in_front_of_api_gateway (ip) port port (#0)
> POST /some_api/v1/files?token=some_token HTTP/1.1
> User-Agent: curl/7.29.0
> Host: haproxy_in_front_of_api_gateway:port
> Accept: */*
> Tyk-Token: some_api_key
> Content-Length: 1020877518
> Expect: 100-continue
> Content-Type: multipart/form-data; boundary=----------------------------eb4816eecb33
> 
< HTTP/1.1 100 Continue
* HTTP 1.0, assume close after body
< HTTP/1.0 502 Bad Gateway
< Cache-Control: no-cache
< Connection: close
< Content-Type: text/html
< 
<html><body><h1>502 Bad Gateway</h1>
The server returned an invalid or incomplete response.
</body></html>
* Closing connection 0

When requesting directly from api gateway:

curl -vvv -k -H 'Tyk-Token: some_api_key' -F "file=@test_file_947M" -XPOST http://tyk_gateway_ip:port/some_api/v1/files?token=some_token
* About to connect() to tyk_gateway_ip port port (#0)
*   Trying ip...
* Connected to ip (ip) port port (#0)
> POST /some_api/v1/files?token=some_token HTTP/1.1
> User-Agent: curl/7.29.0
> Host: ip:port
> Accept: */*
> Tyk-Token: some_api_key
> Content-Length: 1020877518
> Expect: 100-continue
> Content-Type: multipart/form-data; boundary=----------------------------8b49b1a33a76
> 
< HTTP/1.1 100 Continue
* Empty reply from server
* Connection #0 to host ip left intact
curl: (52) Empty reply from server

Tyk gateway log:

fatal error: runtime: out of memory

The rest of the stack trace is here: Paste.ee - tyk-out-of-memory-stack-trace

Some Tyk middleware will buffer the request, so you may be putting the whole file into RAM which will kill the process. What midleware are you using?

M.

Hi Martin,

Thanks for the quick response. If by middleware you mean plugins, we are not using any. It’s just a standard installation of tyk on premise.

Is there some way to avoid loading the entire file into ram ?

BR/Sid

I meant any other middleware like body transforms or detailed request logging?

Also using querystring params for auth can cause it in older versions

Hi Martin,

Thanks for the tip. Seems like it doesn’t load the entire file into memory when we set “enable_detailed_recording”: false in /opt/tyk-gateway/tyk.conf

BR/Sid