Multipart / streaming mode

Support
1

Hello TYK Team,

I am trying to expose an API that allows for the upload of large files (4 GB and more) via TYK.

My initial tests of sending in form-data/multipart via curl show that TYK buffers the stream before sending it to the backend, which poses a real memory problem. I have developed a Go client that does multipart, and the streaming works correctly.

Upon analyzing the TCP streams, I observe that in the Go client, which correctly handles streaming, it sends the file in “chunked” mode.

However, in a traditional web client, we cannot force the client to “chunk” the file parts.

Would you have a solution to recommend to resolve this problem and force TYK to stream even in non-“chunked” mode?

For your information, I have tested the flush_interval option set to 1, but without result.

Thank you.

With clientGO:

trace tcp with go client

POST /api/upload HTTP/1.1
Host: localhost:8081
User-Agent: Go-http-client/1.1
Transfer-Encoding: chunked
Content-Type: multipart/form-data; boundary=92461e7b6d843c95f69b5e1911fc30e61de7ee42cdb5dfc57352d19c245b
Accept-Encoding: gzip

ae
–92461e7b6d843c95f69b5e1911fc30e61de7ee42cdb5dfc57352d19c245b
Content-Disposition: form-data; name=“file”; filename=“test1m.txt”
Content-Type: application/octet-stream

8000
012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012
…
–92461e7b6d843c95f69b5e1911fc30e61de7ee42cdb5dfc57352d19c245b–

0

HTTP/1.1 200 OK
Content-Length: 38
Content-Type: text/plain; charset=utf-8
Date: Thu, 18 Apr 2024 09:19:46 GMT
X-Ratelimit-Limit: 0
X-Ratelimit-Remaining: 0
X-Ratelimit-Reset: 0

trace curl --request POST --url http://localhost:8081/api/upload --header 'Content-Type: multipart/form-data' --form 'file=@.//test1m.bin'

POST /api/upload HTTP/1.1
Host: localhost:8081
User-Agent: curl/7.88.1
Accept: /
Content-Length: 1048782
Content-Type: multipart/form-data; boundary=------------------------cffb6b7aa793d53d
Expect: 100-continue

HTTP/1.1 100 Continue

--------------------------cffb6b7aa793d53d
Content-Disposition: form-data; name=“file”; filename=“test1m.bin”
Content-Type: application/octet-stream

01234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345
…
45678901234567890123456789012345678901234567890123456789
--------------------------3a227cc5be04a20b–
HTTP/1.1 200 OK
Content-Length: 38
Content-Type: text/plain; charset=utf-8
Date: Thu, 18 Apr 2024 09:32:38 GMT
X-Ratelimit-Limit: 0
X-Ratelimit-Remaining: 0
X-Ratelimit-Reset: 0

2

I found the code that prevents streaming and involves buffering the stream in the ReverseProxy.go part:

func copyRequest(r *http.Request) (*http.Request, error) {
fmt.Println(“copyRequest”)
var err error

if r.ContentLength == -1 {
	return r, nil
}

if r.Body != nil {
	r.Body, err = copyBody(r.Body, false)
}
return r, err

}
When I pass through my client that sends in multipart, it sends in chunked and most importantly without specifying the content-length. As a result, the copyRequest function does not execute the “copyBody”. By modifying this function for the test in the following way:

func copyRequest(r *http.Request) (*http.Request, error) {
fmt.Println(“copyRequest”)
var err error

/*
if r.ContentLength == -1 {
	return r, nil
}

if r.Body != nil {
	r.Body, err = copyBody(r.Body, false)
}
*/
if r.Body != nil {
	return r, nil
}
return r, err

}
I get the desired result, the transfer is well done in streaming mode, I can see the progressive reception on the backend side.

I would like to understand why we need to copy the body of the request at the moment of the reverseProxy, that is, at the Outbound.

Thank you, the team.

3

@Alexandre_Wintrebert Hello and thanks for your question.

I have forwarded your inquiry internally. I will reply once I get a response.

Cheers,
Olu

4

Hello Olu,
do you have any information on my question?
I think I will make a PR about it. I hope it will be accepted.

Thanks

5

I haven’t received an update on your question yet. However, please feel free to proceed with creating a PR.

Good luck! :crossed_fingers: