Middleware and request with binary content

Hello,

I think there might be a bug in TYK regarding middleware scripts. When the request that is handled in the middleware script contains binary data it seems that the data in the request gets corrupted.

We have a backend API that we wish to publish using TYK. One endpoint of the backend enables user to upload images. The image data is transferred to backend using a PUT HTTP request. Everything works fine until I add a middleware script to the endpoint. The script is the one I took from your documentation and it actually does nothing but some logging: https://tyk.io/tyk-documentation/customise-tyk/javascript-middleware/middleware-scripting-guide/. As soon as the script is added I see already in the Log browser of the TYK Dasboard that the body of the request gets changed. Also backend gives an error response because it is not abled to process the image.

It seems that somehow the binary body of the request gets changed when the request is passed to the middleware script. Is this a known bug? When could I expect to get a fix for it?

-Antti Huokko

Hi, can you provide more details on how the data is affected?
Does the output show a different length?

Hello,

The content length is also affected. I add here the beginning ok and not ok requests. The data is copy pasted from the TYK Log Viewer. There you can clearly see the changes to the body.

------ OK REQUEST -----

PUT /apiserver/v1/user/profileimage HTTP/1.1
Host: sally-iam-apimanagement.amersportsdigital.com
User-Agent: okhttp/3.4.1
Connection: close
Content-Length: 79812
Accept-Encoding: gzip
Authorization: Bearer eyAidHlwIjogIkpXVCIsICJraWQiOiAiUDEzc2N0ZXIzZ3dQK2ZveHBLM0ZTVHdRVlFZPSIsICJhbGciOiAiUlMyNTYiIH0.eyAiYXRfaGFzaCI6ICJ3YmhzZUFxMVVVMkxkcTUtNjlfdllnIiwgInN1YiI6ICI4NTUzZmRhNS1hMTA3LTRkNzktOGIzYi1hNDgzNjY2NzA1ODEiLCAiYXVkaXRUcmFja2luZ0lkIjogIjhjODQ5M2FmLTA3YzEtNDY4OS1hOGRiLTY5ZTc2N2YzM2E1Mi0yNjYiLCAiaXNzIjogImh0dHBzOi8vc2FsbHktaWFtLWZyb250ZW5kLTAxLmFtZXJzcG9ydHNkaWdpdGFsLmNvbTo0NDMvc3NvL29hdXRoMi9TVCIsICJ0b2tlbk5hbWUiOiAiaWRfdG9rZW4iLCAibm9uY2UiOiAiYW1lcjEyMzQiLCAiYXVkIjogInN0LWNsaWVudCIsICJvcmcuZm9yZ2Vyb2NrLm9wZW5pZGNvbm5lY3Qub3BzIjogIjdkN2QxYzBjLTAwYjEtNDEzNi1hMDNjLWYzOWZmYjdhNjBmZCIsICJhenAiOiAic3QtY2xpZW50IiwgImF1dGhfdGltZSI6IDE0NzY2ODgyOTAsICJyZWFsbSI6ICIvU1QiLCAiZXhwIjogMTQ3NjY5MTg5MCwgInRva2VuVHlwZSI6ICJKV1RUb2tlbiIsICJpYXQiOiAxNDc2Njg4MjkwIH0.q1iGV8jD18Cznnidf86fDV3kx3iL1nQ0HfGL91MdZgQFh4wBaxRk3uPvMe2QR_-bwbNSUobjuo6hzsTHMosuNk7HQd6otfxmF1hjQmudEsLT-BCd7vN76kdMWFuz-HQV9dPJdRwzqfDyBq1YI6LW1NaGURZHdOmqd9sO4ihaehPNx2LTCqGTTPFcB4Xm4rgDagmh6mICnIZqEOxxZfkrpfOIeN1G-yr1i5ErN0p5J6SBp6LUEoY3TQX6QXI7pOATXi0vNGdXLO2GO_Sh2OwR2eVsd-d7HbdhlcgtEUKkmUNMJheH-qeEOHlGQxE-_eZI4ZIjvqlz79AMuNO95OAmOw
Connection: close
Content-Type: application/octet-stream
Sttauthorization: csnqmh7vf1fqrft0aje6ut6s45s6j5au
X-Forwarded-For: 109.70.165.70
X-Forwarded-Proto: https
X-Real-Ip: 109.70.165.70

ÿØÿàJFIFÿÛC
aa



ÿÛC

ÿÀ†"ÿÄa
ÿĵ}!1AQaa"q2‘¡#B±ÁRÑð$3br‚
%&’()*456789:CDEFGHIJSTUVWXYZcdefghijstuvwxyzƒ„
†‡ˆ‰Š’“”•–—˜™š¢£¤¥¦§¨©ª²³´µ¶·¸¹ºÂÃÄÅÆÇÈÉÊÒÓÔÕÖ×ØÙÚáâãäåæçèéêñòóôõö÷øùúÿÄa
ÿĵaw!1AQaaq"2B‘¡±Á #3RðbrÑ
$4á%ñ&’()*56789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz‚ƒ„
†‡ˆ‰Š’“”•–—˜™š¢£¤¥¦§¨©ª²³´µ¶·¸¹ºÂÃÄÅÆÇÈÉÊÒÓÔÕÖ×ØÙÚâãäåæçèéêòóôõö÷øùúÿÚ?üû¶™ÇcŒûŸëL»Œ‰
£pù¿ýu-©•%Îâc|)ãüúRʙW$íÉ鞟JóögjWFTÇfv¯·§ò h—
Iû¬sS
ì¼îëÉõô4Òy9Á!qôë[&BE2²$ù'ééV7ïV>üŸNj+Ž_qe¹ãa©ô¨ÒV°?ìŠv’Çû¢p>lƒ×úÊ¤EùÈ mÏ9æšß4~\ô‡øӘÜÝ[€F;ÿœÑ¸]Ê-ÐóïïUf]’¶Á ǯõ«.ÇcnvŸ¥UsûÇe†ãÏôЯaNá·qa#$šžaiWa?F~b«×8AÍI#,¹pr3Š]abΞsp HO?0ã×J“ÉäE–ec½¶'¿?Z笊Ç!9À<gšÛ
ÜD$ŽaÔu)iDˆ
õÏcP6E°Úä:õ©/.r€AߥGi#«:`sߊ»

------ NOK REQUEST -----

PUT /apiserver/v1/user/profileimage HTTP/1.1
Host: sally-iam-apimanagement.amersportsdigital.com
User-Agent: okhttp/3.4.1
Connection: close
Content-Length: 149450
Accept-Encoding: gzip
Authorization: Bearer eyAidHlwIjogIkpXVCIsICJraWQiOiAiUDEzc2N0ZXIzZ3dQK2ZveHBLM0ZTVHdRVlFZPSIsICJhbGciOiAiUlMyNTYiIH0.eyAiYXRfaGFzaCI6ICJ3YmhzZUFxMVVVMkxkcTUtNjlfdllnIiwgInN1YiI6ICI4NTUzZmRhNS1hMTA3LTRkNzktOGIzYi1hNDgzNjY2NzA1ODEiLCAiYXVkaXRUcmFja2luZ0lkIjogIjhjODQ5M2FmLTA3YzEtNDY4OS1hOGRiLTY5ZTc2N2YzM2E1Mi0yNjYiLCAiaXNzIjogImh0dHBzOi8vc2FsbHktaWFtLWZyb250ZW5kLTAxLmFtZXJzcG9ydHNkaWdpdGFsLmNvbTo0NDMvc3NvL29hdXRoMi9TVCIsICJ0b2tlbk5hbWUiOiAiaWRfdG9rZW4iLCAibm9uY2UiOiAiYW1lcjEyMzQiLCAiYXVkIjogInN0LWNsaWVudCIsICJvcmcuZm9yZ2Vyb2NrLm9wZW5pZGNvbm5lY3Qub3BzIjogIjdkN2QxYzBjLTAwYjEtNDEzNi1hMDNjLWYzOWZmYjdhNjBmZCIsICJhenAiOiAic3QtY2xpZW50IiwgImF1dGhfdGltZSI6IDE0NzY2ODgyOTAsICJyZWFsbSI6ICIvU1QiLCAiZXhwIjogMTQ3NjY5MTg5MCwgInRva2VuVHlwZSI6ICJKV1RUb2tlbiIsICJpYXQiOiAxNDc2Njg4MjkwIH0.q1iGV8jD18Cznnidf86fDV3kx3iL1nQ0HfGL91MdZgQFh4wBaxRk3uPvMe2QR_-bwbNSUobjuo6hzsTHMosuNk7HQd6otfxmF1hjQmudEsLT-BCd7vN76kdMWFuz-HQV9dPJdRwzqfDyBq1YI6LW1NaGURZHdOmqd9sO4ihaehPNx2LTCqGTTPFcB4Xm4rgDagmh6mICnIZqEOxxZfkrpfOIeN1G-yr1i5ErN0p5J6SBp6LUEoY3TQX6QXI7pOATXi0vNGdXLO2GO_Sh2OwR2eVsd-d7HbdhlcgtEUKkmUNMJheH-qeEOHlGQxE-_eZI4ZIjvqlz79AMuNO95OAmOw
Connection: close
Content-Type: application/octet-stream
Sttauthorization: csnqmh7vf1fqrft0aje6ut6s45s6j5au
X-Forwarded-For: 109.70.165.70
X-Forwarded-Proto: https
X-Real-Ip: 109.70.165.70

����JFIF��C
aa



��C

���"��a
���}!1AQaa"q2���#B��R��$3br�
%&’()*456789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz���������������������������������������������������������������������������a
���aw!1AQaaq"2�B���� #3R�br�
$4�%�&’()*56789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz��������������������������������������������������������������������������?�����c����L���
�p���u-��%��c|)���RʙW$��鞟J��gjWFT�fv������h�
I��sS�`�������4�y9�!q��[&BE2�$�’��V7�V>��Nj+�_qe��a����V�?��v����p>

Here is the curl command that you can use to reproduce the problem: curl http://localhost:8080/mytest/ --upload-file HelloWorld.png

I am also pretty sure I found the cause of the problem. In plugins.go line 102 you set the body using string(originalBody). If the body is binary data it gets some string enocoding (probably UTF-8) here. And then you use the same data to the new request body in plugins.go line 163 that is now broken because it is encoded to string along the way.

We have issued a patch for this. The JS middleware can set a IgnoreBody flag, if this flag is set to true, any body transformation will be ignored. You may still set headers or perform additional operations.

In the context of your test request, this means that the original body will be used, and no encoding step will be performed. Sample code:

var samplePreProcessMiddleware = new TykJS.TykMiddleware.NewMiddleware({})
samplePreProcessMiddleware.NewProcessRequest(function(request, session) {
  request.IgnoreBody = true
  return samplePreProcessMiddleware.ReturnData(request, {})
})

Hello,

Thank you for a quick fix. We are using TYK via the Docker images. For development environment we use the tyk_quickstart and for production we will use the tyk-hybrid-docker. Could you still verify which of the docker images now contain fix. At least the tykio/tyk-gateway says that it is last updated a month ago so it does not contain the fix? The tykio/tyk-hybrid-docker was updated yesterday. Does it now contain the fix?

-Antti

Hi Antti,

The fix is currently only available in hybrid - as this is the easiest platform for us to patch and release. The docker images or the QuickStart do not contain the fix, these are a little behind our overall patch/release cycle.

Our standard deb/yum versions also do not contain the fix yet.

M.