Uploaded file gets corrupted when using GRPC custom middleware in tyk-gateway 2.7.0

Support
1

Hi,

When I tried to upload a file, the file ended up being corrupted after it went through my custom authentication middleware.

I’ve created the “Custom Authentication Plugin with NodeJS” following https://tyk.io/docs/customise-tyk/plugins/rich-plugins/grpc/custom-auth-nodejs/
It is stated that we must clone the tyk-protobuf repository (GitHub - TykTechnologies/tyk-protobuf: Protocol Buffer definitions for Tyk gRPC middlewares.).

Which lead me to my issue.
All the protobuf definitions misses all the new features from tyk 2.3.6+,
including the fixes for the following issues in tyk 2.7.0:

MiniRequestObject misses the Method, RequestUri, Scheme and RawBody properties.

I’ve created a pull request for the binding update (Bindings update (Tyk v.2.7.0) by haroun · Pull Request #1 · TykTechnologies/tyk-protobuf · GitHub).
However, it seems something is still missing because the binary file is still corrupted even if I’ve updated the proto bindings in my node project.

Can you show me what is missing in my case?
Please let me know if you need more information about my custom middleware or how I use tyk.

Regards

2

Hi, your pull request is correct. When using the new bindings, are you getting any error?

Can you share your middleware code?

Best

3 
const {resolve} = require('path');
const grpc = require('grpc');
const introspectInternal = require('./introspect-internal');

const port = process.env.PORT || 5566;
const listenAddress = `0.0.0.0:${port}`;

const tyk = grpc.load({
  file: 'coprocess_object.proto',
  root: resolve(__dirname, 'tyk-protobuf/proto')
}).coprocess;

// Keep lowercase headers
// getHeaders :: Object -> Object
const getHeaders = request => Object.entries(request.headers)
  .reduce((acc, [header, value]) => {
    acc[header.toLowerCase()] = value;

    return acc;
  }, {});

// Handler
// handleError :: (Error, Object) -> undefined
const handleError = (err, req) => {
  console.error('[GRPC]', err.toString(), JSON.stringify(err.stack || ''), '- headers:', JSON.stringify(req.request.headers));

  const response = JSON.parse(JSON.stringify(req));
  
  response.request.return_overrides.response_code = 401;
  response.request.return_overrides.headers = {
    'Content-Type': 'application/json'
  };
  response.request.return_overrides.response_error = err.message;

  return response;
};

// Middleware
// authMiddleware :: (Object, Function) -> undefined
const authMiddleware = async (obj, callback) => {
  // Deep clone
  // const response = JSON.parse(JSON.stringify(obj)); // result in key not authorised
  const response = obj;

  // Keep lowercase headers
  const headers = getHeaders(response.request);

  // We take the value from the "Authorization" header:
  const authorizationToken = (headers.authorization || ''))
    .substring('Bearer '.length);
  if (authorizationToken === '') {
    throw new Error('"authorization" header is missing or empty');
  }

  const req = {
    headers,
    body: {
      token: authorizationToken
    },
    params: {},
    url: response.request.url,
    tokenResponse: {}
  };

  console.log('[GRPC] req:', JSON.stringify(req));

  const token = await introspectInternal.introspect(req);
  req.tokenResponse.Authorization = token.authorization;
  req.tokenResponse['X-Forwarded-Authorization'] = req.token;

  // The token should be attached to the object metadata, this is used internally for key management:
  response.metadata = Object.assign(
    {token: req.tokenResponse['X-Forwarded-Authorization']},
    req.tokenResponse
  );

  // If the Authrorization and X-Forwarded-Authorization are empty in tokenResponse, we return the call:
  if (!req.tokenResponse.Authorization && !req.tokenResponse['X-Forwarded-Authorization']) {
    callback(null, obj);

    return;
  }

  if (req.tokenResponse['X-Forwarded-Authorization']) {
    response.request.set_headers['X-Forwarded-Authorization'] = 'Bearer ' + req.tokenResponse['X-Forwarded-Authorization'];
  }
  response.request.set_headers.Authorization = 'Bearer ' + req.tokenResponse.Authorization;

  // At this point the token is valid and a session state object is initialized and attached to the coprocess.Object:
  const session = new tyk.SessionState();
  session.alias = req.tokenResponse['X-Forwarded-Authorization'];
  response.session = session;

  callback(null, response);
};

// The dispatch function is called for every hook:
const dispatch = async (call, callback) => {
  try {
    const obj = call.request;
    // We dispatch the request based on the hook name, we pass obj.request which is the coprocess.Object:
    switch (obj.hook_name) {
      case 'MyAuthMiddleware':
        await authMiddleware(obj, callback);
        break;
      default:
        callback(null, obj);
        break;
    }
  } catch (err) {
    const response = handleError(err, call.request);
    callback(null, response);
  }
};
const dispatchEvent = () => {};
const reload = () => {};

const main = () => {
  const server = new grpc.Server();
  server.addService(tyk.Dispatcher.service, {
    dispatch,
    dispatchEvent,
    reload
  });
  server.bind(listenAddress, grpc.ServerCredentials.createInsecure());
  console.log('[GRPC]', `GRPC server running on ${listenAddress}`);
  server.start();
};

module.exports = main;
4

Actually it’s not that the file is corrupted, in fact body is empty if I send a file.

curl -X POST \
  http://localhost/test \
  -H 'Authorization: Bearer abcd' \
  -H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \
  -F type=test \
  -F document=@/tmp/example.jpg

I’m not 100% sure but it seems raw_body is not used to rebuild body.
If I remove the document field from my request, body is filled.

Am I supposed to rebuild body from raw_body in my custom middleware?

5

Can you share the verbose output of your Curl command?

I suggest you check the length of the incoming body:

console.log("body.length is", response.request.body.length)
console.log("raw_body.length is", response.request.raw_body.length)
6 
body.length is 0
raw_body.length is 118190
7

What about the Curl output? I suggest using:

curl -v -X POST \
  http://localhost/test \
  -H 'Authorization: Bearer abcd' \
  -H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \
  -F type=test \
  -F document=@/tmp/example.jpg

It would be useful to see the headers that are sent.

8 
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 80 (#0)
> POST /test HTTP/1.1
> Host: localhost:80
> User-Agent: curl/7.54.0
> Accept: */*
> Authorization: Bearer abcd
> Content-Length: 798
> Expect: 100-continue
> content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW; boundary=------------------------42f5a57c8daa8cf0
> 
< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< Content-Length: 0
< Date: Fri, 10 Aug 2018 06:34:01 GMT
< X-Ratelimit-Limit: 0
< X-Ratelimit-Remaining: 0
< X-Ratelimit-Reset: 0
< Connection: close
< 
* Closing connection 0

However I tried with a JSON file instead of a JPG one, and it worked

body.length is 928
raw_body.length is 928

Do you think the problem could be related to

if utf8.Valid(miniRequestObject.RawBody) {

in coprocess.go (tyk/coprocess.go at 7172b91d6cb0fb57b37ea452b57ca45515b33e71 · TykTechnologies/tyk · GitHub)?

9

I also checked the mime type of my files using file -I <filepath>

It seems to work as long as the result is not xxx/yyy: charset=binary