Please help me with Custom Middleware

Imported Google Group message. Original thread at: Redirecting to Google Groups Import Date: 2016-01-19 21:41:57 +0000.
Sender:Jelda Lam.
Date:Tuesday, 5 January 2016 04:03:04 UTC.

Hi All,

I’ve tried tyk recently. Everything works smoothly when I come into middleware use case.
All I want is something could handle token from request and authenticate it with our exist credential system.
I read the document and come to middleware script.

Firstly, I’ve tried simple middleware with the setup directly on Ubuntu host and I keep receiving panic message from logs

panic serving interface conversion: interface is nil, not main.SessionState

So I tried to install it with docker compose with the same config. I notice it does not log in /var/logs/upstart I finally I found it log in docker logs

I use tyk 1.9

My middleware script

var samplePostProcessMiddleware = new TykJS.TykMiddleware.NewMiddleware({});

samplePostProcessMiddleware.NewProcessRequest(function(request, session) {
log(“Running sample POST PROCESSOR JSVM middleware”)

request.SetHeaders["MyCount"] = 5;


return samplePostProcessMiddleware.ReturnData(request, {});

});

log(“Sample POST middleware initialised”);

My API Definition

{
“id”: “568a01606d97a80001000001”,
“name”: “Test API”,
“slug”: “test-api”,
“api_id”: “e49f7dbafdfb46ac5ab4490eb105a495”,
“org_id”: “568211ba712d570001000005”,
“use_keyless”: false,
“use_oauth2”: false,
“oauth_meta”: {
“allowed_access_types”: [],
“allowed_authorize_types”: [],
“auth_login_redirect”: “”
},
“auth”: {
“use_param”: false,
“use_cookie”: false,
“auth_header_name”: “Authorization”
},
“use_basic_auth”: false,
“enable_jwt”: false,
“jwt_signing_method”: “”,
“notifications”: {
“shared_secret”: “”,
“oauth_on_keychange_url”: “”
},
“enable_signature_checking”: false,
“hmac_allowed_clock_skew”: -1,
“definition”: {
“location”: “header”,
“key”: “x-api-version”
},
“version_data”: {
“not_versioned”: true,
“versions”: {
“Default”: {
“name”: “Default”,
“expires”: “”,
“paths”: {
“ignored”: [],
“white_list”: [],
“black_list”: []
},
“use_extended_paths”: true,
“extended_paths”: {
“ignored”: [],
“white_list”: [],
“black_list”: [],
“cache”: [],
“transform”: [],
“transform_response”: [],
“transform_headers”: [],
“transform_response_headers”: [],
“hard_timeouts”: [],
“circuit_breakers”: [],
“url_rewrites”: [],
“virtual”: [],
“size_limits”: []
},
“global_headers”: {},
“global_headers_remove”: [],
“global_size_limit”: 0
}
}
},
“uptime_tests”: {
“check_list”: [],
“config”: {
“expire_utime_after”: 0,
“service_discovery”: {
“use_discovery_service”: false,
“query_endpoint”: “”,
“use_nested_query”: false,
“parent_data_path”: “”,
“data_path”: “”,
“port_data_path”: “”,
“use_target_list”: false,
“cache_timeout”: 60,
“endpoint_returns_list”: false
},
“recheck_wait”: 0
}
},
“proxy”: {
“listen_path”: “/test-api/”,
“target_url”: “http://httpbin.org/”,
“strip_listen_path”: true,
“enable_load_balancing”: false,
“target_list”: [],
“check_host_against_uptime_tests”: false,
“service_discovery”: {
“use_discovery_service”: false,
“query_endpoint”: “”,
“use_nested_query”: false,
“parent_data_path”: “”,
“data_path”: “hostname”,
“port_data_path”: “port”,
“use_target_list”: false,
“cache_timeout”: 60,
“endpoint_returns_list”: false
}
},
“custom_middleware”: {
“pre”: [],
“post”: [{
“name”: “samplePostProcessMiddleware”,
“path”: “/opt/tyk-gateway/middleware/postWare.js”,
“require_session”: false
}],
“response”: []
},
“cache_options”: {
“cache_timeout”: 60,
“enable_cache”: true,
“cache_all_safe_requests”: false,
“enable_upstream_cache_control”: false
},
“session_lifetime”: 0,
“active”: true,
“auth_provider”: {
“name”: “”,
“storage_engine”: “”,
“meta”: {}
},
“session_provider”: {
“name”: “”,
“storage_engine”: “”,
“meta”: null
},
“event_handlers”: {
“events”: {}
},
“enable_batch_request_support”: false,
“enable_ip_whitelisting”: false,
“allowed_ips”: [],
“dont_set_quota_on_create”: false,
“expire_analytics_after”: 0,
“response_processors”: [],
“CORS”: {
“enable”: false,
“allowed_origins”: [],
“allowed_methods”: [],
“allowed_headers”: [],
“exposed_headers”: [],
“allow_credentials”: false,
“max_age”: 24,
“options_passthrough”: false,
“debug”: false
},
“domain”: “”,
“do_not_track”: false,
“tags”: []
}

This is log when I define new API

“Reload signal received, reloading endpoints”
level=info msg=“Detected 1 APIs”
level=info msg=“–> Loading API: Test API”
level=info msg=“----> Tracking: (no host)”
level=info msg=“----> Checking security policy: Token”
level=info msg=“Loading uptime tests…”
level=info msg="Loaded 0 policies "
level=info msg=“API reload complete”

Seem like it doesn’t recognize middleware config

My Questions is

  1. Is the docker version the same with Ubuntu version?
  2. Is there any way to make a connection to the database like Postgres or MongoDB in middleware context?
  3. Why the logs aren’t writing to upstart when it come to docker?
  4. Can you guys help me out with simple middleware config.

Imported Google Group message.
Sender:Martin Buhr.
Date:Tuesday, 5 January 2016 07:24:59 UTC.

Hi Jelda,

You’ll first need to check if the JSVM is enabled in the Tyk.conf file, if it is disabled (by default I believe), then it will never load middleware:

https://tyk.io/v1.9/configuration/configuration/#enable-jsvm:a66b35d20295cb764719ac8bd35837ec

Secondly, I think because this is a post processor, you also might need to enable the session by setting requires_session to true.

Many thanks,
Martin


From: Jelda Lam [email protected]
Sent: Tuesday, January 5, 2016 04:03
Subject: Please help me with Custom Middleware
To: Tyk Community Support [email protected]

Hi All,

I’ve tried tyk recently. Everything works smoothly when I come into middleware use case.
All I want is something could handle token from request and authenticate it with our exist credential system.
I read the document and come to middleware script.

Firstly, I’ve tried simple middleware with the setup directly on Ubuntu host and I keep receiving panic message from logs

panic serving interface conversion: interface is nil, not main.SessionState

So I tried to install it with docker compose with the same config. I notice it does not log in /var/logs/upstart I finally I found it log in docker logs

I use tyk 1.9

My middleware script

var samplePostProcessMiddleware = new TykJS . TykMiddleware . NewMiddleware ({});

samplePostProcessMiddleware . NewProcessRequest ( function ( request , session ) {

log ( "Running sample  POST PROCESSOR JSVM middleware" )

request . SetHeaders [ "MyCount" ] = 5 ;


return samplePostProcessMiddleware . ReturnData ( request , {});

});

log ( “Sample POST middleware initialised” );

My API Definition

  • show quoted text -

  • show quoted text -

level = info msg = “–> Loading API: Test API”
level = info msg = “----> Tracking: (no host)”
level = info msg = “----> Checking security policy: Token”
level = info msg = “Loading uptime tests…”
level = info msg = "Loaded 0 policies "
level = info msg = “API reload complete”

Seem like it doesn’t recognize middleware config

My Questions is

  1. Is the docker version the same with Ubuntu version?
  2. Is there any way to make a connection to the database like Postgres or MongoDB in middleware context?
  3. Why the logs aren’t writing to upstart when it come to docker?
  4. Can you guys help me out with simple middleware config.


You received this message because you are subscribed to the Google Groups “Tyk Community Support” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web, visit https://groups.google.com/d/msgid/tyk-community-support/8eac0315-9bb1-46d6-a106-d8a71139e00b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Imported Google Group message.
Sender:Jelda Lam.
Date:Tuesday, 5 January 2016 07:24:59 UTC.

Hi Jelda,

You’ll first need to check if the JSVM is enabled in the Tyk.conf file, if it is disabled (by default I believe), then it will never load middleware:

https://tyk.io/v1.9/configuration/configuration/#enable-jsvm:a66b35d20295cb764719ac8bd35837ec

Secondly, I think because this is a post processor, you also might need to enable the session by setting requires_session to true.

Many thanks,
Martin


From: Jelda Lam [email protected]
Sent: Tuesday, January 5, 2016 04:03
Subject: Please help me with Custom Middleware
To: Tyk Community Support [email protected]

Hi All,

I’ve tried tyk recently. Everything works smoothly when I come into middleware use case.
All I want is something could handle token from request and authenticate it with our exist credential system.
I read the document and come to middleware script.

Firstly, I’ve tried simple middleware with the setup directly on Ubuntu host and I keep receiving panic message from logs

panic serving interface conversion: interface is nil, not main.SessionState

So I tried to install it with docker compose with the same config. I notice it does not log in /var/logs/upstart I finally I found it log in docker logs

I use tyk 1.9

My middleware script

var samplePostProcessMiddleware = new TykJS . TykMiddleware . NewMiddleware ({});

samplePostProcessMiddleware . NewProcessRequest ( function ( request , session ) {

log ( "Running sample  POST PROCESSOR JSVM middleware" )

request . SetHeaders [ "MyCount" ] = 5 ;


return samplePostProcessMiddleware . ReturnData ( request , {});

});

log ( “Sample POST middleware initialised” );

My API Definition

  • show quoted text -

  • show quoted text -

level = info msg = “–> Loading API: Test API”
level = info msg = “----> Tracking: (no host)”
level = info msg = “----> Checking security policy: Token”
level = info msg = “Loading uptime tests…”
level = info msg = "Loaded 0 policies "
level = info msg = “API reload complete”

Seem like it doesn’t recognize middleware config

My Questions is

  1. Is the docker version the same with Ubuntu version?
  2. Is there any way to make a connection to the database like Postgres or MongoDB in middleware context?
  3. Why the logs aren’t writing to upstart when it come to docker?
  4. Can you guys help me out with simple middleware config.


You received this message because you are subscribed to the Google Groups “Tyk Community Support” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web, visit https://groups.google.com/d/msgid/tyk-community-support/8eac0315-9bb1-46d6-a106-d8a71139e00b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Imported Google Group message.
Sender:Martin Buhr.
Date:Tuesday, 5 January 2016 09:03:20 UTC.

Hi Martin,

I got it worked, thank you very much.
As you say, JSVM is disabled.
By the way, I would like to know how much middleware affects the performance, do you have any numbers.
Do the team have any road map about improving the capabilities of middleware.

Imported Google Group message.
Sender:Jelda Lam.
Date:Tuesday, 5 January 2016 15:37:55 UTC.

Hi Jelda,

We’re open to proposals on what to do regarding middleware.

It does affect performance, it depends on your use case really. I have no numbers to share I’m afraid.

What kind of capabilities are you looking for?

Many thanks,
Martin

  • show quoted text -

  • show quoted text -


You received this message because you are subscribed to the Google Groups “Tyk Community Support” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web, visit https://groups.google.com/d/msgid/tyk-community-support/6241662d-2db2-42b0-9e3e-216d78a820ec%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Imported Google Group message.
Sender:Jelda Lam.
Date:Wednesday, 6 January 2016 04:00:12 UTC.

Hi Martin,

My use case is having process token from the request and authenticate it with our exist credential system before it go to upstream.
So I think the capabilities of connection to database is better than having it call to another API.

I also have a question about Session in Middleware script.

My script

var samplePostProcessMiddleware = new TykJS.TykMiddleware.NewMiddleware({});

samplePostProcessMiddleware.NewProcessRequest(function(request, session) {

log("My Count POST PROCESSOR")

if (session.meta_data) {

if (!session.meta_data[“MyCount”]) session.meta_data[“MyCount”] = “1”;

log(JSON.stringify(session.meta_data));

session.meta_data[“MyCount”] = ((session.meta_data[“MyCount”] * 1) + 1) + “”;

log(JSON.stringify(session.meta_data));

request.SetHeaders[“MyCount”] = session.meta_data[“MyCount”];
}

return samplePostProcessMiddleware.ReturnData(request, session.meta_data);

});

log(“My Count POST PROCESSOR initialised”);

I notice session value doesn’t update overtime.
Here’s my log

time=“2016-01-05T09:33:00Z” level=info msg="[JSVM] [LOG]: {“MyCount”:“14”}"
time=“2016-01-05T09:33:00Z” level=info msg="[JSVM] [LOG]: {“MyCount”:“15”}"
time=“2016-01-05T09:33:02Z” level=info msg="[JSVM] [LOG]: My Count POST PROCESSOR"
time=“2016-01-05T09:33:02Z” level=info msg="[JSVM] [LOG]: {“MyCount”:“14”}"
time=“2016-01-05T09:33:02Z” level=info msg="[JSVM] [LOG]: {“MyCount”:“15”}"
time=“2016-01-05T09:33:03Z” level=info msg="[JSVM] [LOG]: My Count POST PROCESSOR"
time=“2016-01-05T09:33:03Z” level=info msg="[JSVM] [LOG]: {“MyCount”:“14”}"
time=“2016-01-05T09:33:03Z” level=info msg="[JSVM] [LOG]: {“MyCount”:“15”}"
time=“2016-01-05T09:33:13Z” level=info msg="[JSVM] [LOG]: My Count POST PROCESSOR"
time=“2016-01-05T09:33:13Z” level=info msg="[JSVM] [LOG]: {“MyCount”:“15”}"
time=“2016-01-05T09:33:13Z” level=info msg="[JSVM] [LOG]: {“MyCount”:“16”}"
time=“2016-01-05T09:33:21Z” level=info msg="[JSVM] [LOG]: My Count POST PROCESSOR"
time=“2016-01-05T09:33:21Z” level=info msg="[JSVM] [LOG]: {“MyCount”:“15”}"
time=“2016-01-05T09:33:21Z” level=info msg="[JSVM] [LOG]: {“MyCount”:“16”}"
time=“2016-01-05T09:33:22Z” level=info msg="[JSVM] [LOG]: My Count POST PROCESSOR"
time=“2016-01-05T09:33:22Z” level=info msg="[JSVM] [LOG]: {“MyCount”:“15”}"
time=“2016-01-05T09:33:22Z” level=info msg="[JSVM] [LOG]: {“MyCount”:“16”}"
time=“2016-01-05T09:33:23Z” level=info msg="[JSVM] [LOG]: My Count POST PROCESSOR"
time=“2016-01-05T09:33:23Z” level=info msg="[JSVM] [LOG]: {“MyCount”:“16”}"
time=“2016-01-05T09:33:23Z” level=info msg="[JSVM] [LOG]: {“MyCount”:“17”}"

Is Session have a delay update or something?

Thanks

Imported Google Group message.
Sender:Martin Buhr.
Date:Wednesday, 6 January 2016 06:31:42 UTC.

Hi Jelda,

I see - we are currently working on a tool that helps create tokens based on a third-party identity provider (I.e tokens are issued on the instruction of a third party DB or service, not verified during the flow since they have been created by a trusted party), would that be of use?

As for sessions, session state data is cached on device and is written back to Redis off thread to speed things up, you also might see copies of session data go into copies of the sandbox environment so it’s not a great place to store fast moving data, if you want to update the session there’s a built in API for that that’s more reliable (take a look at our blog post on IP rate limiting with middleware)

Cheers,
Martin


From: Jelda Lam [email protected]
Sent: Wednesday, January 6, 2016 04:00
Subject: Re: Please help me with Custom Middleware

To: Tyk Community Support [email protected]
Cc: [email protected]

Hi Martin,

My use case is having process token from the request and authenticate it with our exist credential system before it go to upstream.
So I think the capabilities of connection to database is better than having it call to another API.

I also have a question about Session in Middleware script.

My script

var samplePostProcessMiddleware = new TykJS . TykMiddleware . NewMiddleware ({});

samplePostProcessMiddleware . NewProcessRequest ( function ( request , session ) {

log ( "My Count POST PROCESSOR" )

if ( session . meta_data ) {

if (! session . meta_data [ “MyCount” ]) session . meta_data [ “MyCount” ] = “1” ;

log ( JSON . stringify ( session . meta_data ));

session . meta_data [ “MyCount” ] = (( session . meta_data [ “MyCount” ] * 1 ) + 1 ) + “” ;

log ( JSON . stringify ( session . meta_data ));

request . SetHeaders [ “MyCount” ] = session . meta_data [ “MyCount” ];
}

return samplePostProcessMiddleware . ReturnData ( request , session . meta_data );

});

log ( “My Count POST PROCESSOR initialised” );

I notice session value doesn’t update overtime.
Here’s my log

time = “2016-01-05T09:33:00Z” level = info msg = “[JSVM] [LOG]: {"MyCount":"14"}”
time = “2016-01-05T09:33:00Z” level = info msg = “[JSVM] [LOG]: {"MyCount":"15"}”
time = “2016-01-05T09:33:02Z” level = info msg = “[JSVM] [LOG]: My Count POST PROCESSOR”
time = “2016-01-05T09:33:02Z” level = info msg = “[JSVM] [LOG]: {"MyCount":"14"}”
time = “2016-01-05T09:33:02Z” level = info msg = “[JSVM] [LOG]: {"MyCount":"15"}”
time = “2016-01-05T09:33:03Z” level = info msg = “[JSVM] [LOG]: My Count POST PROCESSOR”
time = “2016-01-05T09:33:03Z” level = info msg = “[JSVM] [LOG]: {"MyCount":"14"}”
time = “2016-01-05T09:33:03Z” level = info msg = “[JSVM] [LOG]: {"MyCount":"15"}”
time = “2016-01-05T09:33:13Z” level = info msg = “[JSVM] [LOG]: My Count POST PROCESSOR”
time = “2016-01-05T09:33:13Z” level = info msg = “[JSVM] [LOG]: {"MyCount":"15"}”
time = “2016-01-05T09:33:13Z” level = info msg = “[JSVM] [LOG]: {"MyCount":"16"}”
time = “2016-01-05T09:33:21Z” level = info msg = “[JSVM] [LOG]: My Count POST PROCESSOR”
time = “2016-01-05T09:33:21Z” level = info msg = “[JSVM] [LOG]: {"MyCount":"15"}”
time = “2016-01-05T09:33:21Z” level = info msg = “[JSVM] [LOG]: {"MyCount":"16"}”
time = “2016-01-05T09:33:22Z” level = info msg = “[JSVM] [LOG]: My Count POST PROCESSOR”
time = “2016-01-05T09:33:22Z” level = info msg = “[JSVM] [LOG]: {"MyCount":"15"}”
time = “2016-01-05T09:33:22Z” level = info msg = “[JSVM] [LOG]: {"MyCount":"16"}”
time = “2016-01-05T09:33:23Z” level = info msg = “[JSVM] [LOG]: My Count POST PROCESSOR”
time = “2016-01-05T09:33:23Z” level = info msg = “[JSVM] [LOG]: {"MyCount":"16"}”
time = “2016-01-05T09:33:23Z” level = info msg = “[JSVM] [LOG]: {"MyCount":"17"}”

Is Session have a delay update or something?

Thanks

You received this message because you are subscribed to the Google Groups “Tyk Community Support” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web, visit https://groups.google.com/d/msgid/tyk-community-support/1850860e-49f0-4b51-a595-b945f6a87982%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Imported Google Group message.
Sender:Jelda Lam.
Date:Thursday, 7 January 2016 04:49:54 UTC.

Thank you very much Martin. You did a very good support.