This statement in the docs says
Extensibility: JSVM is a limited interpreter, although it can use some NPM modules, it isn’t NodeJS so writing interoperable code (especially with other DBs) is difficult.
My question is, is there any more specificity on what those “some npm modules” are? How can I levarage NPM modules libs in the middleware/*.js files, install them across all gateway nodes etc. I don’t see any docs on the proper Tyk procedures for this kind of thing
Right, we should clarify this in docs. Basically on start, Tyk loads all js files inside middleware directory. If node module has no “node” dependencies like file/network access, e.g. written purely in javascript, you can compile it to the single js file, and put to middleware folder. We while I say that JS should not have network/file functionality, similar to how nodejs works, we include some helpers (defined on server side), for doing networks calls and logging //tyk.io/docs/plugins/supported-languages/javascript-middleware/javascript-api/.
Worth noticing that JSVM and VirtualPath built-in to Tyk runs inside Tyk process, and while it is fast enough it indeed not as customizable as pure Node environment.
Note that in addition to built-in JS environment, Tyk offers 2 powerful rich plugins types. First is Python plugins, using built-in Python interpreter. And second is gRPC based plugins: so you can write middleware in any language/environment you want, the only requirment: it should support gRPC communication protocol.
See more info here https://tyk.io/docs/customise-tyk/plugins/rich-plugins/
Hope it makes sense!
Leonid, Tyk team
Thanks, yes I’ve already read the rich plugin section.
So… if I wanted to get a JVSM middleware JS file to leverage an NPM module for LDAP… what is the process to make that module available?
So, if NPM module do not use C modules, do not use Node specific functions (the same as its dependencies), you can bundle module to single file with tools like browserify javascript - Compile an npm module into a single file, without dependencies - Stack Overflow
Note that in case of LDAP module, it for sure use network request calls, and from Tyk side expose only TykMakeHttpRequest to the JS interpreter, so you will need to write quite big compatibility layer to make it work…
So, just to clarify, you want use LDAP for authentification to API?
I’m pretty sure that JSVM in this case not the best idea. What will work way better is a Tyk Identity Broker GitHub - TykTechnologies/tyk-identity-broker: Tyk Authentication Proxy for third-party login which build exactly for this case and has built-in LDAP functionality.
Yeah I’ve looked at the Identity Broker, but I think its design is more applicable for an entirely different flow that what I am talking about (i.e. basically a authn backend for existing token based flows, not basic auth) (i.e. based on the diagram here: //tyk.io/docs/tyk-identity-broker/)
My use case is as follows:
a) User sends an API request w/ a basic-auth header to an exposed API on a gateway
b) I need to take that basic auth uname/pw and validate it against ldap
c) Return a 401 on authn fail, otherwise let the request proceeed
Please correct me if I am wrong, but the identity broker as I understand it, is not really a solution for this scenario.
I need some sort of middleware hook that can do this against ldap, I don’t see anything out of the box in Tyk which just supports this. (other than the IDP support for LDAP for securing the dashboard/portal only, but not just a user-defined API the gateways serve up)
Simply not possible w/ id broker: Basic Auth against LDAP integration · Issue #11 · TykTechnologies/tyk-identity-broker · GitHub
You need a custom auth handler, which you need to write with JS or with a Rich Plugin.
This repository here has a demo go-based gRPC auth plugin. but this can also be done in Python or JSVM.
Auth plugins make it possible to cache auth data so you can save on round trips of the validation. The docs aren;t very complete at the moment (we’re working on that), but we’re more than happy to help out.
M.
There’s an [example JSVM auth plugin here]
(https://github.com/TykTechnologies/tyk/blob/master/middleware/otto_auth_mw_example.js)
Cool, yes I’ve already written a JSVM middleware auth plugin that calls a remote http service to do the auth…
So does the creation and return of that custom “session” object effectively cache the auth and prevent more remote calls to the backend?
So if stateless clients are calling my API, and my middleware hook does something w/ the basic auth against a remote system to get back a 200/401. How would session management work? How does Tyk correlate a 2nd request from that REST client passing the Authorization header again, to this session that would be established. (i.e. what would the session key be then that is stored in redis?)
No you need to configure that cache, what the call does is create a session object for use during the request. You need to add an ID extractor to give Tyk a place to look for unique ID information that it can use to cache:
,
"id_extractor": {
"extract_from": "header",
"extract_with": "value",
"extractor_config": {
"header_name": "Authorization"
}
},
If this is a basic auth header for example, with the above config, Tyk will MD5 that header, then check for it in the cache, if there’s a miss, it runs the auth middleware.
If it returns a session object, then Tyk will cache that object against the header and the next time the request comes through, there’s no need for a round-trip.
It’s advisable to force cache timeouts for these, you can do this with:
session_lifetime: The session lifetime will override the expiry date if it has been set on a key (in seconds). for example, if a key has been created that never expires, then it will remain in the session cache forever unless manually deleted. If a re-auth needs to be forced or a default expiry needs to be applied to all keys, then use this feature to set the session expiry for an entire API.
In the API Definition, or you can set:
"global_session_lifetime": 10,
"force_global_session_lifetime": true,
In your tyk.conf, but this is a global setting so applies to all API tokens
If you do not use the ID extractor, you are responsible for tracking the session object yourself, so for example you could use the Tyk plugin API call set_data() to write a key to redis with your session object, then do the same thing that the ID extractor does, just in your function. Though I don’t think this API is available to JSVM yet, only Python and Lua.
Thanks for that detail. I guess its unclear to me what defines a normal middleware JS plugin VS an “auth” middleware one, not sure what makes that differentiation as it pertains to setup of it.
Here is my example:
A) I have an custom API defined via the portal dashboard, w/ api_id “bcd4dcea7…”
B) I created a custom middleware JS file on the gateway under middleware/bcd4dcea7…/pre/myAuth.js
It looks like this
log("====> myAuth Auth initialising");
var myAuth = new TykJS.TykMiddleware.NewMiddleware({});
myAuth.NewProcessRequest(function(request, session) {
log("----> Running myAuth JSVM Auth Middleware");
var rawAuthorization = request.Headers["Authorization"];
if (rawAuthorization && rawAuthorization.length > 0) {
rawAuthorization = rawAuthorization[0];
} else {
request.ReturnOverrides.ResponseCode = 401;
request.ReturnOverrides.ResponseError = 'Authentication required';
return myAuth.ReturnData(request, {});
}
log(rawAuthorization);
var newRequest = {
"Method": "GET",
"Body": "",
"Headers": {
"Authorization":rawAuthorization
},
"Domain": "http://myauthserver:8080",
"Resource": "/auth/doit"
};
var resp = TykMakeHttpRequest(JSON.stringify(newRequest));
var respObj = JSON.parse(resp);
log(respObj.Code);
if (respObj.Code != 200) {
request.ReturnOverrides.ResponseCode = 401;
request.ReturnOverrides.ResponseError = 'MY Auth Failed: ' + respObj.code;
return myAuth.ReturnData(request, {});
}
return myAuth.ReturnAuthData(request, session);
});
// Ensure init with a post-declaration log message
log("====> myAuth initialised");
C) Where should I be configuring that “ID extractor” via the dashboard or gateways to denote that the “id extractor” you have above should be used for my API w/ id “bcd4dcea7…”? Is that in mongo on a per-api config definition? Or forced globally? What if only some of my APIs have the need for my custom basic auth plugin + the id extractor?
Nowadays you just use a bundle and the tyk-cli. So all you need is a manifest file and your files, then Tyk handles the rest.
With the bundle, you can set the bundle ID in the dashboard or in the API definition as specified in the doc mentioned above. This is probably the easiest way to deploy code.
However you can also manually do it in the API Definition custom_middleware section, though what’s missing in this doc is the classes to use (it only has pre and post), for auth you would use:
"custom_middleware": {
"auth_check": [
{
"name": "myAuth",
"path": "middleware/myAuth.js"
}
],
The syntax is the same as the manifests, here’s the one for the python demo. You just need to define the files.
The ID Extractor goes into the custom_middleware section of your API Definition.
Since it goes into the API definition, it only applies to that API.
So, my API definition JSON config is in Mongo… is directly modifying it there to add the “custom_middleware” block advised if I am using the dashboard to manage/create the API? Or import/export the config after editing it?
Part of what I am doing is evaluating this for use by non-developers in a “hybrid” setup so I need to understand the Tyk recommended procedure for doing such a thing.
I tried this method by adding an “auth” folder under my path on the gateway but it does not seem to work…is this not supported? (my middleware IS invoked if in “pre”) https://tyk.io/docs/customise-tyk/plugins/javascript-middleware/install-middleware/tyk-hybrid/
I’m just running the Docker quickstart: https://github.com/TykTechnologies/tyk_quickstart/blob/master/docker-compose.yml
Actually cancel that, the “/auth/myAuth.js” was invoked after changing the “Authentication mode” in the dashboard for the API to be “Use Custom Auth (plugin)”
Ok so here is what I have, honestly I feel like the docs are not complete or I am missing something as things are just not tying together for me.
a) My API definition:
{
"id": "5970bd3ae4b3480001d6a0e0",
"name": "MY Auth",
"slug": "my/auth/10",
"api_id": "06ca7b5dfbaf4d2d59a2eaa81cf8ca1c",
"org_id": "596cc1d5e4b3480001d6a0d9",
"use_keyless": false,
"use_oauth2": false,
"use_openid": false,
"openid_options": {
"providers": [],
"segregate_by_client": false
},
"oauth_meta": {
"allowed_access_types": [],
"allowed_authorize_types": [],
"auth_login_redirect": ""
},
"auth": {
"use_param": false,
"param_name": "",
"use_cookie": false,
"cookie_name": "",
"auth_header_name": ""
},
"use_basic_auth": false,
"enable_jwt": false,
"use_standard_auth": false,
"enable_coprocess_auth": true,
"jwt_signing_method": "",
"jwt_source": "",
"jwt_identity_base_field": "",
"jwt_client_base_field": "",
"jwt_policy_field_name": "",
"notifications": {
"shared_secret": "",
"oauth_on_keychange_url": ""
},
"enable_signature_checking": false,
"hmac_allowed_clock_skew": -1,
"base_identity_provided_by": "",
"definition": {
"location": "header",
"key": "x-api-version"
},
"version_data": {
"not_versioned": true,
"versions": {
"Default": {
"name": "Default",
"expires": "",
"paths": {
"ignored": [],
"white_list": [],
"black_list": []
},
"use_extended_paths": true,
"extended_paths": {},
"global_headers": {},
"global_headers_remove": [],
"global_size_limit": 0,
"override_target": ""
}
}
},
"uptime_tests": {
"check_list": [],
"config": {
"expire_utime_after": 0,
"service_discovery": {
"use_discovery_service": false,
"query_endpoint": "",
"use_nested_query": false,
"parent_data_path": "",
"data_path": "",
"port_data_path": "",
"target_path": "",
"use_target_list": false,
"cache_timeout": 60,
"endpoint_returns_list": false
},
"recheck_wait": 0
}
},
"proxy": {
"preserve_host_header": false,
"listen_path": "/my/auth/10",
"target_url": "http://192.168.0.148:9080/service/authws",
"strip_listen_path": true,
"enable_load_balancing": false,
"target_list": [],
"check_host_against_uptime_tests": false,
"service_discovery": {
"use_discovery_service": false,
"query_endpoint": "",
"use_nested_query": false,
"parent_data_path": "",
"data_path": "hostname",
"port_data_path": "port",
"target_path": "/api-slug",
"use_target_list": false,
"cache_timeout": 60,
"endpoint_returns_list": false
}
},
"disable_rate_limit": false,
"disable_quota": false,
"custom_middleware": {
"pre": [],
"post": [],
"post_key_auth": [],
"auth_check": {
"name": "myAuth",
"path": "middleware/myAuth.js",
"require_session": false
},
"response": [],
"driver": "",
"id_extractor": {
"extract_from": "header",
"extract_with": "value",
"extractor_config": {
"header_name": "Authorization"
}
}
},
"custom_middleware_bundle": "",
"cache_options": {
"cache_timeout": 60,
"enable_cache": true,
"cache_all_safe_requests": false,
"cache_response_codes": [],
"enable_upstream_cache_control": false
},
"session_lifetime": 1,
"active": true,
"auth_provider": {
"name": "",
"storage_engine": "",
"meta": {}
},
"session_provider": {
"name": "",
"storage_engine": "",
"meta": null
},
"event_handlers": {
"events": {}
},
"enable_batch_request_support": false,
"enable_ip_whitelisting": false,
"allowed_ips": [],
"dont_set_quota_on_create": false,
"expire_analytics_after": 0,
"response_processors": [],
"CORS": {
"enable": false,
"allowed_origins": [],
"allowed_methods": [],
"allowed_headers": [],
"exposed_headers": [],
"allow_credentials": false,
"max_age": 24,
"options_passthrough": false,
"debug": false
},
"domain": "",
"do_not_track": false,
"tags": [],
"enable_context_vars": false
}
b) My plugin (under middleware/myAuth.js) on the gateway node
log("====> myAuth Auth initialising");
var myAuth = new TykJS.TykMiddleware.NewMiddleware({});
myAuth.NewProcessRequest(function(request, session) {
log("----> Running myAuth JSVM Auth Middleware");
var rawAuthorization = request.Headers["Authorization"];
log("session= " + JSON.stringify(session));
if (rawAuthorization && rawAuthorization.length > 0) {
rawAuthorization = rawAuthorization[0];
} else {
request.ReturnOverrides.ResponseCode = 401;
request.ReturnOverrides.ResponseError = 'Authentication required';
return myAuth.ReturnData(request, {});
}
log(rawAuthorization);
var newRequest = {
"Method": "GET",
"Body": "",
"Headers": {
"Authorization":rawAuthorization,
},
"Domain": "http://192.168.0.148:9080",
"Resource": "/auth/util/doit"
};
var resp = TykMakeHttpRequest(JSON.stringify(newRequest));
var respObj = JSON.parse(resp);
log(respObj.Code);
if (respObj.Code != 200) {
request.ReturnOverrides.ResponseCode = 401;
request.ReturnOverrides.ResponseError = 'MY Auth Failed: ' + respObj.code;
return myAuth.ReturnData(request, {});
}
var newSession = {
"allowance": 100,
"rate": 100,
"per": 1,
"quota_max": -1,
"quota_renews": 1406121006,
"access_rights": {}
};
return myAuth.ReturnAuthData(request, newSession);
});
// Ensure init with a post-declaration log message
log("====> myAuth initialised");
c) My first request to the endpoint, I get through w/ a valid Authorization header (i.e. the plugin hits my backend auth API ok) and this is logged
2017-07-20T14:35:21.391318719Z time="Jul 20 14:35:21" level=info msg="----> Running myAuth JSVM Auth Middleware" type=log-msg
2017-07-20T14:35:21.391853783Z time="Jul 20 14:35:21" level=info msg="session= {\"access_rights\":null,\"alias\":\"\",\"allowance\":0,\"apply_policy_id\":\"\",\"basic_auth_data\":{\"hash_type\":\"\",\"password\":\"\"},\"data_expires\":0,\"enable_detail_recording\":false,\"expires\":0,\"hmac_enabled\":false,\"hmac_string\":\"\",\"id_extractor_deadline\":0,\"is_inactive\":false,\"jwt_data\":{\"secret\":\"\"},\"last_check\":0,\"last_updated\":\"\",\"meta_data\":null,\"monitor\":{\"trigger_limits\":null},\"oauth_client_id\":\"\",\"oauth_keys\":null,\"org_id\":\"\",\"per\":0,\"quota_max\":0,\"quota_remaining\":0,\"quota_renewal_rate\":0,\"quota_renews\":0,\"rate\":0,\"session_lifetime\":0,\"tags\":null}" type=log-msg
2017-07-20T14:35:21.391925198Z time="Jul 20 14:35:21" level=info msg="Basic XXXX==" type=log-msg
2017-07-20T14:35:22.474491045Z time="Jul 20 14:35:22" level=info msg=200 type=log-msg
d) My 2nd request to the endpoint (i would expect session to be populated now?) But its not, just looks like another empty one (logs below)
2017-07-20T14:35:40.263778912Z time="Jul 20 14:35:40" level=info msg="----> Running myAuth JSVM Auth Middleware" type=log-msg
2017-07-20T14:35:40.264316972Z time="Jul 20 14:35:40" level=info msg="session= {\"access_rights\":null,\"alias\":\"\",\"allowance\":0,\"apply_policy_id\":\"\",\"basic_auth_data\":{\"hash_type\":\"\",\"password\":\"\"},\"data_expires\":0,\"enable_detail_recording\":false,\"expires\":0,\"hmac_enabled\":false,\"hmac_string\":\"\",\"id_extractor_deadline\":0,\"is_inactive\":false,\"jwt_data\":{\"secret\":\"\"},\"last_check\":0,\"last_updated\":\"\",\"meta_data\":null,\"monitor\":{\"trigger_limits\":null},\"oauth_client_id\":\"\",\"oauth_keys\":null,\"org_id\":\"\",\"per\":0,\"quota_max\":0,\"quota_remaining\":0,\"quota_renewal_rate\":0,\"quota_renews\":0,\"rate\":0,\"session_lifetime\":0,\"tags\":null}" type=log-msg
2017-07-20T14:35:40.264381096Z time="Jul 20 14:35:40" level=info msg="Basic XXXX==" type=log-msg
2017-07-20T14:35:40.950387636Z time="Jul 20 14:35:40" level=info msg=200 type=log-msg
e) Assuming a session would be established after the FIRST successful invocation, and the session key properly based on the Authorization header per the “id_extractor” config… will Tyk still call the auth plugin on every invocation and it is up to my plugin to determine (based on something in the session) if it should call back again to the auth endpoint? OR will tyk upon seeing a valid session SKIP calling my plugin, and only call again when the key expires.
Perhaps something is still missing in my config?
This is where the bundles come in, with a bundle you just specify the bundle ID in the dashboard and it gets downloaded, validated and installed for you without having to add stuff manually.
The dashboard doesn’t support adding the block manually, you would need to use the rest api.
or use a bundle… which is he simplest method.
For your other questions I’ve handed over to our plugin expert, he’ll respond shortly.
thanks!
Yes I’ll eventually get to the bundled packaging, for now I’m just trying to get something working to prove it out as part of this eval. Its definitely being invoked, just the session part I’m hung/blocked by right now as described above in my last post
FYI, just upgraded to the latest of the docker quickstart which is showing 1.3.7 (but same result, seeing auth plugin continually re-invoked)