I don’t know exactly from which version (maybe 2.3.3) our tyk installation starts to consume larges amount of memory over time without any decrease.
I see this behavior only in our production redis cluster, in the other environments with only a unique redis server this won’t happen. The memory it’s used and released.
Actually we are using tyk 2.3.6 with a redis cluster version 3.0.7, and in a few days the tyk program goes from 2% to the 80% of a 4Gb Virtual Machine.
Any thoughts about ?
Do you have health check api enabled?
Actually no, I disable it after seeing one of your messages about it
Is there anything in the Tyk logs, such as warnings or errors of any kind?
How fast does the memory get eaten up? Does it always happen and at the same speed?
We haven’t seen this issue ourselves yet, so it would be helpful if you could give more information - like your tyk.conf - to be able to reproduce it reliably.
Thank you!
Can you share your tyk.conf please?
{ "listen_port": 8080, "secret": "********************************", "node_secret": "", "template_path": "./templates", "tyk_js_path": "./js/tyk.js", "middleware_path": "./middleware", "policies": { "policy_source": "", "policy_connection_string": "", "policy_record_name": "" }, "use_db_app_configs": false, "db_app_conf_options": { "connection_string": "", "node_is_segmented": false, "tags": null }, "app_path": "./apps/", "storage": { "type": "redis", "enable_cluster": true, "hosts": { "api1.local": "6379", "api2.local": "6379", "api3.local": "6379", "api1.local": "6479", "api2.local": "6479", "api3.local": "6479" }, "username": "", "password": "", "database": 0, "optimisation_max_idle": 2000, "optimisation_max_active": 4000 }, "enable_analytics": true, "analytics_config": { "type": "", "ignored_ips": [], "enable_detailed_recording": false, "enable_geo_ip": false, "geo_ip_db_path": "" }, "health_check": { "enable_health_checks": false, "health_check_value_timeouts": 60 }, "optimisations_use_async_session_write": false, "allow_master_keys": false, "hash_keys": false, "suppress_redis_signal_reload": false, "suppress_default_org_store": false, "sentry_code": "", "use_sentry": false, "enforce_org_data_age": false, "enforce_org_data_detail_logging": false, "enforce_org_quotas": false, "experimental_process_org_off_thread": false, "enable_non_transactional_rate_limiter": false, "enable_sentinel_rate_limiter": false, "Monitor": { "enable_trigger_monitors": false, "configuration": { "method": "", "target_path": "", "template_path": "", "header_map": null, "event_timeout": 0 }, "global_trigger_limit": 0, "monitor_user_keys": false, "monitor_org_keys": false }, "oauth_refresh_token_expire": 0, "oauth_token_expire": 0, "slave_options": { "use_rpc": false, "connection_string": "", "rpc_key": "", "api_key": "", "enable_rpc_cache": false, "bind_to_slugs": false, "disable_keyspace_sync": false, "group_id": "" }, "disable_virtual_path_blobs": false, "local_session_cache": { "disable_cached_session_state": false, "cached_session_timeout": 0, "cached_session_eviction": 0 }, "http_server_options": { "override_defaults": false, "read_timeout": 0, "write_timeout": 0, "use_ssl": false, "certificates": null, "server_name": "", "min_version": 0, "flush_interval": 0 }, "service_discovery": { "default_cache_timeout": 0 }, "close_connections": false, "auth_override": { "force_auth_provider": false, "auth_provider": { "name": "", "storage_engine": "", "meta": null }, "force_session_provider": false, "session_provider": { "name": "", "storage_engine": "", "meta": null } }, "uptime_tests": { "disable": true, "config": { "failure_trigger_sample_size": 0, "time_wait": 0, "checker_pool_size": 0, "enable_uptime_analytics": false } }, "hostname": "", "enable_api_segregation": false, "control_api_hostname": "", "enable_custom_domains": false, "enable_jsvm": true, "hide_generator_header": false, "event_handlers": { "events": null }, "event_trigers_defunct": null }
After a restart, in one hour it goes from 400Mb to 900Mb
Ok - so you are using a very old rate limiter, the one you are using will lock traffic against redis connections, which can degrade performance and cause this behaviour in newer versions of tyk (it is a deprecated feature still included for backwards compatibility).
Take a look at the section “use the optimisation settings” on this link:
https://tyk.io/docs/deploy-tyk-premise-production/
And set those options as they are in the snippet.
The main thing here is you have disabled the non-transactional rate limiter.
This should clean up your memory consumption.
Thanks, I try it and let you know …
After some more digging, the memory increase usage happens in all our instances (with redis cluster or not)
The machines without a redis cluster respawn after a crash (ubuntu 14.04 upstart)
The production machines with redis-cluster (Ubuntu 16.04) don’t respawn and crash
With the latest optimizations from the docs, the memory increase happens again taking a whole 4Gb in a few hours.
We rollback to version 2.3.4 with the same tyk.conf and ALL seems stable right now !
Oscillating from 1.9% to 2.5% memory usage in a 4Gb virtual machine
We use javascript middleware in almost all APIs to increase key expiration time.
I read the releases page to see 2.3.5 changes, but and I don’t know if the leak was found it and patched in this new version.
This makes it look like the leak is somehow related to the javascript middleware. Does the leak persist if you remove the javascript middleware from the API?
It would also be helpful if you could provide the API definition and js middleware code to reproduce this leak.
Thank you!
This is our js middleware:
`var refreshSecurity = new TykJS.TykMiddleware.NewMiddleware({});
refreshSecurity.NewProcessRequest(function(request, session) {
if (session.expires > 0){
session.expires = ((Date.now() / 1000) | 0) + 1800;
TykSetKeyData(request.Headers[“Security”], JSON.stringify(session), 1);
}
return refreshSecurity.ReturnData(request);
});
log(“refreshSecurity POST middleware initialised”);`
And this is our main API
`{
“id”: “******************000001”,
“name”: “Security”,
“slug”: “security”,
“api_id”: “**************************000001”,
“org_id”: “******************000001”,
“use_keyless”: false,
“use_oauth2”: false,
“auth”: {
“use_param”: false,
“use_cookie”: false,
“auth_header_name”: “security”
},
“use_basic_auth”: false,
“enable_jwt”: false,
“enable_signature_checking”: false,
“hmac_allowed_clock_skew”: -1,
“definition”: {
“location”: “header”,
“key”: “x-api-version”
},
“version_data”: {
“not_versioned”: false,
“versions”: {
“v1”: {
“name”: “v1”,
“expires”: “”,
“override_target”: “http://java/security-v1”,
“use_extended_paths”: true,
"extended_paths": {
"ignored": [{"method_actions": {"GET": {"code": 200, "action": "no_action"}}, "path": "test"}, {"method_actions": {"GET": {"code": 200, "action": "no_action"}}, "path": "resources"}, {"method_actions": {"GET": {"code": 200, "action": "no_action"}}, "path": "authorization/***"}, {"method_actions": {"GET": {"code": 200, "action": "no_action"}}, "path": "authorization/***"}, {"method_actions": {"GET": {"code": 200, "action": "no_action"}}, "path": "authorization/***"}, {"method_actions": {"POST": {"code": 200, "action": "no_action"}}, "path": "authorization/***"}, {"method_actions": {"POST": {"code": 200, "action": "no_action"}}, "path": "authorization/***"}, {"method_actions": {"POST": {"code": 200, "action": "no_action"}}, "path": "authorization/***"}, {"method_actions": {"POST": {"code": 200, "action": "no_action"}}, "path": "authorization/***"}, {"method_actions": {"POST": {"code": 200, "action": "no_action"}}, "path": "authorization/timeout"}]
},
"global_headers": {
"x-env": "***"
}
}
}
},
"proxy": {
"listen_path": "/security",
"target_url": "",
"strip_listen_path": true,
"enable_load_balancing": false,
"check_host_against_uptime_tests": false
},
"custom_middleware": {
"post": [
{
"name": "refreshSecurity",
"path": "/opt/tyk/middleware/refreshSecurity.js",
"require_session": true
}
]
},
"cache_options": {
"cache_timeout": 60,
"enable_cache": true,
"cache_all_safe_requests": false,
"enable_upstream_cache_control": false
},
"session_lifetime": 0,
"active": true,
"event_handlers": {
"events": {
"KeyExpired": [
{
"handler_name": "eh_web_hook_handler",
"handler_meta": {
"_id": "************000001000001",
"event_timeout": 10,
"header_map": {
"x-env": "***",
"x-api-version": "v1"
},
"method": "POST",
"name": "ExpireKey",
"org_id": "******************000001",
"target_path": "https://api/security/authorization/timeout",
"template_path": "/opt/tyk/templates/default_webhook.json"
}
}
]
}
},
"enable_non_transactional_rate_limiter": true,
"enable_batch_request_support": false,
"enable_ip_whitelisting": false,
"dont_set_quota_on_create": false,
"expire_analytics_after": 0,
"CORS": {
"enable": true,
"allowed_origins": ["*"],
"allowed_methods": ["GET", "OPTIONS", "POST", "DELETE", "PUT"],
"allowed_headers": [
"accept",
"content-type",
"x-api-version",
"security"
],
"exposed_headers": [
"x-generator",
"security"
],
"allow_credentials": true,
"max_age": 24,
"options_passthrough": false,
"debug": false
},
"domain": "api",
"do_not_track": false,
"tags": []`
Does the leak persist if you remove the JS middleware?
I can’t do that easily…
I take it that the very same JS middleware works fine without any memory leaks on 2.3.4? What about 2.3.5?
The narrower the scope of the search, the faster we can get to the culprit 
Yes correct 2.3.4 works without ‘leaks’ , with 2.3.5/2.3.6 we have the memory consumption problem…
Great, thanks. I’ll investigate what changes were made in 2.3.5 and report back any findings.
I switch from 2.3.4 to 2.3.6 at 14:00 tyk process start using 1.6% memory from 4Gb machine.
In one hour of lower usage (most people are eating) at 15:00 the process use 6% of memory.
All morning with 2.3.4 don’t surpass 2.5%
Hi xagaba,
I’ve been able to reproduce the memory leak in both master and 2.3.5. This looks like a goroutine leak introduced by a JSVM fix added in 2.3.5.
I’ve opened an issue here: JSVM leak introduced in 2.3.5 · Issue #804 · TykTechnologies/tyk · GitHub
We will release 2.3.7 with a fix - please stay tuned.
Thanks!