Tyk load test on ec2 instances

Namrata_Lele · January 26, 2017, 5:57am

Hello,

We have the following setup for tyk in AWS
m4.xlarge for the redis and mongodb boxes; c4.xlarge for the others; all in us-west-2a (same AZ); all in the same VPC

tyk gateway on one ec2 instance 2) tyk pump and dashboard on another ec2 instance
3 nginx upstreams setup on upstream hosts
mongodb on another ec2 instance
redis on another ec2 instance

We are running load test using apache benchmark command similar to this :
ab -c 100 -e output/tyk-2.2017-01-26T05:36:49+0000.c10-keepalive.csv -t 30 -k http://tyk-2:8080/upstream/

We are observing high latency for tyk compared to other gateways that we have setup in AWS.
It seems way off than the benchmarks posted. Are we missing something in the setup ?

Done! ============ Results ==============
Testing tyk-2 with concurrency 100 (started at 2017-01-26T05:39:26+0000)
output/tyk-2.2017-01-26T05:39:26+0000.c100.txt
Complete requests 32780
Failed requests 0
Requests per second 1092.01 [#/sec] (mean)
50 %ile (ms) 12.524
66 %ile (ms) 16.443
75 %ile (ms) 24.620
80 %ile (ms) 34.217
90 %ile (ms) 220.688
95 %ile (ms) 503.755
98 %ile (ms) 1082.949
99 %ile (ms) 1267.516
100 %ile (ms) 15052.472
Testing tyk-2 with concurrency 100 -keepalive (started at 2017-01-26T05:39:56+0000)
output/tyk-2.2017-01-26T05:39:56+0000.c100-keepalive.txt
Complete requests 31877
Failed requests 0
Requests per second 1062.55 [#/sec] (mean)
50 %ile (ms) 12.403
66 %ile (ms) 16.238
75 %ile (ms) 22.344
80 %ile (ms) 28.235
90 %ile (ms) 207.803
95 %ile (ms) 467.641
98 %ile (ms) 1444.592
99 %ile (ms) 1874.346
100 %ile (ms) 3942.286

Thanks,
Namrata

Martin · January 26, 2017, 7:06am

Hi Namrata,

Can you share your tyk.conf?

Many thanks,
Martin

Martin · January 26, 2017, 7:13am

Also, have followed some of the guidelines in our production guide?

https://tyk.io/tyk-documentation/deploy-tyk-premise-production/

dmankin · January 26, 2017, 5:55pm

Here’s our config:

{
    "listen_port": 8080,
    "secret": "xx",
    "node_secret": "xx",
    "template_path": "/opt/tyk-gateway/templates",
    "tyk_js_path": "/opt/tyk-gateway/js/tyk.js",
    "middleware_path": "/opt/tyk-gateway/middleware",
    "use_db_app_configs": true,
    "db_app_conf_options": {
        "connection_string": "http://tyk-1:8001",
        "node_is_segmented": false,
        "tags": ["test2"]
    },
    "app_path": "/opt/tyk-gateway/apps/",
    "storage": {
        "type": "redis",
        "host": "redis-1",
        "port": 6379,
        "username": "",
        "password": "xx",
        "database": 0,
        "optimisation_max_idle": 100
    },
    "enable_analytics": true,
    "analytics_config": {
        "type": "mongo",
        "csv_dir": "/tmp",
        "mongo_url": "",
        "mongo_db_name": "",
        "mongo_collection": "",
        "purge_delay": -1,
        "ignored_ips": []
    },
    "health_check": {
        "enable_health_checks": true,
        "health_check_value_timeouts": 60
    },
    "optimisations_use_async_session_write": true,
    "enable_non_transactional_rate_limiter": true,
    "enable_sentinel_rate_limiter": false,
    "allow_master_keys": false,
    "policies": {
        "policy_source": "service",
        "policy_connection_string": "http://tyk-1:8001",
        "policy_record_name": "tyk_policies"
    },
    "hash_keys": true,
    "close_connections": true,
    "allow_insecure_configs": true,
     "coprocess_options": {
        "enable_coprocess": false,
        "coprocess_grpc_server": "tcp://tyk-grpc:50051"
    },
    "enable_bundle_downloader": false,
    "bundle_base_url": "http://e204ee17.ngrok.io/",
    "global_session_lifetime": 100,
    "force_global_session_lifetime": false,
    "max_idle_connections_per_host": 100
}

We’re running tyk in docker 1.12.6 on AMI linux

Your blog post has some interesting file descriptor settings we’ll apply.

That said, the blog post does not show any variance numbers. We’re especially concerned about the tyk performance in the “long tail” of 95th, 98th, and 99th percentile. The median looks good even with concurrency of 100 (which is hard to compare to “2000 users” — what does that really mean?), but the higher percentiles look bad.

Martin · January 26, 2017, 6:49pm

I see - we’ll take a look at your config to see where the tweaks are.

As for the numbers in the charts, we don’t have the percentile numbers I’m afraid, however the concurrency values are in the second (blue) charts under each latency graph, the y axis here is the number of requests per second, whereas in the top chart it is the number of “users”, which we agree isn’t a standard measure but the one that our load testing provider blitz.io provides.

Martin · January 27, 2017, 7:22am

Hi,

I’ve added the optimization settings that should improve performance, in particular the health checks are expensive and will cause Redis to be your bottleneck:

{
	"listen_port": 8080,
	"secret": "xx",
	"node_secret": "xx",
	"template_path": "/opt/tyk-gateway/templates",
	"tyk_js_path": "/opt/tyk-gateway/js/tyk.js",
	"middleware_path": "/opt/tyk-gateway/middleware",
	"use_db_app_configs": true,
	"db_app_conf_options": {
		"connection_string": "http://tyk-1:8001",
		"node_is_segmented": false,
		"tags": ["test2"]
	},
	"app_path": "/opt/tyk-gateway/apps/",
	"storage": {
		"type": "redis",
		"host": "redis-1",
		"port": 6379,
		"username": "",
		"password": "xx",
		"database": 0,
		"optimisation_max_idle": 3000,
		"optimisation_max_active": 5000
	},
	"enable_analytics": true,
	"analytics_config": {
		"type": "mongo",
		"csv_dir": "/tmp",
		"mongo_url": "",
		"mongo_db_name": "",
		"mongo_collection": "",
		"purge_delay": -1,
		"ignored_ips": []
	},
	"health_check": {
		"enable_health_checks": false,
		"health_check_value_timeouts": 60
	},
	"optimisations_use_async_session_write": true,
	"enable_non_transactional_rate_limiter": true,
	"enable_sentinel_rate_limiter": false,
	"allow_master_keys": false,
	"policies": {
		"policy_source": "service",
		"policy_connection_string": "http://tyk-1:8001",
		"policy_record_name": "tyk_policies"
	},
	"hash_keys": true,
	"close_connections": true,
	"allow_insecure_configs": true,
	"coprocess_options": {
		"enable_coprocess": false,
		"coprocess_grpc_server": "tcp://tyk-grpc:50051"
	},
	"enable_bundle_downloader": false,
	"bundle_base_url": "http://e204ee17.ngrok.io/",
	"global_session_lifetime": 100,
	"force_global_session_lifetime": false,
	"max_idle_connections_per_host": 300,
	"experimental_process_org_off_thread": true,
	"local_session_cache": {
		"disable_cached_session_state": false
	}
}

I hope this improves your results!

M.