Thanks for your reply.
The log did not indicate anything suspicions. The ulimit on open file handles was properly set to be 80000 as well as `security/limits.conf.
However, I have tried your suggestions, namely adding the lines you have specified into the configuration file and as well as turning off syslog completely and I can now see substantial improvement for a 3 minute load test:
Running 3m test @ http://tyk01.us.ad.mydomain.com:8080/v1/deals?api_key=58b76d8c5e321209fcc2ab79b6d59fc6ae6d4ed3700d76ed116c54cf
1 threads and 100 connections
Thread calibration: mean lat.: 8.230ms, rate sampling interval: 45ms
Thread Stats Avg Stdev Max +/- Stdev
Latency 249.05ms 865.12ms 5.28s 92.72%
Req/Sec 3.04k 1.45k 7.93k 67.67%
Latency Distribution (HdrHistogram - Recorded Latency)
50.000% 2.93ms
75.000% 27.63ms
90.000% 96.38ms
99.000% 4.57s
99.900% 5.07s
99.990% 5.17s
99.999% 5.24s
100.000% 5.29s
The api is defined via a json file in gui + caching turned on in options there:
1 {
2 "id": "58a4fe196404310a035c7174",
3 "name": "Deal Service",
4 "slug": "deal-service",
5 "api_id": "c26c389462a9435375602711e50794f5",
6 "org_id": "58990a186404311bde66ab64",
7 "use_keyless": false,
8 "use_oauth2": false,
9 "use_openid": false,
10 "openid_options": {
11 "providers": [],
12 "segregate_by_client": false
13 },
14 "oauth_meta": {
15 "allowed_access_types": [],
16 "allowed_authorize_types": [],
17 "auth_login_redirect": ""
18 },
19 "auth": {
20 "use_param": true,
21 "param_name": "",
22 "use_cookie": false,
23 "cookie_name": "",
24 "auth_header_name": "api_key"
25 },
26 "use_basic_auth": false,
27 "enable_jwt": false,
28 "use_standard_auth": true,
29 "enable_coprocess_auth": false,
...and so on...
So the result looks somtehing like:
Uploading…
Uploading…
So something happens at 60%'s percentile after which there is a significant growth in latency. It becomes more apparent when I do 10 minute tests. The latency reaches around 45ms for 60%. Any ideas about what could be the source of it? It does not correspond to cpu or memory load on the server.