API Key or Basic Auth Header is getting removed

saloni512 · November 15, 2021, 7:44am

HI Team,

I am facing an issue where after sometime time (may be in 1-2 min) my basic auth or key associated with the API getting removed after few minutes automatically. Even though I have setup expire time as -1 which means never expire:

curl -X POST -H “x-tyk-authorization: 352d20ee67be67f6340b4c0605b044b7”
-s
-H “Content-Type: application/json”
-X POST
-d ‘{
“allowance”: 1000,
“rate”: 1000,
“per”: 1,
“quota_max”: -1,
“expires”: -1,
“org_id”: “3”,
“quota_renews”: 1449051461,
“quota_remaining”: -1,
“quota_renewal_rate”: 60,
“access_rights”: {
“CGW-key-header”: {
“api_id”: “CGW-key-header”,
“api_name”: “CGW-key-header”,
“versions”: [
“Default”
]
}
},
“meta_data”: {}
}’ http://ec2-18-142-155-240.ap-southeast-1.compute.amazonaws.com:8080/tyk/keys/create | python -mjson.tool

Please suggest as its very tough to manage.

Olu · November 15, 2021, 9:46am

Could you try using zero and let us know the result.

saloni512 · November 15, 2021, 9:55am

I already tried with 0 but still same behavior. Not sure why

Please suggest something to fix.

Olu · November 15, 2021, 10:02am

Okay. Can you share your Tyk and Redis version?

Also, can you check the result of the /tyk/keys/{keyID} API and compare it with the value in Redis. Tyk uses redis to store the key information so It might be worth checking the TTL value in Redis as well.

Last but not least, can you cross check the org ID and API version are both valid. If they are then try removing the version info from the Key below

“versions”: [
“Default”
]

saloni512 · November 15, 2021, 10:14am

Below are versions:

Tyk API Gateway: v3.2.1
Redis Version: Redis server v=3.2.12 sha=00000000:0 malloc=jemalloc-3.6.0 bits=64 build=7897e7d0e13773f

org id i have put is 3 and version is default.

saloni512 · November 15, 2021, 10:30am

Even I created key by removing version as suggested but still the same behavior:

curl -X POST -H “x-tyk-authorization: 352d20ee67be67f6340b4c0605b044b7”
-s
-H “Content-Type: application/json”
-X POST
-d '{
“allowance”: 1000,
“rate”: 1000,
“per”: 1,
“quota_max”: -1,
“expires”: -1,
“org_id”: “3”,
“quota_renews”: 1449051461,
“quota_remaining”: -1,
“quota_renewal_rate”: 60,
“access_rights”: {
“CGW-key-header”: {
“api_id”: “CGW-key-header”,
“api_name”: “CGW-key-header”

  }
},
"meta_data": {}

}’ http://ec2-18-142-155-240.ap-southeast-1.compute.amazonaws.com:8080/tyk/keys/create | python -mjson.tool

saloni512 · November 15, 2021, 10:31am

Below is my api definition:

{
“name”: “CGW-key-header”,
“slug”: “CGW-key-header”,
“listen_port”: 0,
“protocol”: “”,
“enable_proxy_protocol”: false,
“api_id”: “CGW-key-header”,
“org_id”: “3”,
“use_keyless”: false,
“use_oauth2”: false,
“use_openid”: false,
“openid_options”: {
“providers”: null,
“segregate_by_client”: false
},
“oauth_meta”: {
“allowed_access_types”: null,
“allowed_authorize_types”: null,
“auth_login_redirect”: “”
},
“auth”: {
“use_param”: false,
“param_name”: “”,
“use_cookie”: false,
“cookie_name”: “”,
“auth_header_name”: “Authorization”,
“use_certificate”: false,
“validate_signature”: false,
“signature”: {
“algorithm”: “”,
“header”: “”,
“secret”: “”,
“allowed_clock_skew”: 0,
“error_code”: 0,
“error_message”: “”
}
},
“auth_configs”: null,
“use_basic_auth”: false,
“basic_auth”: {
“disable_caching”: false,
“cache_ttl”: 0,
“extract_from_body”: false,
“body_user_regexp”: “”,
“body_password_regexp”: “”
},
“use_mutual_tls_auth”: false,
“client_certificates”: null,
“upstream_certificates”: null,
“pinned_public_keys”: null,
“enable_jwt”: false,
“use_standard_auth”: false,
“use_go_plugin_auth”: false,
“enable_coprocess_auth”: false,
“jwt_signing_method”: “”,
“jwt_source”: “”,
“jwt_identity_base_field”: “”,
“jwt_client_base_field”: “”,
“jwt_policy_field_name”: “”,
“jwt_default_policies”: null,
“jwt_issued_at_validation_skew”: 0,
“jwt_expires_at_validation_skew”: 0,
“jwt_not_before_validation_skew”: 0,
“jwt_skip_kid”: false,
“jwt_scope_to_policy_mapping”: null,
“jwt_scope_claim_name”: “”,
“notifications”: {
“shared_secret”: “”,
“oauth_on_keychange_url”: “”
},
“enable_signature_checking”: false,
“hmac_allowed_clock_skew”: 0,
“hmac_allowed_algorithms”: null,
“request_signing”: {
“is_enabled”: false,
“secret”: “”,
“key_id”: “”,
“algorithm”: “”,
“header_list”: null,
“certificate_id”: “”,
“signature_header”: “”
},
“base_identity_provided_by”: “”,
“definition”: {
“location”: “header”,
“key”: “x-api-version”,
“strip_path”: false
},
“version_data”: {
“not_versioned”: true,
“default_version”: “”,
“versions”: {
“Default”: {
“name”: “Default”,
“expires”: “”,
“paths”: {
“ignored”: null,
“white_list”: null,
“black_list”: null
},
“use_extended_paths”: true,
“extended_paths”: {},
“global_headers”: {
“Authorization”: “Basic Y2liYWRtaW46YWRtaW4=”
},
“global_headers_remove”: [“Authorization”],
“global_response_headers”: null,
“global_response_headers_remove”: null,
“ignore_endpoint_case”: false,
“global_size_limit”: 0,
“override_target”: “”
}
}
},
“uptime_tests”: {
“check_list”: null,
“config”: {
“expire_utime_after”: 0,
“service_discovery”: {
“use_discovery_service”: false,
“query_endpoint”: “”,
“use_nested_query”: false,
“parent_data_path”: “”,
“data_path”: “”,
“port_data_path”: “”,
“target_path”: “”,
“use_target_list”: false,
“cache_timeout”: 0,
“endpoint_returns_list”: false
},
“recheck_wait”: 0
}
},
“proxy”: {
“preserve_host_header”: false,
“listen_path”: “/CloudGateway/rest/”,
“target_url”: “https://54.254.102.140:9443”,
“disable_strip_slash”: false,
“strip_listen_path”: false,
“enable_load_balancing”: false,
“target_list”: null,
“check_host_against_uptime_tests”: false,
“service_discovery”: {
“use_discovery_service”: false,
“query_endpoint”: “”,
“use_nested_query”: false,
“parent_data_path”: “”,
“data_path”: “”,
“port_data_path”: “”,
“target_path”: “”,
“use_target_list”: false,
“cache_timeout”: 0,
“endpoint_returns_list”: false
},
“transport”: {
“ssl_insecure_skip_verify”: false,
“ssl_ciphers”: null,
“ssl_min_version”: 0,
“ssl_max_version”: 0,
“ssl_force_common_name_check”: false,
“proxy_url”: “”
}
},
“disable_rate_limit”: false,
“disable_quota”: false,
“custom_middleware”: {
“pre”: null,
“post”: null,
“post_key_auth”: null,
“auth_check”: {
“name”: “”,
“path”: “”,
“require_session”: false,
“raw_body_only”: false
},
“response”: null,
“driver”: “”,
“id_extractor”: {
“extract_from”: “”,
“extract_with”: “”,
“extractor_config”: null
}
},
“custom_middleware_bundle”: “”,
“cache_options”: {
“cache_timeout”: 0,
“enable_cache”: false,
“cache_all_safe_requests”: false,
“cache_response_codes”: null,
“enable_upstream_cache_control”: false,
“cache_control_ttl_header”: “”,
“cache_by_headers”: null
},
“session_lifetime”: 0,
“active”: true,
“internal”: false,
“auth_provider”: {
“name”: “”,
“storage_engine”: “”,
“meta”: null
},
“session_provider”: {
“name”: “”,
“storage_engine”: “”,
“meta”: null
},
“event_handlers”: {
“events”: null
},
“enable_batch_request_support”: false,
“enable_ip_whitelisting”: false,
“allowed_ips”: null,
“enable_ip_blacklisting”: false,
“blacklisted_ips”: null,
“dont_set_quota_on_create”: false,
“expire_analytics_after”: 0,
“response_processors”: null,
“CORS”: {
“enable”: false,
“allowed_origins”: null,
“allowed_methods”: null,
“allowed_headers”: null,
“exposed_headers”: null,
“allow_credentials”: false,
“max_age”: 0,
“options_passthrough”: false,
“debug”: false
},
“domain”: “”,
“certificates”: null,
“do_not_track”: false,
“tags”: null,
“enable_context_vars”: false,
“config_data”: null,
“tag_headers”: null,
“global_rate_limit”: {
“rate”: 0,
“per”: 0
},
“strip_auth_data”: false,
“enable_detailed_recording”: false,
“graphql”: {
“enabled”: false,
“execution_mode”: “”,
“version”: “”,
“schema”: “”,
“type_field_configurations”: null,
“playground”: {
“enabled”: false,
“path”: “”
},
“engine”: {
“field_configs”: null,
“data_sources”: null
}
}
}

Olu · November 15, 2021, 11:01am

Can you confirm that the key is being created in Redis?

You can check from the redis-cli using the command
get apikey-{KEY_HASH_GOES_HERE}

Can you also check for all keys with the pattern command
keys *apikey-*

You can run the command just after the API call and let us know the result.

saloni512 · November 15, 2021, 11:18am

Below is the output, looks like redis database is null:

[root@ip-10-10-31-38 apps]# redis-cli
127.0.0.1:6379> keys apikey-
(empty list or set)
127.0.0.1:6379> get apikey
(nil)
127.0.0.1:6379> get apikey*
(nil)
127.0.0.1:6379>

Olu · November 15, 2021, 1:59pm

Hi @saloni512,

None of the command above would yield an output. The asterisk needs to be inputted. We want to ascertain that the keys are in redis. Either of the command below should work

keys apikey-* This to show all keys that have the prefix apikey

keys * This to show all keys in redis*

Also, I was unable to reproduce the issue. The keys still exist after 5 minutes based on your gateway version, redis version, key, and api definition.

saloni512 · November 15, 2021, 2:12pm

Please find below output:

127.0.0.1:6379> keys apikey-*
(empty list or set)
127.0.0.1:6379>
127.0.0.1:6379> keys *

“tyk-liveness-probe”
“backup1”
“backup4”
“backup2”
“redis-test-35821d18-3b1b-4919-8182-78a6b1636be1”
“backup3”
“host-checker:PollerActiveInstanceID”
127.0.0.1:6379>

Olu · November 15, 2021, 4:11pm

From the looks of things, there are no API keys in redis (which were my initial thoughts).

Can you do the following

Check that Tyk is properly connected to the right redis instance. You can and share your config file and gateway logs.
Check the redis keys immediately when a new key is created. What is the result? Try to run the GET command to get the newly created key using the Keys gateway API. Can you also try using another API tool like postman.
Try to hack in a new test key into redis. You can use the command below, then try to list the keys from the gateway API

set apikey-test "{\"last_check\":0,\"allowance\":1000,\"rate\":1000,\"per\":1,\"throttle_interval\":0,\"throttle_retry_limit\":0,\"max_query_depth\":0,\"date_created\":\"2021-11-15T13:39:55.4023398Z\",\"expires\":-1,\"quota_max\":-1,\"quota_renews\":1636983655,\"quota_remaining\":-1,\"quota_renewal_rate\":60,\"access_rights\":{\"CGW-key-header\":{\"api_name\":\"CGW-key-header\",\"api_id\":\"CGW-key-header\",\"versions\":[\"Default\"],\"allowed_urls\":null,\"restricted_types\":null,\"limit\":null,\"field_access_rights\":null,\"allowance_scope\":\"\"}},\"org_id\":\"default\",\"oauth_client_id\":\"\",\"oauth_keys\":null,\"certificate\":\"\",\"basic_auth_data\":{\"password\":\"\",\"hash_type\":\"\"},\"jwt_data\":{\"secret\":\"\"},\"hmac_enabled\":false,\"enable_http_signature_validation\":false,\"hmac_string\":\"\",\"rsa_certificate_id\":\"\",\"is_inactive\":false,\"apply_policy_id\":\"\",\"apply_policies\":null,\"data_expires\":0,\"monitor\":{\"trigger_limits\":null},\"enable_detail_recording\":false,\"enable_detailed_recording\":false,\"meta_data\":{},\"tags\":[],\"alias\":\"\",\"last_updated\":\"1636983595\",\"id_extractor_deadline\":0,\"session_lifetime\":0}"

saloni512 · December 2, 2021, 8:03am

HI, its connected to redis instance Please fine below logs and config:

tyk.conf

<<
{

“secret”: “352d20ee67be67f6340b4c0605b044b7”,*
“template_path”: “/opt/tyk-gateway/templates”,*
“tyk_js_path”: “/opt/tyk-gateway/js/tyk.js”,*
“middleware_path”: “/opt/tyk-gateway/middleware”,*
“use_db_app_configs”: false,*
“app_path”: “/opt/tyk-gateway/apps/”,*
“enable_detailed_recording”: true,*
“log_level”: “debug”,*
“http_server_options.ssl_insecure_skip_verify”: true,*
“proxy_ssl_insecure_skip_verify”: true,*
“storage”: {*
“type”: “redis”,*
“username”: “”,*
“password”: “”,*
“database”: 0,*
“optimisation_max_idle”: 2000,*
“optimisation_max_active”: 4000*
},*
“enable_analytics”: false,*
“analytics_config”: {*
“type”: “csv”,*
“csv_dir”: “/tmp”,*
“mongo_url”: “”,*
“mongo_db_name”: “”,*
“mongo_collection”: “”,*
“purge_delay”: -1,*
“ignored_ips”: *
},*
“health_check”: {*
“enable_health_checks”: true,*
“health_check_value_timeouts”: 60*
},*
“optimisations_use_async_session_write”: true,*
“enable_non_transactional_rate_limiter”: true,*
“enable_sentinel_rate_limiter”: false,*
“enable_redis_rolling_limiter”: false,*
“allow_master_keys”: false,*
“policies”: {*
“policy_source”: “file”,*
“policy_record_name”: “/opt/tyk-gateway/policies/policies.json”*
},*
“hash_keys”: true,*
“close_connections”: false,*
“http_server_options”: {*
“enable_websockets”: true*
},*
“allow_insecure_configs”: true,*
“coprocess_options”: {*
“enable_coprocess”: true,*
“coprocess_grpc_server”: “”*
},*
“enable_bundle_downloader”: true,*
“bundle_base_url”: “”,*
“global_session_lifetime”: 100,*
“force_global_session_lifetime”: false,*
“max_idle_connections_per_host”: 500,*
“enable_jsvm”: true,*
“hash_key_function”: “murmur64”*
}

TYK Log

<<
Dec 02 01:39:24 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:39:24” level=debug msg=“Error trying to get value:redis: nil”
Dec 02 01:39:24 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:39:24” level=debug msg=“No Primary instance found, assuming control” prefix=host-check-mgr
Dec 02 01:39:34 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:39:34” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:39:44 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:39:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:39:54 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:39:54” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:40:04 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:40:04” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:40:14 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:40:14” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:40:24 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:40:24” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:40:34 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:40:34” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:40:44 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:40:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:40:54 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:40:54” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:41:04 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:41:04” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:41:14 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:41:14” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:41:24 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:41:24” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:41:34 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:41:34” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:41:44 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:41:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:41:56 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:41:55” level=debug msg=“Error trying to get value:redis: nil”
Dec 02 01:41:56 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:41:55” level=debug msg=“No Primary instance found, assuming control” prefix=host-check-mgr
Dec 02 01:42:04 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:42:04” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:42:14 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:42:14” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:42:24 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:42:24” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:42:34 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:42:34” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:42:44 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:42:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:42:54 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:42:54” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:43:04 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:43:04” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:43:15 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:43:14” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:43:24 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:43:24” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:43:34 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:43:34” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:43:44 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:43:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:43:55 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:43:55” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:44:05 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:44:05” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:44:16 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:44:15” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:44:24 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:44:24” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:44:34 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:44:34” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:44:45 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:44:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:44:54 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:44:54” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:45:04 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:45:04” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:45:14 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:45:14” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:45:24 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:45:24” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:45:34 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:45:34” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:45:44 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:45:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:45:54 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:45:54” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:46:05 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:46:04” level=debug msg=“Error trying to get value:redis: nil”
Dec 02 01:46:05 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:46:04” level=debug msg=“No Primary instance found, assuming control” prefix=host-check-mgr
Dec 02 01:46:14 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:46:14” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:46:24 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:46:24” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:46:34 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:46:34” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:46:44 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:46:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:46:54 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:46:54” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:47:04 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:47:04” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:47:14 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:47:14” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:47:24 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:47:24” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:47:35 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:47:34” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:47:45 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:47:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:47:54 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:47:54” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:48:05 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:48:05” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:48:14 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:48:14” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:48:24 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:48:24” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:48:34 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:48:34” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:48:44 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:48:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:48:54 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:48:54” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:49:05 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:49:04” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:49:14 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:49:14” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:49:24 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:49:24” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:49:34 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:49:34” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:49:44 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:49:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:49:54 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:49:54” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:50:04 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:50:04” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:50:14 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:50:14” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:50:24 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:50:24” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:50:34 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:50:34” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:50:44 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:50:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:50:54 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:50:54” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:51:04 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:51:04” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:51:14 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:51:14” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:51:24 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:51:24” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:51:34 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:51:34” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:51:44 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:51:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:51:54 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:51:54” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:52:05 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:52:04” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:52:15 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:52:14” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:52:25 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:52:24” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:52:34 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:52:34” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:52:44 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:52:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:52:54 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:52:54” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:53:05 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:53:04” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:53:14 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:53:14” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:53:25 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:53:25” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:53:36 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:53:36” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:53:44 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:53:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:53:54 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:53:54” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:54:04 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:54:04” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:54:14 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:54:14” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:54:33 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:54:24” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:54:37 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:54:34” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:54:44 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:54:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:54:55 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:54:55” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:55:04 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:55:04” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:55:14 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:55:14” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:55:24 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:55:24” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:55:34 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:55:34” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:55:44 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:55:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:55:55 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:55:55” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:56:04 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:56:04” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:56:14 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:56:14” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:56:24 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:56:24” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:56:34 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:56:34” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:56:44 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:56:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:56:54 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:56:54” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:57:04 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:57:04” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:57:15 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:57:15” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:57:24 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:57:24” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:57:34 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:57:34” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:57:44 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:57:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:57:54 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:57:54” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:58:10 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:58:04” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:58:19 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:58:14” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:58:31 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:58:24” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:58:39 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:58:34” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:58:44 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:58:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:58:54 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:58:54” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:59:04 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:59:04” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:59:14 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:59:14” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:59:24 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:59:24” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:59:34 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:59:34” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:59:44 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:59:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 01:59:54 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 01:59:54” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:00:04 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:00:04” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:00:14 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:00:14” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:00:24 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:00:24” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:00:34 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 0m=+19514.102436609”
Dec 02 02:36:28 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945m=+19454.102436609"
Dec 02 02:36:28 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:36:28” level=debug msg=“Returned: 0”
Dec 02 02:36:34 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:36:34” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:36:44 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:36:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:36:54 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:36:54” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:37:04 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:37:04” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:37:14 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:37:14” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:37:24 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:37:24” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:37:34 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:37:34” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:37:44 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:37:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:37:54 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:37:54” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:38:04 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:38:04” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:38:14 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:38:14” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:38:24 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:38:24” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:38:34 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:38:34” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:38:44 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:38:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:38:54 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:38:54” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:39:04 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:39:04” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:39:14 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:39:14” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:39:24 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:39:24” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:39:34 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:39:34” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:39:44 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:39:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:39:54 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:39:54” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:40:04 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:40:04” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:40:14 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:40:14” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:40:24 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:40:24” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:40:34 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:40:34” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:40:44 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:40:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:40:54 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:40:54” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:41:04 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:41:04” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:41:14 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:41:14” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:41:24 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:41:24” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:41:34 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:41:34” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:41:44 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:41:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:41:54 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:41:54” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:42:04 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:42:04” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:42:14 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:42:14” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:42:24 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:42:24” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:42:34 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:42:34” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr
Dec 02 02:42:44 ip-10-10-31-38.ap-southeast-1.compute.internal tyk[945]: time=“Dec 02 02:42:44” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr]: time=“Dec 02 02:36:28” level=debug msg="Then is: 2021-12-02 02:35:28.282076872 -0500 EST

saloni512 · December 2, 2021, 8:04am

Check the redis keys immediately when a new key is created. What is the result? Try to run the GET command to get the newly created key using the Keys gateway API. Can you also try using another API tool like postman.: I created keys. Checked it in the redis its available but after sometime it will be removed:

<<
keys apikey-*

“apikey-2e622327447fe411”

Try to hack in a new test key into redis. You can use the command below, then try to list the keys from the gateway API. Created test api as per the suggestion:

127.0.0.1:6379> keys apikey-*

“apikey-test”
“apikey-2e622327447fe411”.

Please suggest further, its quite annoying as key is getting expired again and again.

Olu · December 2, 2021, 10:29am

Hi @saloni512, your config file looks fine

Try creating a random key with a different name and value.

set defaultkey “this is a default key”

Then check the TTL after creation. If the result is -1 and the key disappears after a while, then there may be an explicit key-deletion command being initiated somewhere or your keys are being evicted due to memory pressure.

You may want to examine a couple of statistics to identify what the issue is. Please run the redis command below after you have observed the result of adding a random key

info all

Examine and share the # Memory data section. Pay major attention to the used_memory, maxmemory and maxmemory_policy values
Examine and share the # Stats data section. Locate the evicted_keys property for its’s value
Examine and share the # Commandstats data section. Look out for DEL, HDEL or FLUSH commands

As a last resort, you may want to debug what is going on with your redis using the MONITOR command.

I hope this helps

saloni512 · December 10, 2021, 3:19am

HI @Olu, I tried creating default key in redis database explicitly but that too is getting removed from the database.

I tried to run ./tyk ( /opt/tyk-gateaway ), observed below error:
[Dec 09 22:13:44] WARN pub-sub: Reconnecting storage: Redis is either down or ws not configured
[Dec 09 22:13:44] DEBUG host-check-mgr: Active Instance is: 82818525-4403-4704-9425-affacb86f2b5
[Dec 09 22:13:44] DEBUG host-check-mgr: — I am: 520c9ec4-a8c9-48b3-ab7d-b12d5efc9171
[Dec 09 22:13:44] DEBUG host-check-mgr: New master found, no tests running

If I grep redis process, its running:

redis 740 1 0 21:23 ? 00:00:06 /usr/bin/redis-server 0.0.0.0:6379

netstat output:

[root@ip-10-10-31-38 tyk-gateway]# netstat -alnp | grep 740
tcp 0 0 0.0.0.0:6379 0.0.0.0:* LISTEN 740/redis-server 0.
tcp 0 0 10.10.31.38:6379 18.142.155.240:45643 ESTABLISHED 740/redis-server 0.
tcp 0 0 10.10.31.38:6379 18.142.155.240:41384 ESTABLISHED 740/redis-server 0.
tcp 0 0 10.10.31.38:6379 18.142.155.240:41365 ESTABLISHED 740/redis-server 0.
tcp 0 0 10.10.31.38:6379 18.142.155.240:41338 ESTABLISHED 740/redis-server 0.
tcp 0 0 10.10.31.38:6379 18.142.155.240:45342 ESTABLISHED 740/redis-server 0.
tcp 0 0 10.10.31.38:6379 18.142.155.240:42774 ESTABLISHED 740/redis-server 0.
unix 3 [ ] STREAM CONNECTED 15610 740/redis-server 0.

Please suggest whats the issue here and how to fix, I didnt change any configuration at the redis side or TYK .

Olu · December 10, 2021, 4:01pm

You might need a Redis expert to help troubleshoot the issue with your Redis. We can only help point in right the direction where the issue could be but cannot assist further in troubleshooting.

Or you could simply create a new Redis server/instance and examine if the same issue occurs.

I tried to run ./tyk ( /opt/tyk-gateaway ), observed below error:
[Dec 09 22:13:44] WARN pub-sub: Reconnecting storage: Redis is either down or ws not configured

This is simply a warning message that shows since it takes some milliseconds before Tyk establishes a connection with Redis. You can verify if Tyk is connected to Redis with the Liveness health check

curl --location -g --request GET '{{APIM_GATEWAY}}/hello'

Olu · December 13, 2021, 10:33am

Additionally, can you create the key again via Tyk and share the debug log of the master gateway? From the logs, we can see what appears to be 2 instances of a gateway but do not see the creation of a key in the logs.

For Redis, you might want to check and verify the eviction policies set on your redis server.