Heath Checking APIs returning 404 in Tyk CE

Hi,

I’m trying to check the Health of the Gateway and the health of my API created using Tyk Gateway CE.
In the docs it’s pretty straightforward wherein the request url is “http://localhost:8080/tyk/health_check” and “http://localhost:8080/tyk/testApi01/hello” respectively.
Unfortunately when I make an API call I get a 404 - Page Not Found Error Message. Please let me know, if I’m missing anything or if I’m required to make any changes in the tyk.conf file.
Please Note that I have renamed the Gateway Health checking endpoint from /hello to health_check by duly giving the value to TYK_GW_HEALTHCHECKENDPOINTNAME as env variable in docker-compose.yml .

Thanks and Cheers!

Prashanth (@selvamp)

Hi @selvamp

I assume you were following the Tyk Gateway API documentation. I can reproduce the same 404 you are getting and cannot remember if I got it to work in the past.

However, if you follow the liveness health check documentation, you should be able to get the status of the gateway with {hostname}/hello. Let me know of that works. I will get back to you regarding the gateway API

Hi @Olu.
Thanks for the reply.
Yes, that’s right I’m referring to Tyk Gateway API for checking the health of the GW.
Sure, I’ll take a look at Liveness health check.
But it would be awesome if the former works. Thanks! :slight_smile:

With regards,
Prashanth

Got confirmation that the right endpoint is {hostname}/hello without the tyk prefix. There was a documentation change a while back that messed with the right value.

Please use the endpoint below to check the health of an API health

curl --location --request GET ‘http://localhost:8080/tyk/health?api_id=012345689’ --header ‘x-tyk-authorization: {secret}’

Just out of curiosity, any reason why you would prefer the Gateway API (prefixed with tyk) to work?

Thanks @Olu for the confirmation.
The agenda is to check the health of the Gateway and the individual APIs. I was confused by the documentation because i thought {hostname}/hello and {hostname}/tyk/hello are two different APIs that perform two different operations. As long as Health Checking can be achieved, I’m good!
Hope this answers your question!

Regards,
Prashanth (@selvamp)

Hey again @Olu.

Checking the health of the Gateway works flawlessly, i.e. {hostname}/hello endpoint. But checking the health of individual APIs seems to not work for me. cURL command:
curl --location --request GET ‘http://localhost:8080/tyk/health?api_id=httpbintest’ --header ‘x-tyk-authorization: tyk_secret’.
When I’m testing the same in Postman, I’m given with “Error: socket hang up”. Please let me know the next course of action.

Thanks,
Prashanth

Could you check or share the gateway logs in debug mode for issues?

Sure @Olu.
Sharing the Docker Logs in DEBUG mode below:

time="Sep 02 09:09:06" level=debug msg="Requesting API list"
time="Sep 02 09:09:12" level=debug msg="Primary instance set, I am master" prefix=host-check-mgr
time="Sep 02 09:09:22" level=debug msg="Primary instance set, I am master" prefix=host-check-mgr
time="Sep 02 09:09:29" level=warning msg="Attempted administrative access with invalid or missing key!" prefix=main
time="Sep 02 09:09:32" level=debug msg="Primary instance set, I am master" prefix=host-check-mgr
time="Sep 02 09:09:42" level=debug msg="Primary instance set, I am master" prefix=host-check-mgr
time="Sep 02 09:09:42" level=debug msg="Searching for: httpbintest.Throttle"
time="Sep 02 09:09:42" level=debug msg="Incrementing raw key: httpbintest.Throttle"
time="Sep 02 09:09:42" level=debug msg="keyName is: httpbintest.Throttle"
time="Sep 02 09:09:42" level=debug msg="Now is:2021-09-02 09:09:42.5450618 +0000 UTC m=+150.420120501"
time="Sep 02 09:09:42" level=debug msg="Then is: 2021-09-02 09:08:42.5450618 +0000 UTC m=+90.420120501"
time="Sep 02 09:09:42" level=debug msg="Returned: 0"
time="Sep 02 09:09:42" level=debug msg="Count is: 0"
time="Sep 02 09:09:42" level=debug msg="Searching for: httpbintest.QuotaViolation"
time="Sep 02 09:09:42" level=debug msg="Incrementing raw key: httpbintest.QuotaViolation"
time="Sep 02 09:09:42" level=debug msg="keyName is: httpbintest.QuotaViolation"
time="Sep 02 09:09:42" level=debug msg="Now is:2021-09-02 09:09:42.5457971 +0000 UTC m=+150.420855701"
time="Sep 02 09:09:42" level=debug msg="Then is: 2021-09-02 09:08:42.5457971 +0000 UTC m=+90.420855701"
time="Sep 02 09:09:42" level=debug msg="Returned: 0"
time="Sep 02 09:09:42" level=debug msg="Count is: 0"
time="Sep 02 09:09:42" level=debug msg="Searching for: httpbintest.KeyFailure"
time="Sep 02 09:09:42" level=debug msg="Incrementing raw key: httpbintest.KeyFailure"
time="Sep 02 09:09:42" level=debug msg="keyName is: httpbintest.KeyFailure"
time="Sep 02 09:09:42" level=debug msg="Now is:2021-09-02 09:09:42.5461705 +0000 UTC m=+150.421229101"
time="Sep 02 09:09:42" level=debug msg="Then is: 2021-09-02 09:08:42.5461705 +0000 UTC m=+90.421229101"
time="Sep 02 09:09:42" level=debug msg="Returned: 0"
time="Sep 02 09:09:42" level=debug msg="Count is: 0"
time="Sep 02 09:09:42" level=debug msg="Searching for: httpbintest.Request"
time="Sep 02 09:09:42" level=debug msg="Incrementing raw key: httpbintest.Request"
time="Sep 02 09:09:42" level=debug msg="keyName is: httpbintest.Request"
time="Sep 02 09:09:42" level=debug msg="Now is:2021-09-02 09:09:42.5466262 +0000 UTC m=+150.421684901"
time="Sep 02 09:09:42" level=debug msg="Then is: 2021-09-02 09:08:42.5466262 +0000 UTC m=+90.421684901"
time="Sep 02 09:09:42" level=debug msg="Returned: 0"
time="Sep 02 09:09:42" level=debug msg="Count is: 0"
time="Sep 02 09:09:42" level=debug msg="Searching KV for: httpbintest.Request"
time="Sep 02 09:09:42" level=debug msg="Incrementing raw key: httpbintest.Request"
time="Sep 02 09:09:42" level=debug msg="keyName is: httpbintest.Request"
time="Sep 02 09:09:42" level=debug msg="Now is:2021-09-02 09:09:42.5473808 +0000 UTC m=+150.422439501"
time="Sep 02 09:09:42" level=debug msg="Then is: 2021-09-02 09:08:42.5473808 +0000 UTC m=+90.422439501"
time="Sep 02 09:09:42" level=debug msg="Returned: 1"
time="Sep 02 09:09:42" level=debug msg="Found: [1630573782546626200]"
2021/09/02 09:09:42 http: panic serving 172.19.0.1:62302: interface conversion: interface {} is string, not []uint8
goroutine 31 [running]:
net/http.(*conn).serve.func1(0xc00051a1e0)
/usr/local/go/src/net/http/server.go:1801 +0x147
panic(0x1e03e40, 0xc000d2bc80)
/usr/local/go/src/runtime/panic.go:975 +0x47a
github.com/TykTechnologies/tyk/gateway.(*DefaultHealthChecker).ApiHealthValues(0xc000f44680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/__w/tyk/tyk/gateway/api_healthcheck.go:120 +0x75f
github.com/TykTechnologies/tyk/gateway.healthCheckhandler(0x258ca60, 0xc00037e9a0, 0xc000d44000)
/__w/tyk/tyk/gateway/api.go:2124 +0x14c
net/http.HandlerFunc.ServeHTTP(0x2385d28, 0x258ca60, 0xc00037e9a0, 0xc000d44000)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc00003dd40, 0x258ca60, 0xc00037e9a0, 0xc000195e00)
/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:212 +0xd3
github.com/TykTechnologies/tyk/gateway.InstrumentationMW.func1(0x258ca60, 0xc00037e9a0, 0xc000195e00)
/__w/tyk/tyk/gateway/instrumentation_handlers.go:57 +0xf3
net/http.HandlerFunc.ServeHTTP(0xc000cdc4c0, 0x258ca60, 0xc00037e9a0, 0xc000195e00)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/TykTechnologies/tyk/gateway.controlAPICheckClientCertificate.func1(0x258ca60, 0xc00037e9a0, 0xc000195e00)
/__w/tyk/tyk/gateway/server.go:400 +0x208
net/http.HandlerFunc.ServeHTTP(0xc000cdc4e0, 0x258ca60, 0xc00037e9a0, 0xc000195e00)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/TykTechnologies/tyk/gateway.checkIsAPIOwner.func1(0x258ca60, 0xc00037e9a0, 0xc000195e00)
/__w/tyk/tyk/gateway/server.go:497 +0x1cd
net/http.HandlerFunc.ServeHTTP(0xc000ccbcb0, 0x258ca60, 0xc00037e9a0, 0xc000195e00)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/TykTechnologies/tyk/gateway.stripSlashes.func1(0x258ca60, 0xc00037e9a0, 0xc000195e00)
/__w/tyk/tyk/gateway/server.go:386 +0x142
net/http.HandlerFunc.ServeHTTP(0xc000cdc500, 0x258ca60, 0xc00037e9a0, 0xc000195e00)
/usr/local/go/src/net/http/server.go:2042 +0x44
net/http.StripPrefix.func1(0x258ca60, 0xc00037e9a0, 0xc000195d00)
/usr/local/go/src/net/http/server.go:2081 +0x1a2
net/http.HandlerFunc.ServeHTTP(0xc000ccbce0, 0x258ca60, 0xc00037e9a0, 0xc000195d00)
/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc00003dc80, 0x258ca60, 0xc00037e9a0, 0xc000195b00)
/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:212 +0xd3
github.com/TykTechnologies/tyk/gateway.(*handleWrapper).ServeHTTP(0xc000246e98, 0x258ca60, 0xc00037e9a0, 0xc000195b00)
/__w/tyk/tyk/gateway/proxy_muxer.go:52 +0x169
golang.org/x/net/http2/h2c.h2cHandler.ServeHTTP(0x2565140, 0xc000246e98, 0xc0003611c0, 0x258ca60, 0xc00037e9a0, 0xc000195b00)
/go/pkg/mod/golang.org/x/[email protected]/http2/h2c/h2c.go:98 +0x30e
github.com/TykTechnologies/tyk/gateway.(*h2cWrapper).ServeHTTP(0xc000526f80, 0x258ca60, 0xc00037e9a0, 0xc000195b00)
/__w/tyk/tyk/gateway/proxy_muxer.go:40 +0x52
net/http.serverHandler.ServeHTTP(0xc0003d2e00, 0x258ca60, 0xc00037e9a0, 0xc000195b00)
/usr/local/go/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc00051a1e0, 0x2592620, 0xc000a02740)
/usr/local/go/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2969 +0x36c
time="Sep 02 09:09:52" level=debug msg="Primary instance set, I am master" prefix=host-check-mgr

Hope this helps!

Regards,
Prashanth

This is weird. Can you share your gateway config?

Are you referring to tyk.conf or tyk.standalone.conf file?

tyk.conf contents:

{
“listen_address”: “”,
“listen_port”: 8080,
“secret”: “tyk_secret”,
“node_secret”: “tyk_secret”,
“template_path”: “./templates”,
“tyk_js_path”: “./js/tyk.js”,
“middleware_path”: “./middleware”,
“policies”: {
“policy_source”: “service”,
“policy_connection_string”: “http://tyk-dashboard:3000”,
“policy_record_name”: “tyk_policies”,
“allow_explicit_policy_id”: true
},
“use_db_app_configs”: true,
“db_app_conf_options”: {
“connection_string”: “http://tyk-dashboard:3000”,
“node_is_segmented”: false,
“tags”: [
“test”
]
},
“disable_dashboard_zeroconf”: false,
“app_path”: “./test_apps/”,
“storage”: {
“type”: “redis”,
“host”: “tyk-redis”,
“port”: 6379,
“hosts”: null,
“username”: “”,
“password”: “”,
“database”: 0,
“optimisation_max_idle”: 3000,
“optimisation_max_active”: 5000,
“enable_cluster”: false
},
“enable_separate_cache_store”: false,
“enable_analytics”: true,
“analytics_config”: {
“type”: “mongo”,
“ignored_ips”: [],
“enable_detailed_recording”: true,
“enable_geo_ip”: false,
“geo_ip_db_path”: “./GeoLite2-City.mmdb”,
“normalise_urls”: {
“enabled”: true,
“normalise_uuids”: true,
“normalise_numbers”: true,
“custom_patterns”: []
}
},
“health_check”: {
“enable_health_checks”: true,
“health_check_value_timeouts”: 0
},
“optimisations_use_async_session_write”: true,
“allow_master_keys”: false,
“hash_keys”: true,
“hash_key_function”: “murmur64”,
“suppress_redis_signal_reload”: false,
“suppress_default_org_store”: false,
“use_redis_log”: true,
“sentry_code”: “”,
“use_sentry”: false,
“use_syslog”: false,
“use_graylog”: false,
“use_logstash”: false,
“graylog_network_addr”: “”,
“logstash_network_addr”: “”,
“syslog_transport”: “”,
“logstash_transport”: “”,
“syslog_network_addr”: “”,
“enforce_org_data_age”: true,
“enforce_org_data_detail_logging”: false,
“enforce_org_quotas”: true,
“experimental_process_org_off_thread”: false,
“enable_non_transactional_rate_limiter”: true,
“enable_sentinel_rate_limiter”: false,
“Monitor”: {
“enable_trigger_monitors”: false,
“configuration”: {
“method”: “”,
“target_path”: “”,
“template_path”: “”,
“header_map”: null,
“event_timeout”: 0
},
“global_trigger_limit”: 0,
“monitor_user_keys”: false,
“monitor_org_keys”: false
},
“oauth_refresh_token_expire”: 0,
“oauth_token_expire”: 0,
“oauth_redirect_uri_separator”: “;”,
“slave_options”: {
“use_rpc”: false,
“connection_string”: “”,
“rpc_key”: “”,
“api_key”: “”,
“enable_rpc_cache”: false,
“bind_to_slugs”: false,
“disable_keyspace_sync”: false,
“group_id”: “”
},
“disable_virtual_path_blobs”: false,
“local_session_cache”: {
“disable_cached_session_state”: true,
“cached_session_timeout”: 0,
“cached_session_eviction”: 0
},
“http_server_options”: {
“override_defaults”: false,
“read_timeout”: 0,
“write_timeout”: 0,
“use_ssl”: false,
“use_ssl_le”: false,
“enable_websockets”: true,
“certificates”: [],
“server_name”: “”,
“min_version”: 0,
“flush_interval”: 0
},
“service_discovery”: {
“default_cache_timeout”: 0
},
“close_connections”: false,
“auth_override”: {
“force_auth_provider”: false,
“auth_provider”: {
“name”: “”,
“storage_engine”: “”,
“meta”: null
},
“force_session_provider”: false,
“session_provider”: {
“name”: “”,
“storage_engine”: “”,
“meta”: null
}
},
“uptime_tests”: {
“disable”: true,
“config”: {
“failure_trigger_sample_size”: 1,
“time_wait”: 2,
“checker_pool_size”: 50,
“enable_uptime_analytics”: true
}
},
“hostname”: “”,
“enable_api_segregation”: false,
“control_api_hostname”: “”,
“enable_custom_domains”: true,
“enable_jsvm”: true,
“coprocess_options”: {
“enable_coprocess”: false,
“coprocess_grpc_server”: “tcp://localhost:5555”
},
“hide_generator_header”: false,
“event_handlers”: {
“events”: {}
},
“event_trigers_defunct”: {},
“pid_file_location”: “./tyk-gateway.pid”,
“allow_insecure_configs”: true,
“public_key_path”: “”,
“close_idle_connections”: false,
“allow_remote_config”: true,
“enable_bundle_downloader”: false,
“max_idle_connections_per_host”: 500,
“disable_ports_whitelist”: true,
“ports_whitelist”: {
“tcp”: {
“ports”: [
18788,
26290
]
}
},
“enable_hashed_keys_listing”: true,
“enable_batch_request_support”: true,
“log_level”: “debug”
}

Hope this helps!

The standalone config file. I guess the standalone config file is what your CE version is using.

tyk.standalone.conf contents:

{
“listen_port”: 8080,
“secret”: “tyk_secret”,
“template_path”: “/opt/tyk-gateway/templates”,
“tyk_js_path”: “/opt/tyk-gateway/js/tyk.js”,
“middleware_path”: “/opt/tyk-gateway/middleware”,
“use_db_app_configs”: false,
“app_path”: “/opt/tyk-gateway/apps/”,
“storage”: {
“type”: “redis”,
“host”: “tyk-redis”,
“port”: 6379,
“username”: “”,
“password”: “”,
“database”: 0,
“optimisation_max_idle”: 2000,
“optimisation_max_active”: 4000
},
“enable_analytics”: false,
“analytics_config”: {
“type”: “csv”,
“csv_dir”: “/tmp”,
“mongo_url”: “”,
“mongo_db_name”: “”,
“mongo_collection”: “”,
“purge_delay”: -1,
“ignored_ips”: []
},
“health_check”: {
“enable_health_checks”: true,
“health_check_value_timeouts”: 60
},
“optimisations_use_async_session_write”: false,
“enable_non_transactional_rate_limiter”: true,
“enable_sentinel_rate_limiter”: false,
“enable_redis_rolling_limiter”: false,
“allow_master_keys”: false,
“policies”: {
“policy_source”: “file”,
“policy_record_name”: “/opt/tyk-gateway/policies/policies.json”
},
“hash_keys”: true,
“close_connections”: false,
“http_server_options”: {
“enable_websockets”: true
},
“allow_insecure_configs”: true,
“coprocess_options”: {
“enable_coprocess”: true,
“coprocess_grpc_server”: “”
},
“enable_bundle_downloader”: true,
“bundle_base_url”: “”,
“global_session_lifetime”: 100,
“force_global_session_lifetime”: false,
“max_idle_connections_per_host”: 500,
“enable_jsvm”: true,
“enable_hashed_keys_listing”: true
}

Here you go, but I feel that the Gateway is accessing tyk.conf because I have set a secret in tyk.conf and i’m able to access APIs using that secret whereas there is some random secret in tyk.stanadlone.conf file which is not used. Hope this helps!

Regards,
Prashanth

@selvamp I can reproduce the issue with your standalone.conf.

I will crosscheck and confirm what the issue is

Thanks a million @Olu.

Regards,
Prashanth

@selvamp, so I have found the issue to be with the value of health_check_value_timeouts. If you set that to 0, then it works. Anything other than that produces the socket error.

I am trying to find out why it functions this way. I will get back to you in a bit

Hi @Olu. After making the requisite changes, the API is working perfectly. Thanks for the help!

As of the above, you may let me know, it would be interesting to know how the function is intended to work.

Thanks and Regards,
Prashanth