Service Discovery Eureka - All hosts are down

Support
1

Hi:

I’m pretty sure I have SD with eureka configured properly, but I’m getting a 503 “all hosts are down” message whenever I try to access my service.

The gateway error logs are showing:

image

My Query endpoint returns the following JSON, which shows the status as “UP”:

{
“application”: {
“name”: “my-application”,
“instance”: [
{
“instanceId”: “myhost.com:my-application:9595”,
“hostName”: “myhost.com”,
“app”: “my-application”,
“ipAddr”: “1.2.3.4”,
“status”: “UP”,
“overriddenstatus”: “UNKNOWN”,
“port”: {
“$”: 9595,
“@enabled”: “true”
},
“securePort”: {
“$”: 443,
“@enabled”: “false”
},
“countryId”: 1,
“dataCenterInfo”: {
“@class”: “com.netflix.appinfo.InstanceInfo$DefaultDataCenterInfo”,
“name”: “MyOwn”
},
“leaseInfo”: {
“renewalIntervalInSecs”: 30,
“durationInSecs”: 90,
“registrationTimestamp”: 1515736282704,
“lastRenewalTimestamp”: 1515974134182,
“evictionTimestamp”: 0,
“serviceUpTimestamp”: 1515736282704
},
“metadata”: {
“management.port”: “9595”
},
“homePageUrl”: “http://myhost.com:9595/”,
“statusPageUrl”: “http://myhost.com:9595/info”,
“healthCheckUrl”: “http://myhost.com:9595/health”,
“vipAddress”: “my-application”,
“secureVipAddress”: “my-application”,
“isCoordinatingDiscoveryServer”: “false”,
“lastUpdatedTimestamp”: “1515736282704”,
“lastDirtyTimestamp”: “1513563166754”,
“actionType”: “ADDED”
},
{
“instanceId”: “myhost.com:my-application:9595”,
“hostName”: “myhost.com”,
“app”: “my-application”,
“ipAddr”: “1.2.3.5”,
“status”: “UP”,
“overriddenstatus”: “UNKNOWN”,
“port”: {
“$”: 9595,
“@enabled”: “true”
},
“securePort”: {
“$”: 443,
“@enabled”: “false”
},
“countryId”: 1,
“dataCenterInfo”: {
“@class”: “com.netflix.appinfo.InstanceInfo$DefaultDataCenterInfo”,
“name”: “MyOwn”
},
“leaseInfo”: {
“renewalIntervalInSecs”: 30,
“durationInSecs”: 90,
“registrationTimestamp”: 1515736282704,
“lastRenewalTimestamp”: 1515974151246,
“evictionTimestamp”: 0,
“serviceUpTimestamp”: 1515736282704
},
“metadata”: {
“management.port”: “9595”
},
“homePageUrl”: “http://myhost.com:9595/”,
“statusPageUrl”: “http://myhost.com:9595/info”,
“healthCheckUrl”: “http://myhost.com:9595/health”,
“vipAddress”: “my-application”,
“secureVipAddress”: “my-application”,
“isCoordinatingDiscoveryServer”: “false”,
“lastUpdatedTimestamp”: “1515736282704”,
“lastDirtyTimestamp”: “1513235040406”,
“actionType”: “ADDED”
}
]
}
}

Has anyone ever come across this?

I’m on gateway 2.4.2 and Dashboard 1.4.2

2

Hi Crafton,

Was this working for you in a previous version of Tyk?

How is your service discovery configured for your API in Tyk?

Regards
David

3

Hi David:

This is my first attempt, so haven’t tested it in previous versions. Here’s how it’s configured:

scrnshot

4

Based on the Eureka example config we have in our docs, can you try changing “Are the values nested?” to false.

5

I tried that and the result is the same. The nested value is set to true by default if you select “Eureka” from the SD dropdown list in the dashboard. Additionally, setting to false removes the “Parent data path:” field from the UI.

6

Hi Crafton,

The configuration supplied by the dropdown is a guideline, I believe that it actually may have been added wrongly - looking at the object, and looking at the config provided in our docs, the one in the docs is correct. I think what will need to happen is that you add these fields via the Dashboard API (Get the API Definition, then edit it and PUT it back).

I will raise a bug report internally to make sure it gets fixed.

Many thanks,
Martin

7

Hi Martin:

Thanks for that. Setting via the API works. It however seems to incorrectly report the service status (returning “all hosts are down”) on random requests. I’ll probably have to chase this up another time and post if i can find some information as to why this happens.

Cheers,

Crafton

8

This could be the service discovery cache, you can change this in the tyk.conf under:

"service_discovery": {
    "default_cache_timeout": 200
}

Can you share your `tyk.conf`?
9

Here’s my conf file:

{
“listen_port”: 8080,
“node_secret”: “11111111111111111111111111111111”,
“secret”: “1111111111111111111111111111122222”,
“template_path”: “/tyk-gateway/templates”,
“tyk_js_path”: “/tyk-gateway/js/tyk.js”,
“use_db_app_configs”: true,
“db_app_conf_options”: {
“connection_string”: “http://dashboard:3000”,
“node_is_segmented”: false,
“tags”: []
},
“disable_dashboard_zeroconf”: false,
“app_path”: “/tyk-gateway/tyk-gateway/apps”,
“middleware_path”: “/tyk-gateway/middleware”,
“storage”: {
“type”: “redis”,
“host”: “myredis”,
“port”: 6379,
“username”: “”,
“password”: “1231231231312321313”,
“database”: 0,
“optimisation_max_idle”: 2000,
“optimisation_max_active”: 4000
},
“enable_analytics”: true,
“analytics_config”: {
“type”: “mongo”,
“pool_size”: 100,
“csv_dir”: “/tmp”,
“mongo_url”: “mongodb://blah:blahblah@myhost,mydb,mydb1/tyk_analytics”,
“mongo_db_name”: “”,
“mongo_collection”: “tyk_analytics”,
“purge_delay”: 100,
“ignored_ips”: [],
“enable_detailed_recording”: true,
“enable_geo_ip”: false,
“geo_ip_db_path”: “”,
“normalise_urls”: {
“enabled”: true,
“normalise_uuids”: true,
“normalise_numbers”: true,
“custom_patterns”: []
}
},
“health_check”: {
“enable_health_checks”: true,
“health_check_value_timeouts”: 60
},
“optimisations_use_async_session_write”: true,
“allow_master_keys”: false,
“policies”: {
“policy_source”: “service”,
“policy_connection_string”: “http://mydash:3000”,
“policy_record_name”: “tyk_policies”,
“allow_explicit_policy_id”: true
},
“hash_keys”: true,
“suppress_redis_signal_reload”: false,
“use_redis_log”: true,
“close_connections”: true,
“enable_non_transactional_rate_limiter”: true,
“enable_sentinel_rate_limiter”: false,
“experimental_process_org_off_thread”: true,
“local_session_cache”: {
“disable_cached_session_state”: false
},
“http_server_options”: {
“enable_websockets”: true
},
“uptime_tests”: {
“disable”: false,
“config”: {
“enable_uptime_analytics”: true,
“failure_trigger_sample_size”: 3,
“time_wait”: 300,
“checker_pool_size”: 50
}
},
“hostname”: “”,
“enable_custom_domains”: true,
“enable_jsvm”: true,
“oauth_redirect_uri_separator”: “;”,
“coprocess_options”: {
“enable_coprocess”: false,
“coprocess_grpc_server”: “”
},
“allow_insecure_configs”: true,
“public_key_path”: “”,
“close_idle_connections”: false,
“allow_remote_config”: false,
“enable_bundle_downloader”: true,
“bundle_base_url”: “”,
“global_session_lifetime”: 100,
“force_global_session_lifetime”: false,
“max_idle_connections_per_host”: 100
}