Can the /etc/nginx/conf.d/*.tconf files be regenerated from the mongo db?
I’ve restarted our tyk servers but the .tconf files got lost in the process! If memory serves these are the files that control where to forward apis requests.
Thanks,
Ian.
Yes they can, the Tyk Host Manager is still bundled with Tyk Dashboard, you just need to configure it to point at your MongoDB and run it.
Hi Martin,
I think I must be doing something wrong. I have the mongo, redis, tyk_dashboard, tyk_gateway and host-manager up and running but the host-manager docker file still has no *.tconf files.
I think it is picking up the database though. Logs from the tyk_nginx container are:
“docker logs tyk_nginx
time=“2016-02-09T19:34:38Z” level=info msg=“Loading configuration”
time=“2016-02-09T19:34:38Z” level=info msg=“Connecting to Redis”
time=“2016-02-09T19:34:38Z” level=info msg=“Connecting to redis on: tyk_redis:6379”
time=“2016-02-09T19:34:38Z” level=info msg=“Creating new Redis connection pool”
time=“2016-02-09T19:34:38Z” level=info msg=“Connecting to Mongo”
time=“2016-02-09T19:34:38Z” level=info msg=“Loading templates”
time=“2016-02-09T19:34:38Z” level=info msg=“getTemplates took 2.310568ms”
time=“2016-02-09T19:34:38Z” level=info msg=“Syncronising active API configurations”
time=“2016-02-09T19:34:38Z” level=info msg=“Generated portal config for Org: 56544efd68b264000100006f”
time=“2016-02-09T19:34:38Z” level=info msg=“Generated portal config for Org: 563b778168b2640001000039”
time=“2016-02-09T19:34:38Z” level=info msg=“Generated portal config for Org: 564f1f9c68b264000100006c”
time=“2016-02-09T19:34:38Z” level=info msg=“Generated portal config for Org: 5654d92868b2640001000072”
time=“2016-02-09T19:34:38Z” level=info msg=“Generated portal config for Org: 564de8d968b2640001000069”
time=“2016-02-09T19:34:38Z” level=info msg=“Generated portal config for Org: 562756de68b2640001000018”
time=“2016-02-09T19:34:38Z” level=info msg=“Generated portal config for Org: 56275a0b68b264000100001a”
time=“2016-02-09T19:34:38Z” level=info msg=“Generated portal config for Org: 56265e1568b2640001000013”
time=“2016-02-09T19:34:38Z” level=info msg=“GetApiHashes took 1.924505ms”
time=“2016-02-09T19:34:38Z” level=info msg=“Found New API’s: 85”
time=“2016-02-09T19:34:38Z” level=info msg=“Updated APi’s: 0”
time=“2016-02-09T19:34:38Z” level=info msg=“Removed APi’s: 8”
time=“2016-02-09T19:34:38Z” level=info msg=“Reloading NGinX”
time=“2016-02-09T19:34:38Z” level=info msg=“Reloading NginX settings”
time=“2016-02-09T19:34:38Z” level=info msg=“Reload successful””
Is there something else I might be missing?
Thanks,
Ian.
Hi Ian,
It’s working - and generating files somewhere 
Can you share the host manager config file - it will tell you where the files are being written to.
Hi Martin,
Where would I find that configuration file? I’ve been looking at a few different ones this morning.
While searching I have found the *.tconf files - they are in /etc/nginx/conf.d/.
However, I’m still getting 404’s from tyk when I try to access an API. Which makes me think I just haven’t configured nginx to correctly load those APIs.
There is no output from docker logs on tyk_nginx, tyk_dashboard or tyk_gateway.
Thanks,
Ian.
There’s a host_manager.json file alongside the host manger binary.
Ok, so you’re using Docker here, what version of our stack are you running, because things have changed since.
Hi Martin,
The config file is below but I don’t see where the location of the *.tconf files are specified:
docker exec tyk_nginx cat /opt/tyk-dashboard/host-manager/host_manager.json
{
“mongo_url”: “mongodb://tyk_mongo/tyk_analytics”,
“redis_port”: 6379,
“redis_host”: “tyk_redis”,
“redis_password”: “”,
“tyk_ports”: [8080],
“tyk_secret”: “secret”,
“license_owner”: “Your Name”,
“manage_nginx”: true,
“manage_tyk_nodes”: false
}
All the docker images are the latest - not sure how to check which version specifically that is.
Thanks,
Ian.
Right, so if you are using latest then the host manager is no longer used, domain handling has been pushed into the stack and is now part of Tyk.
I’d suggest going through the quickstart on our site to familiarise yourself with the new version
Hi Martin,
Thanks for that. I’ve had a look through the linked page. If I’m understanding things the hot-reloading and API locations have been off-loaded from the tyk-host-manager to the dashboard/gateway?
In which case I should be able to access the APIs by directing requests to 8080 - as I will have started the docker containers with -p 8080:8080 for the gateway?
Now I think I’ll still need an nginx or apache in front as I want to secure the site with SSL. But I should be able to just configure nginx to listen on port 443 with my certs and redirect all traffic to port 8080 on the gateway container?
Thanks,
Ian.
Also will I have this latest version if I pulled from docker-hub?
Thanks,
Ian.
Ok a bit more digging. I’ve tracked the transaction from the host-manager to the tyk-gateway. The gateway is then returning the 404 which is subsequently forwarded to the client.
In case this sheds any light on the problem.
Thanks,
Ian.
Hi Ian,
Hot reloading was actually always handled by a redis pub/sub request, the manual hot-reload via host manager was a fallback, legacy method that stopped bing used in around v1.6 (it’s 1.9 now).
If you are using the dashboard, hot reloads are taken care of for you by redis notifications. If you aren’t, then calling the hot-reload (group) endpoint on one node will trigger a cluster reload.
Regarding SSL: If you look through the docs a bit further, you’ll see that SSL support has also been built into the stack, so you don;t need NGinX for that either, jsut the certs and some configuration.
Now, if you are using the latest docker build, the host manager isn’t even set up, and the bootstrap / quickstart / compose scripts are all different. I can only help you with the latest versions here I’m afraid, not how to shoe-horn host manager into the current docker configs.
If you are stuck with using a specific version, then you should change your compose setup to use the version-tagged containers instead of latest. We keep all the old version images as tags, so if yo know what you were running you can easily re-create your stack.
Hi again Martin,
Apologies - I realise this has gotten very off the original topic.
Ok I’ve started up the latest version using docker-compose. And I can see that the gateway is starting up successfully and has loaded all of the APIs from our existing database but I still get a 404 not found.
From the docker logs there are a couple of lines that I’m hoping might shed some light on things.
-
time=“2016-02-10T16:15:17Z” level=info msg="Hostname set: " - does this mean I haven’t set some host name?
-
time=“2016-02-10T16:15:17Z” level=info msg="–> Loading API: golgitest"
time=“2016-02-10T16:15:17Z” level=info msg="----> Tracking: (no host)"
time=“2016-02-10T16:15:17Z” level=info msg="----> Checking security policy: Token" - again no host? Is this important?
Any ideas why the gateway would be loading the APIs but still sending 404? I have connected directly to the API behind tyk and it is working as expected.
Thanks,
Ian.
If you’ve set domains for your API’s you will need to configure them in your DB/Dashboard.
Just set the custom domain field.
The “domain” field in the api description as returned from the dashboard using a GET to api/apis/{id}?
I’ve set this field on one of our test APIs via the dashboard and see it as a the set host when the gateway reloads the APIs but I’m still getting a 404.
Can I configure the gateway to output more information into the logs? To try and get some idea why it is not recognising the API?
Thanks,
Ian.
Yes - the domain field binds your API’s listen_path to a domain (the api slug is not used here, since there’s no template generation needed).
Did you also adjust the listen_path?
You can run it with --debug flag, but it’s very verbose and very messy, I doubt it will shed light on your problem, your APIs are just configured for NginX, not for Tyk 1.9.
Hi Martin,
Thanks very much for that. I’ve modified the listen_path on one of the APIs and that has partially resolved the issue. However, I’m having trouble accessing all the available APIs on this API.
Specifically, I can access https://hostname/first/ when listen_path is set to “/first/” but I cannot access https://hostname/first/second/ ? is there a way to wildcard this and also to pass the “second” and subsequent elements of the path to the API?
Thanks,
Ian.
I think you’re misconfiguring, here’s an example:
- Tyk Gateway hostname: http://gateway.domain.com
- Target: http://your-internal-host.com/api1
- API Name: API 1
- Listen-path: /api1/
Here’s a sample request now:
GET http://gateway.domain.com/{listen_path}/{resource}/{id}
Becomes:
GET http://your-internal-host.com/api1/{resource}/{id}
See how the listen path has been stripped? Everything after the listen path will be sent upstream to the API.
The listen path lets Tyk route your requests to the right underlying API, it’s like targeting an ID, they can’t be shared and must be unique per custom domain.
I think I have it configured like that but still no luck. Here’s a sample of the config I’m working with. Does anything look out of place? The string api-name is a fixed string that identifies the API to our system.
Thanks,
Ian.
{
“api_model”:{
},
"api_definition":{
"id":"56587a5668b264xxxxxxxx",
"name":"api-name",
"slug":"api-name",
"api_id":"62edada6536446725xxxxxxxxx",
"org_id":"564f1f9c68b26400xxxxxxxx",
"use_keyless":false,
"use_oauth2":false,
"oauth_meta":{
"allowed_access_types":[],
"allowed_authorize_types":[],
"auth_login_redirect":""
},
"auth":{
"use_param":false,
"use_cookie":false,
"auth_header_name":"Authorization"
},
"use_basic_auth":false,
"enable_jwt":false,
"jwt_signing_method":"",
"notifications":{
"shared_secret":"",
"oauth_on_keychange_url":""
},
"enable_signature_checking":false,
"hmac_allowed_clock_skew":0,
"definition":{
"location":"",
"key":""
},
"version_data":{
"not_versioned":true,
"versions":{
"Default":{
"name":"Default",
"expires":"",
"paths":{
"ignored":[],
"white_list":[],
"black_list":[]
},
"use_extended_paths":true,
"extended_paths":{
"ignored":[],
"white_list":[],
"black_list":[],
"cache":[],
"transform":[],
"transform_response":[],
"transform_headers":[],
"transform_response_headers":[],
"hard_timeouts":[],
"circuit_breakers":[],
"url_rewrites":[],
"virtual":[],
"size_limits":[]
},
"global_headers":{
},
"global_headers_remove":[],
"global_size_limit":0
}
}
},
"uptime_tests":{
"check_list":[],
"config":{
"expire_utime_after":0,
"service_discovery":{
"use_discovery_service":false,
"query_endpoint":"",
"use_nested_query":false,
"parent_data_path":"",
"data_path":"",
"port_data_path":"",
"use_target_list":false,
"cache_timeout":0,
"endpoint_returns_list":false
},
"recheck_wait":0
}
},
"proxy":{
"listen_path":"/api-name/",
"target_url":"http://api-name.api-name.domain.com",
"strip_listen_path":true,
"enable_load_balancing":false,
"target_list":[],
"check_host_against_uptime_tests":false,
"service_discovery":{
"use_discovery_service":false,
"query_endpoint":"",
"use_nested_query":false,
"parent_data_path":"",
"data_path":"",
"port_data_path":"",
"use_target_list":false,
"cache_timeout":0,
"endpoint_returns_list":false
}
},
"custom_middleware":{
"pre":[],
"post":[],
"response":[]
},
"cache_options":{
"cache_timeout":0,
"enable_cache":false,
"cache_all_safe_requests":false,
"enable_upstream_cache_control":false
},
"session_lifetime":0,
"active":true,
"auth_provider":{
"name":"",
"storage_engine":"",
"meta":null
},
"session_provider":{
"name":"",
"storage_engine":"",
"meta":null
},
"event_handlers":{
"events":{
}
},
"enable_batch_request_support":false,
"enable_ip_whitelisting":false,
"allowed_ips":[],
"dont_set_quota_on_create":false,
"expire_analytics_after":0,
"response_processors":[],
"CORS":{
"enable":true,
"allowed_origins":[],
"allowed_methods":["POST",
"GET",
"PUT"],
"allowed_headers":["Authorization"],
"exposed_headers":[],
"allow_credentials":false,
"max_age":0,
"options_passthrough":false,
"debug":false
},
"domain":"http://api-name.domain.com",
"do_not_track":false,
"tags":[]
},
"hook_references":[],
"is_site":false,
"sort_by":0
}
Is the duplication here on purpose or a typo?