Docker tykio/tyk-dashboard:v0.9.4 and tykio/tyk-gateway:v1.9 Consul and Service Discovery

Imported Google Group message. Original thread at: Redirecting to Google Groups Import Date: 2016-01-19 21:36:07 +0000.
Sender:Carlos Carvalho.
Date:Monday, 14 December 2015 14:29:07 UTC.

Hello,

I’ve been trying to incorporate tyk with Consul for service discovery to work. But I’m having a hard time starting to do this.

First Problem I have encountered - By your documentation the example reefers to etcd integration.Can you please send me a example of an consul implementation.

Second Problem I have encountered - The tyk dashboard doesn’t present Service Discovery options to work with. The ultimate goal will be to put that configuration when importing the API through tyk.conf definition file but some visual help of this is a life saver.

Could you help me?

Thanks

Imported Google Group message.
Sender:Martin Buhr.
Date:Monday, 14 December 2015 14:53:01 UTC.

Hi Carlos,

Yes we focussed on etcd, the service discovery setup for a Tyk API is best documented by our tests, here’s an example:

Imagine this is the configuration Consul will provide for our Upstream service:

[

{
“Node”: “foobar”,
“Address”: “10.1.10.12”,
“ServiceID”: “redis”,
“ServiceName”: “redis”,
“ServiceTags”: null,
“ServiceAddress”: “”,
“ServicePort”: 8000
},
{
“Node”: “foobar2”,
“Address”: “10.1.10.13”,
“ServiceID”: “redis”,
“ServiceName”: “redis”,
“ServiceTags”: null,
“ServiceAddress”: “”,
“ServicePort”: 8000
}
]

Then in order to create a load-balanced upstream service list for an API Definition, you can set it up with the following settings in the dashboard:
“Enable round-robin load balancing” - Checked
“Enable Service Discovery” - Checked
“Query Endpoint” - The http:// path to the consul service resource
“Does this endpoint return a list?” - Checked
“Are the values nested?” -Unchecked
“Data path:” - Address
“Is port information separate from the hostname?” - Checked
“Port data path:” - ServicePort
“Does the endpoint provide a target list?” - Checked
This should set everything up so that Tyk can extract the data from a consul endpoint for any defined API (the values in monospace should be used as is).

For your integration scenario: You use the Advanced API to push a new API Definition into Tyk, that API Definition has the above settings enabled (they all correspond to definition settings), that API definition will now use the consul endpoint to extract the data it needs to connect to the service.

We do not “Directly” integrate with Consul or Etcd, we integrate with an http endpoint that returns JSON data, you need to configure the data paths to extract the data from the JSON document that gets returned. It’s a little tricky, but it’s generic to work with almost any kind of configuration provider without a direct integration.

Let me know if that clarifies things a little.

Cheers,
Martin

Imported Google Group message.
Sender:Carlos Carvalho.
Date:Monday, 14 December 2015 16:07:04 UTC.

Hello Martin,

I have been trying to use this api defnition part of proxy element

“proxy” : {
“listen_path” : “/oservice/”,
“target_url” : “http://oservice:5000/”,
“strip_listen_path” : true,
“enable_load_balancing”: true,
“service_discovery”: {
“use_discovery_service”: true,
“query_endpoint”: “http://consul:8500/v1/catalog/service/o-service”,
“use_nested_query”: false,
“data_path”: “array.Address”,
“port_data_path”: “array.ServicePort”,
“use_target_list”: false,
“cache_timeout”: 10
}
},

But i fell like i’m misunderstanding it. Another thing that i mentioned earlier is the fact that the version i’m using on my docker images the options of service discovery don’t appear to me. Even after importing the api definiton.

What I’m doing wrong?

  • show quoted text -

Imported Google Group message.
Sender:Martin Buhr.
Date:Monday, 14 December 2015 16:27:37 UTC.

Hi Carlos,

Well, there’s a lot that can go wrong :slight_smile:

I think the best thing to share would be the output of your compose service, that way we can see the data structure we are dealing with.

I think you will also need to enable “use_target_list” as this will say that the inbound data is a list and should be copied to your LB config. Here’s the full data structure:

proxy: {
listen_path: “/closed/”,
target_url: “http://httpbin.org/”,
strip_listen_path: true,
enable_load_balancing: true,
target_list: [],
check_host_against_uptime_tests: false,
service_discovery: {
use_discovery_service: true,
query_endpoint: “http://oservice:5000/”,
use_nested_query: false,
parent_data_path: “”,

  data_path: "array.Address",
  port_data_path: "array.ServicePort",

  use_target_list: true,
  cache_timeout: 60,
  endpoint_returns_list: true
}

},

At least, that’s me filling in the gaps :slight_smile: But I’d need to actually see your data structure in order to provide guidance.

Cheers,
Martin

Imported Google Group message.
Sender:Carlos Carvalho.
Date:Monday, 14 December 2015 16:41:43 UTC.

Hi Martin,

Sorry I don’t understand what you mean by compose service.

If it helps i’m using registrator with consul to register several services that are docker containers.

  • show quoted text -

Imported Google Group message.
Sender:Martin Buhr.
Date:Monday, 14 December 2015 16:45:46 UTC.

Sorry, I meant Consul, I have compose on the brain today.

Can you just anonymise some output from that Consul endpoint so we can take a look at the JSON?

Cheers,
Martin

Imported Google Group message.
Sender:Carlos Carvalho.
Date:Monday, 14 December 2015 17:08:48 UTC.

Martin I made a request to the particular service (http://consul:8500/v1/catalog/service/oservice) and the output was this

[{“Node”:“consul”,“Address”:“172.17.42.1”,“ServiceID”:“registrator:order_service:5000”,“ServiceName”:“oservice”,“ServiceTags”:null,“ServiceAddress”:“172.17.42.5”,“ServicePort”:5000}]

But i have another question:

I will have several services that will need service discovery how can i define multiple if you wrote the query endpoint as the service and not of the consul key store. or another place that i can consult my list of service like this for example:

proxy: {
“listen_path” : “/oservice/”,
“target_url” : “http://oservice:5000/”,
strip_listen_path: true,

enable_load_balancing: true,
target_list: [],
check_host_against_uptime_tests: false,
service_discovery: {
  use_discovery_service: true,

  query_endpoint: "http://consul:8500/v1/catalog/services/",

  use_nested_query: false,
  parent_data_path: "",

  data_path: "array.Address",
  port_data_path: "array.ServicePort",

  use_target_list: true,
  cache_timeout: 60,
  endpoint_returns_list: true
}

},

Thanks

  • show quoted text -

Imported Google Group message.
Sender:Martin Buhr.
Date:Monday, 14 December 2015 17:57:53 UTC.

Hi,

So first off, I think you’ll need to remove the “array.” Part of your setting as there is no field in your object called ‘array’.

As for your other question, Tyk can uses service discovery tools to tell it where to look for upstream servers, not for new API definitions, you need to add those to Tyk for each API you want to manage.

The reason is that in many cases a service consists of containers or servers that change all the type, instead of re-loading them into Tyk each time, Tyk pulls it’s host list from a service discovery endpoint. You still need to configure the service in Tyk itself.

I can see a use case there, where APIs are discovered and exposed from Consul, it’s an interesting idea. But since Tyk is a gateway and is meant to allow a user to selectively expose services to third parties - including potentially amalgamating many microservice into a single facade, I’m not sure how useful it would be.

Let me know if that makes things clearer.

Cheers,
Martin


From: Carlos Carvalho [email protected]
Sent: Monday, December 14, 2015 17:08
Subject: Re: Docker tykio/tyk-dashboard:v0.9.4 and tykio/tyk-gateway:v1.9 Consul and Service Discovery
To: Tyk Community Support [email protected]

Martin I made a request to the particular service ( http://consul:8500/v1/catalog/service/oservice) and the output was this

  • show quoted text -


You received this message because you are subscribed to the Google Groups “Tyk Community Support” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web, visit https://groups.google.com/d/msgid/tyk-community-support/4239bb50-f572-42a2-9c24-d5c287f1fccd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Imported Google Group message.
Sender:Carlos Carvalho.
Date:Tuesday, 15 December 2015 09:24:06 UTC.

Hi Martin sorry to be once again a bother,

I will try to follow your guide lines.

But I have still the problem that for some reason the options for service discovery are not visible on the dahsboard UI interface. Even when I created a new API Definition manually.

Do you know if the versions of gateway and dashboard that I’m using have a bug of some sort regarding this issue?

Thanks

  • show quoted text -

Imported Google Group message.
Sender:Martin Buhr.
Date:Tuesday, 15 December 2015 09:41:15 UTC.

Hi Carlos,

What version are you running? Both Tyk Dashboard and Tyk Gateway output the version number (the console output will tell you when they start)

The SD options are on the API Detail screen, in the first tab, in the group called “Service Discovery”, you need to enable it for the options to appear.

Many thanks,
Martin

Imported Google Group message.
Sender:Carlos Carvalho.
Date:Tuesday, 15 December 2015 09:51:11 UTC.

I’m using Docker tykio/tyk-dashboard:v0.9.4 and tykio/tyk-gateway:v1.9 images versions.

And as you can see in the API Definition Detail doesn’t appear to be present SD Group you mentioned.

  • show quoted text -

Imported Google Group message.
Sender:Martin Buhr.
Date:Tuesday, 15 December 2015 11:58:16 UTC.

Hi Carlos,

You definitely need to update to the mates dashboard version.

We’re at dashboard v0.9.7 now.

Cheers,
Martin

  • show quoted text -

  • show quoted text -


You received this message because you are subscribed to the Google Groups “Tyk Community Support” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web, visit https://groups.google.com/d/msgid/tyk-community-support/2abf8913-c9b4-4b84-9e98-a528554df08b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.