Hybrid gateway only updates APIs when I restart

Support Installation and Deployment
1

Still on my path of getting a hybrid test setup with tyk.io. So far, so good - I have things basically working now, however the APIs I create on the cloud admin dashboard only show up in the hybrid gateway when I restart the gateway. Meaning, new APIs I create on the dashboard don’t show up on the hybrid gateway until I restart it.

Any ideas here? Also could someone confirm how this is supposed to work, does the hybrid docker container poll the cloud service at some interval? Does the cloud service initiate a connection to the hybrid gateway when a new endpoint is created, …etc?

I’ve also seen mention of port 9090 and specifically that “you will need to ensure port 9090 is open for inbound and outbound connections to the server”. However nothing is listening (TCP-wise) on port 9090 (on the docker container). Is this a typo and it’s in fact only needed outbound from the hybrid gateway?

2

Port 9090 is outbound.

We pushed a fix last week that should have resolved the reload issue - however if you are lol experiencing it then maybe it hasn’t been fully resolved. Have you updated to the latest container version?

The workaround is to use the gateway rest api Hot reload endpoint:

https://tyk.io/docs/tyk-rest-api/hot-reload/

This will reload your gateways without ben need for a restart or dropping connections.

3

Oh - ok, thank you.

I’m afraid I’m not certain if I’m using the latest. How can I confirm that? I’m using this image:

REPOSITORY                TAG                 IMAGE ID            CREATED             SIZE
tykio/tyk-hybrid-docker   latest              e6895f09e0bc        46 hours ago        1.52 GB
tykio/tyk-hybrid-docker   <none>              5ebc8d31527f        4 weeks ago         1.51 GB

I also see this from within the container:

root@11b2de993cf0:/opt/tyk# ./tyk --version
v2.3.6
4

I can yank the docker file and build it myself maybe?

5

You’re on the latest version - we’re investigating now.

6

Ok - thanks. Just out of curiosity, can you confirm what’s supposed to happen with this under normal circumstances? E.G. does the cloud call the Hybrid gateway API, or does the gateway poll the cloud?

7

The gateway polls the cloud

8

Ok thanks. If I can be of any assistance from a test perspective, just let me know.

9

Oops - sorry. I irresponsibly neglected to mention that I am running this on an AWS ECS container instance. Therefore I am not using the start.sh to start it up. I have modeled all (ecs) env vars after what’s in the start.sh, however this is a difference. I just didn’t want to leave that out as my understanding of what is and is not relevant with tyk.io is weak at this point.

10

That should be fine - could you see if the reload is working again for you now?

11

“could you see if the reload is working again for you now?” To be clear, what I was expecting to happen - is that I add an API to the admin cloud dashboard and then my hybrid-gateway picks it up. That’s what wasn’t working. Shall I test that again?

12

That’s what should happen, it should be “update” or “add” in cloud dashboard and the gateways update.

We’ve been having trouble with this mechanism for a while now, which is why we recommend using the hot reload endpoint. We’re still not sure why exactly it randomly fails.

13

If you are still finding that the reloads aren’t working consistently, it’s just because the signal isn’t getting through to your gateways.

For now, to unblock you - I’d suggest using this:

curl -X GET -H "X-Tyk-Authorization: {secret}" https://{gateway}:{port}/tyk/reload/group

It will always work.

While we try to pin down this new flakiness.

14

Ok - yes, that hot reload worked. Thank you. Anything I can do to help solve the other issue?

I’m trying to make a case for how easy this is to manage just through the cloud dashboard. I want to leave out the part about this hot-reload if possible.

In any case, I’m sure you have thorough test beds, but if you need another example, I’d be happy to get you into my instance (just say the word).

Thanks again for your prompt and thoughtful responses!

-Matt

15

Good to hear that fixed it - one way to make it easier is to have thebjot reload curl command to be run every few minutes as a crown job - only one gateway needs to receive the notification, the whole cluster will reload.

We’all continue digging on our end, we think we know why it’s happening and will be issuing a fix soon.

16

only one gateway needs to receive the notification, the whole cluster will reload

I’m still not quite there with this workaround. I have two systems in a cluster and when I send this command (with the /tyk/reload/group endpoint) it only reloads on the one I send it to. The other node does not reload. Any ideas on how to debug this?

17

They both need to be connected to the same redis DB for the group reload to work (in fact if you are running more than one they need this anyway).

We include redis for single node setups, but it’s not viable for scaling systems.

18

Same problem here : when adding API into cloud dashboard, the gateway are not taking it unless I restart the docker container.

The call to the reload api is not an viable alternative for me.

19

Hi guillaume,

Currently this is the only work around until the hybrid back-end is patched.

One option is to cron the reload command every X seconds on the host, this works around the push issue.

Appologies for the inconvenience.

M.

20

@Martin can you clarify this a bit - why do we need a shared redis if we’re running two hybrid docker instances? (other than this endpoint updating bug)?