v1.7.1 gateway panic

Support Installation and Deployment
1

Imported Google Group message. Original thread at: Redirecting to Google Groups Import Date: 2016-01-19 21:17:52 +0000.
Sender:Christoph.
Date:Sunday, 12 July 2015 15:35:13 UTC+1.

Hi,

I am evaluation Tyk for use in our company and, based on the quickstart, I set everything up as docker containers deployed onto an Ubuntu VM hosted on Azure. Now for the second time I observed a “crash” of the gateway container. All I could find in the log is a panic message. Here are the log entries that lead up to the panic:

time=“2015-07-09T14:23:35Z” level=warning msg=“Restart signal (redis) received, restarting muxers”
time=“2015-07-09T14:23:35Z” level=info msg=“Using App Configuration from Mongo DB”
time=“2015-07-09T14:23:35Z” level=info msg=“Checking for transform paths…”
time=“2015-07-09T14:23:35Z” level=info msg=“Checking for transform paths…”
time=“2015-07-09T14:23:35Z” level=info msg=“Checking for transform paths…”
time=“2015-07-09T14:23:35Z” level=info msg=“Checking for transform paths…”
time=“2015-07-09T14:23:35Z” level=info msg=“Loading API configurations.”
time=“2015-07-09T14:23:35Z” level=info msg=“Loading API Spec for: Hello World”
time=“2015-07-09T14:23:35Z” level=info msg=“Connecting to redis on: redis:6379”
time=“2015-07-09T14:23:35Z” level=info msg=“Connecting to redis on: redis:6379”
time=“2015-07-09T14:23:35Z” level=info msg=“Health Checker initialised.”
time=“2015-07-09T14:23:35Z” level=info msg=“Connecting to redis on: redis:6379”
time=“2015-07-09T14:23:35Z” level=info msg=“Connecting to redis on: redis:6379”
time=“2015-07-09T14:23:35Z” level=info msg=“Loading Middleware”
time=“2015-07-09T14:23:35Z” level=info msg=“Connecting to redis on: redis:6379”
time=“2015-07-09T14:23:35Z” level=info msg=“Rate limits available at: /91558bbea1854354623735938efd8371/tyk/rate-limits/”
time=“2015-07-09T14:23:35Z” level=info msg=“Loading API Spec for: User Management API”
time=“2015-07-09T14:23:35Z” level=info msg=“Connecting to redis on: redis:6379”
time=“2015-07-09T14:23:35Z” level=info msg=“Connecting to redis on: redis:6379”
time=“2015-07-09T14:23:35Z” level=info msg=“Health Checker initialised.”
time=“2015-07-09T14:23:35Z” level=info msg=“Connecting to redis on: redis:6379”
time=“2015-07-09T14:23:35Z” level=info msg=“Connecting to redis on: redis:6379”
time=“2015-07-09T14:23:35Z” level=info msg=“Loading Middleware”
time=“2015-07-09T14:23:35Z” level=info msg=“Connecting to redis on: redis:6379”
panic: http: invalid pattern

goroutine 9 [running]:
net/http.(*ServeMux).Handle(0xc208230b40, 0xfb9c60, 0x0, 0x7f41c6eebe80, 0xc208555a70)
/usr/local/go/src/net/http/server.go:1551 +0xf8
main.loadApps(0xc20850fb80, 0x2, 0x2, 0xc208230b40)
/home/tyk/go/src/github.com/lonelycode/tyk/main.go:404 +0x15e4
main.ReloadURLStructure()
/home/tyk/go/src/github.com/lonelycode/tyk/main.go:493 +0x11c
main.HandleRedisReloadMsg(0xc20867d540, 0x19, 0xc2085ba370, 0x45, 0x45)
/home/tyk/go/src/github.com/lonelycode/tyk/reload_signal.go:39 +0x2a7
main.(*RedisStorageManager).StartPubSubHandler(0xc20822a160, 0xbfae10, 0x19, 0xda94b0, 0x0, 0x0)
/home/tyk/go/src/github.com/lonelycode/tyk/storage_handlers.go:540 +0x3e6
main.StartPubSubLoop()
/home/tyk/go/src/github.com/lonelycode/tyk/reload_signal.go:18 +0x7c
created by main.main
/home/tyk/go/src/github.com/lonelycode/tyk/main.go:646 +0x452

… followed by many more go routine stack traces.

Unfortunately, I can’t tell which action triggered the crash; all I know is a colleague played with the dashboard at that time. Since the dashboard is still up and running, he did not notice that something broke in the background.

Now whenever I try to restart the gateway, it immediately exits again and the log shows the same panic. I therefore suspect that some data was stored either in redis or mongo that does not agree with the HTTP muxer.

Is this a known issue? How can I debug the issue?

Best regards
Christoph

2

Imported Google Group message.
Sender:Martin Buhr.
Date:Sunday, 12 July 2015 15:50:43 UTC+1.

Also you can fix this by going to the dashboard and setting a listen path in the API detail view (if you can’t see the listen path editing screen, can edit the config option in tyk_analytics.conf “hide_listen_path”: false.

When you see the listen paths for your APIs they will either be the same or that they are empty (or you have a duplicate).

Simply save the APIs with unique listen paths and then restart the Tyk gateway.

That fixes the problem too :slight_smile:

On Sun, Jul 12, 2015 at 7:35 AM -0700, “Christoph” [email protected] wrote:

  • show quoted text -

–
You received this message because you are subscribed to the Google Groups “Tyk Community Support” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web, visit https://groups.google.com/d/msgid/tyk-community-support/5ad9c62c-a540-4180-a339-faa717ae9217%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

Imported Google Group message.
Sender:.
Date:Sunday, 12 July 2015 15:46:24 UTC+1.

Hi,

What version of the dashboard are you using?

I would suggest upgrading to the latest dashboard version (0.9.5.3).

There’s a bug in 0.9.5.2 that causes the listen path to be generated blank - you can fix this by going to the tyk_analytics.conf and setting “force_api_defaults”: true in the version you are running.

The panic is caused because Tyk is loading the API and trying to set a listen path for it, and if there are two it had a problem because the listen path is empty / broken.

Let me know if either of those options sort out the problem.

Thanks,
Martin

  • show quoted text -

  • show quoted text -

–
You received this message because you are subscribed to the Google Groups “Tyk Community Support” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web, visit https://groups.google.com/d/msgid/tyk-community-support/5ad9c62c-a540-4180-a339-faa717ae9217%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Imported Google Group message.
Sender:Christoph.
Date:Sunday, 12 July 2015 15:50:43 UTC+1.

Also you can fix this by going to the dashboard and setting a listen path in the API detail view (if you can’t see the listen path editing screen, can edit the config option in tyk_analytics.conf “hide_listen_path”: false.

When you see the listen paths for your APIs they will either be the same or that they are empty (or you have a duplicate).

Simply save the APIs with unique listen paths and then restart the Tyk gateway.

That fixes the problem too :slight_smile:

On Sun, Jul 12, 2015 at 7:35 AM -0700, “Christoph” [email protected] wrote:

  • show quoted text -

–
You received this message because you are subscribed to the Google Groups “Tyk Community Support” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web, visit https://groups.google.com/d/msgid/tyk-community-support/5ad9c62c-a540-4180-a339-faa717ae9217%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

5

Imported Google Group message.
Sender:Martin Buhr.
Date:Sunday, 12 July 2015 17:37:10 UTC+1.

Hi Martin,

thanks, with the hint from your second mail I was able to identify the
misconfigured API, fix it, and successfully re-start the gateway
again.

I did not find the dashboard version - does the dashboard web UI
actually show it somewhere? All I know is that the image the dashboard
docker container was based on had been created 4 days ago. (The docker
quickstart script merely references the “latest” tag.) Now I re-pulled
the dashboard docker image and I got a new image; this time it was
created 3 days ago, according to “sudo docker images”. So it is
possible I was still running dashboard version 0.9.5.2.

Best regards
Christoph

  • show quoted text -
6

Imported Google Group message.
Sender:Martin Buhr.
Date:Sunday, 12 July 2015 17:49:09 UTC+1.

Hi,

The docker image is fully up to date, so you now have he latest version if you re pulled it.

The version can be seen when the application starts (just look at the log output from dashboard docker image, it’s printed out when the application started).

We’ll be adding the version number to the UI soon (there’s already a hit issue raised for it)

Good to hear the problem was solved,

Cheers,
Martin

On Sun, Jul 12, 2015 at 9:37 AM -0700, “Christoph Ludwig” [email protected] wrote:

Hi Martin,

thanks, with the hint from your second mail I was able to identify the
misconfigured API, fix it, and successfully re-start the gateway
again.

I did not find the dashboard version - does the dashboard web UI
actually show it somewhere? All I know is that the image the dashboard
docker container was based on had been created 4 days ago. (The docker
quickstart script merely references the “latest” tag.) Now I re-pulled
the dashboard docker image and I got a new image; this time it was
created 3 days ago, according to “sudo docker images”. So it is
possible I was still running dashboard version 0.9.5.2.

Best regards
Christoph

2015-07-12 16:50 GMT+02:00 Martin Buhr :

Also you can fix this by going to the dashboard and setting a listen path in
the API detail view (if you can’t see the listen path editing screen, can
edit the config option in tyk_analytics.conf “hide_listen_path”: false.

When you see the listen paths for your APIs they will either be the same or
that they are empty (or you have a duplicate).

Simply save the APIs with unique listen paths and then restart the Tyk
gateway.

That fixes the problem too :slight_smile:

On Sun, Jul 12, 2015 at 7:35 AM -0700, “Christoph”
wrote:

Hi,

I am evaluation Tyk for use in our company and, based on the quickstart, I
set everything up as docker containers deployed onto an Ubuntu VM hosted on
Azure. Now for the second time I observed a “crash” of the gateway
container. All I could find in the log is a panic message. Here are the log
entries that lead up to the panic:

time=“2015-07-09T14:23:35Z” level=warning msg=“Restart signal (redis)
received, restarting muxers”
time=“2015-07-09T14:23:35Z” level=info msg=“Using App Configuration from
Mongo DB”
time=“2015-07-09T14:23:35Z” level=info msg=“Checking for transform
paths…”
time=“2015-07-09T14:23:35Z” level=info msg=“Checking for transform
paths…”
time=“2015-07-09T14:23:35Z” level=info msg=“Checking for transform
paths…”
time=“2015-07-09T14:23:35Z” level=info msg=“Checking for transform
paths…”
time=“2015-07-09T14:23:35Z” level=info msg=“Loading API configurations.”
time=“2015-07-09T14:23:35Z” level=info msg=“Loading API Spec for: Hello
World”
time=“2015-07-09T14:23:35Z” level=info msg=“Connecting to redis on:
redis:6379”
time=“2015-07-09T14:23:35Z” level=info msg=“Connecting to redis on:
redis:6379”
time=“2015-07-09T14:23:35Z” level=info msg=“Health Checker initialised.”
time=“2015-07-09T14:23:35Z” level=info msg=“Connecting to redis on:
redis:6379”
time=“2015-07-09T14:23:35Z” level=info msg=“Connecting to redis on:
redis:6379”
time=“2015-07-09T14:23:35Z” level=info msg=“Loading Middleware”
time=“2015-07-09T14:23:35Z” level=info msg=“Connecting to redis on:
redis:6379”
time=“2015-07-09T14:23:35Z” level=info msg=“Rate limits available at:
/91558bbea1854354623735938efd8371/tyk/rate-limits/”
time=“2015-07-09T14:23:35Z” level=info msg=“Loading API Spec for: User
Management API”
time=“2015-07-09T14:23:35Z” level=info msg=“Connecting to redis on:
redis:6379”
time=“2015-07-09T14:23:35Z” level=info msg=“Connecting to redis on:
redis:6379”
time=“2015-07-09T14:23:35Z” level=info msg=“Health Checker initialised.”
time=“2015-07-09T14:23:35Z” level=info msg=“Connecting to redis on:
redis:6379”
time=“2015-07-09T14:23:35Z” level=info msg=“Connecting to redis on:
redis:6379”
time=“2015-07-09T14:23:35Z” level=info msg=“Loading Middleware”
time=“2015-07-09T14:23:35Z” level=info msg=“Connecting to redis on:
redis:6379”
panic: http: invalid pattern

goroutine 9 [running]:
net/http.(*ServeMux).Handle(0xc208230b40, 0xfb9c60, 0x0, 0x7f41c6eebe80,
0xc208555a70)
/usr/local/go/src/net/http/server.go:1551 +0xf8
main.loadApps(0xc20850fb80, 0x2, 0x2, 0xc208230b40)
/home/tyk/go/src/github.com/lonelycode/tyk/main.go:404 +0x15e4
main.ReloadURLStructure()
/home/tyk/go/src/github.com/lonelycode/tyk/main.go:493 +0x11c
main.HandleRedisReloadMsg(0xc20867d540, 0x19, 0xc2085ba370, 0x45, 0x45)
/home/tyk/go/src/github.com/lonelycode/tyk/reload_signal.go:39

  • show quoted text -