Tyk Service Discovery fails under load

1

I’m using Tyk with Service Discovery.
When sending about 200 requests in parallel, I get the following error

2016/07/14 08:07:51 http: panic serving 10.88.226.106:34234: interface conversion: interface is nil, not string
goroutine 8321467 [running]:
net/http.(*conn).serve.func1(0xc82078ce70, 0x7fc3d0738130, 0xc820020530)
/usr/local/go/src/net/http/server.go:1287 +0xb5
main.(*ServiceDiscovery).GetObject(0xc820bf4510, 0xc820864520, 0x0, 0x0)
/home/tyk/go/src/github.com/lonelycode/tyk/service_discovery.go:114 +0xa3
main.(*ServiceDiscovery).GetHostname(0xc820bf4510, 0xc820864520, 0x0, 0x0)
/home/tyk/go/src/github.com/lonelycode/tyk/service_discovery.go:126 +0x73
main.(*ServiceDiscovery).GetSubObject(0xc820bf4510, 0xc820864520, 0x0, 0x0)
/home/tyk/go/src/github.com/lonelycode/tyk/service_discovery.go:192 +0x43
main.(*ServiceDiscovery).ProcessRawData(0xc820bf4510, 0xc820c4a7b0, 0x24, 0x0, 0x0, 0x0, 0x0)
/home/tyk/go/src/github.com/lonelycode/tyk/service_discovery.go:260 +0xc9f
main.(*ServiceDiscovery).GetTarget(0xc820bf4510, 0xc8203187e0, 0x55, 0x0, 0x0, 0x0, 0x0)
/home/tyk/go/src/github.com/lonelycode/tyk/service_discovery.go:271 +0xc4
main.GetURLFromService(0xc820a8ca00, 0x0, 0x0, 0x0, 0x0)
/home/tyk/go/src/github.com/lonelycode/tyk/tyk_reverse_proxy_clone.go:29 +0x1a6
main.TykNewSingleHostReverseProxy.func1(0xc8209fd0a0)
/home/tyk/go/src/github.com/lonelycode/tyk/tyk_reverse_proxy_clone.go:102 +0x71
main.(*ReverseProxy).WrappedServeHTTP(0xc8208f9380, 0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0, 0x57874800, 0x0)
/home/tyk/go/src/github.com/lonelycode/tyk/tyk_reverse_proxy_clone.go:359 +0x4e0
main.(*ReverseProxy).ServeHTTP(0xc8208f9380, 0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0, 0xc820b417d0)
/home/tyk/go/src/github.com/lonelycode/tyk/tyk_reverse_proxy_clone.go:293 +0x4a
main.SuccessHandler.ServeHTTP(0xc8204c6ab0, 0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0, 0x0)
/home/tyk/go/src/github.com/lonelycode/tyk/handler_success.go:264 +0x47f
main.DummyProxyHandler.ServeHTTP(0xc8204c6ab0, 0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/home/tyk/go/src/github.com/lonelycode/tyk/handler_proxy.go:13 +0x42
main.CreateMiddleware.func1.1(0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/home/tyk/go/src/github.com/lonelycode/tyk/middleware.go:54 +0x26f
net/http.HandlerFunc.ServeHTTP(0xc8208f95c0, 0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/usr/local/go/src/net/http/server.go:1422 +0x3a
main.CreateMiddleware.func1.1(0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/home/tyk/go/src/github.com/lonelycode/tyk/middleware.go:54 +0x26f
net/http.HandlerFunc.ServeHTTP(0xc8208f9600, 0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/usr/local/go/src/net/http/server.go:1422 +0x3a
main.CreateMiddleware.func1.1(0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/home/tyk/go/src/github.com/lonelycode/tyk/middleware.go:54 +0x26f
net/http.HandlerFunc.ServeHTTP(0xc8208f9680, 0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/usr/local/go/src/net/http/server.go:1422 +0x3a
main.CreateMiddleware.func1.1(0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/home/tyk/go/src/github.com/lonelycode/tyk/middleware.go:54 +0x26f
net/http.HandlerFunc.ServeHTTP(0xc8208f96c0, 0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/usr/local/go/src/net/http/server.go:1422 +0x3a
main.CreateMiddleware.func1.1(0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/home/tyk/go/src/github.com/lonelycode/tyk/middleware.go:54 +0x26f
net/http.HandlerFunc.ServeHTTP(0xc8208f9780, 0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/usr/local/go/src/net/http/server.go:1422 +0x3a
main.CreateMiddleware.func1.1(0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/home/tyk/go/src/github.com/lonelycode/tyk/middleware.go:54 +0x26f
net/http.HandlerFunc.ServeHTTP(0xc8208f9800, 0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/usr/local/go/src/net/http/server.go:1422 +0x3a
main.CreateMiddleware.func1.1(0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/home/tyk/go/src/github.com/lonelycode/tyk/middleware.go:54 +0x26f
net/http.HandlerFunc.ServeHTTP(0xc8208f9880, 0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/usr/local/go/src/net/http/server.go:1422 +0x3a
main.CreateMiddleware.func1.1(0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/home/tyk/go/src/github.com/lonelycode/tyk/middleware.go:54 +0x26f
net/http.HandlerFunc.ServeHTTP(0xc8208f98c0, 0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/usr/local/go/src/net/http/server.go:1422 +0x3a
main.CreateMiddleware.func1.1(0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/home/tyk/go/src/github.com/lonelycode/tyk/middleware.go:54 +0x26f
net/http.HandlerFunc.ServeHTTP(0xc8208f9940, 0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/usr/local/go/src/net/http/server.go:1422 +0x3a
main.CreateMiddleware.func1.1(0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/home/tyk/go/src/github.com/lonelycode/tyk/middleware.go:54 +0x26f
net/http.HandlerFunc.ServeHTTP(0xc8208f9980, 0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/usr/local/go/src/net/http/server.go:1422 +0x3a
main.CreateMiddleware.func1.1(0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/home/tyk/go/src/github.com/lonelycode/tyk/middleware.go:54 +0x26f
net/http.HandlerFunc.ServeHTTP(0xc8208f99c0, 0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/usr/local/go/src/net/http/server.go:1422 +0x3a
main.CreateMiddleware.func1.1(0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/home/tyk/go/src/github.com/lonelycode/tyk/middleware.go:54 +0x26f
net/http.HandlerFunc.ServeHTTP(0xc8208f9a40, 0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/usr/local/go/src/net/http/server.go:1422 +0x3a
main.CreateMiddleware.func1.1(0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/home/tyk/go/src/github.com/lonelycode/tyk/middleware.go:54 +0x26f
net/http.HandlerFunc.ServeHTTP(0xc8208f9a80, 0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/usr/local/go/src/net/http/server.go:1422 +0x3a
main.CreateMiddleware.func1.1(0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/home/tyk/go/src/github.com/lonelycode/tyk/middleware.go:54 +0x26f
net/http.HandlerFunc.ServeHTTP(0xc8208f9ac0, 0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/usr/local/go/src/net/http/server.go:1422 +0x3a
github.com/rs/cors.(*Cors).Handler.func1(0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/home/tyk/go/src/github.com/rs/cors/cors.go:185 +0x1c5
net/http.HandlerFunc.ServeHTTP(0xc82039cc20, 0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/usr/local/go/src/net/http/server.go:1422 +0x3a
github.com/gorilla/mux.(*Router).ServeHTTP(0xc820368870, 0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/home/tyk/go/src/github.com/gorilla/mux/mux.go:98 +0x29e
net/http.(*ServeMux).ServeHTTP(0xc8206151d0, 0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/usr/local/go/src/net/http/server.go:1699 +0x17d
net/http.serverHandler.ServeHTTP(0xc8204ec180, 0x7fc3d2d93c90, 0xc820a4bc30, 0xc8209fcfc0)
/usr/local/go/src/net/http/server.go:1862 +0x19e
net/http.(*conn).serve(0xc82078ce70)
/usr/local/go/src/net/http/server.go:1361 +0xbee
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:1910 +0x3f6

This happens only with service discovery, not when I set the target URL manually.
Because Tyk seems to cache the upstream service URL after discovery, the lookup service shouldn’t be a performance issue?

2

What version are you running?

3

I’m using 1.9 in docker at the moment.
Of cause I will update now to see if it solves the error, but I thought maybe someone already got a similar error.

4

I see, I would recommend upgrading, there have been changes to the SD code, particularly around handling type conversion panics like the one you describe. You might still get an error, but it will not panic and crash.

1 Like