Crahsing: Concurrent Map Access

Support
1

I am using GitHub - TykTechnologies/tyk-hybrid-docker: Tyk Hybrid Mode Docker Image as the basis for a hybrid tyk deployment. I have modified the Dockerfile to install package lua-nginx-redis and modified the nginx configs to suit my employer’s needs. The nginx configuration is tested and functional. I also set “enable_geo_ip” to false in the tyk.conf. The container is otherwise unmodified.

I have built and deployed this container along with linked redis and api containers using docker 1.7.1 on an Ubuntu 14.04 host. When testing the deployment using 30 concurrent client connections, I encountered repeated crashes in the docker container. Upon examining the logs, I encountered the following message immediately before a very verbose golang stack trace: “fatal error: concurrent map read and map write” the stack trace makes it cleat that this error is occurring in the tyk code.

I know that by default, golang maps are not safe for concurrent access. I have not examined the tyk source code.

The api I hope to deploy behind tyk is expected to handle many more than 30 concurrent connections. Is there a way for me to avoid crashes like this?

2

The stack trace would really help here :slight_smile:

We have hybrid setups pushing 500k requests per day, so we’re very sure about it handling traffic.

3

I’ve tried to replicate this both with the latest Tyk build (dev branch) throwing about 1k concurrents at it (geo_ip disabled) and our hybrid container and can;t seem to get the same issue you are getting, which implies it’s a tricky, nasty little problem that we’ll need the stack trace to pin down.

So it would really help to get it.

On the other hand - we can’t support anything outside of our own official docker hybrid build and setup - we build our docker image automatically on docker hub and that is the one we test and work with.

But I’d still like to get my hands on that stack trace :slight_smile:

4

Martin, thanks so much for your attention! I am not able to recover the stack trace at this time, but I will have it posted by tomorrow morning.

5

No worries - I’m testing the system with the race detector now - there are a few but they are minor and shouldn’t affect throughput or stability.

6

@Martin Thanks for your patience. The host machine was actually unplugged and moved between my first post and your response. I’ve got it running again now. :slight_smile:

Here are the first 106 lines of the stack trace. An additional 5500+ lines of goroutine traces were also produced. I can post some or all of those if needs be.

fatal error: concurrent map read and map write

goroutine 3632 [running]:
runtime.throw(0xe158a0, 0x21)
	/usr/local/go/src/runtime/panic.go:530 +0x90 fp=0xc823d016e0 sp=0xc823d016c8
runtime.mapaccess2_faststr(0xb1f380, 0xc820127860, 0xc8246e4c00, 0x18, 0xc826b81ea0, 0x1)
	/usr/local/go/src/runtime/hashmap_fast.go:307 +0x5b fp=0xc823d01740 sp=0xc823d016e0
main.(*OrganizationMonitor).ProcessRequestOffThread(0xc824701d60, 0x7f6a6c86ecf8, 0xc826b92b60, 0xc826b81ea0, 0x0, 0x0, 0x0, 0x0, 0x10cad82d0962d12b)
	/home/tyk/go/src/github.com/lonelycode/tyk/middleware_organisation_activity.go:136 +0x26d fp=0xc823d01810 sp=0xc823d01740
main.(*OrganizationMonitor).ProcessRequest(0xc824701d60, 0x7f6a6c86ecf8, 0xc826b92b60, 0xc826b81ea0, 0x0, 0x0, 0x0, 0x0, 0x5e91810b3d3ef424)
	/home/tyk/go/src/github.com/lonelycode/tyk/middleware_organisation_activity.go:37 +0x75 fp=0xc823d01860 sp=0xc823d01810
main.CreateMiddleware.func1.1(0x7f6a6c86ecf8, 0xc826b92b60, 0xc826b81ea0)
	/home/tyk/go/src/github.com/lonelycode/tyk/middleware.go:44 +0x178 fp=0xc823d01918 sp=0xc823d01860
net/http.HandlerFunc.ServeHTTP(0xc823e79680, 0x7f6a6c86ecf8, 0xc826b92b60, 0xc826b81ea0)
	/usr/local/go/src/net/http/server.go:1618 +0x3a fp=0xc823d01938 sp=0xc823d01918
main.CreateMiddleware.func1.1(0x7f6a6c86ecf8, 0xc826b92b60, 0xc826b81ea0)
	/home/tyk/go/src/github.com/lonelycode/tyk/middleware.go:54 +0x26f fp=0xc823d019f0 sp=0xc823d01938
net/http.HandlerFunc.ServeHTTP(0xc823e796c0, 0x7f6a6c86ecf8, 0xc826b92b60, 0xc826b81ea0)
	/usr/local/go/src/net/http/server.go:1618 +0x3a fp=0xc823d01a10 sp=0xc823d019f0
github.com/gorilla/mux.(*Router).ServeHTTP(0xc82020cc80, 0x7f6a6c86ecf8, 0xc826b92b60, 0xc826b81ea0)
	/home/tyk/go/src/github.com/gorilla/mux/mux.go:98 +0x29e fp=0xc823d01b28 sp=0xc823d01a10
net/http.(*ServeMux).ServeHTTP(0xc820010cc0, 0x7f6a6c86ecf8, 0xc826b92b60, 0xc826b81ea0)
	/usr/local/go/src/net/http/server.go:1910 +0x17d fp=0xc823d01b80 sp=0xc823d01b28
net/http.serverHandler.ServeHTTP(0xc825bd2b00, 0x7f6a6c86ecf8, 0xc826b92b60, 0xc826b81ea0)
	/usr/local/go/src/net/http/server.go:2081 +0x19e fp=0xc823d01be0 sp=0xc823d01b80
net/http.(*conn).serve(0xc826b63900)
	/usr/local/go/src/net/http/server.go:1472 +0xf2e fp=0xc823d01fa8 sp=0xc823d01be0
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc823d01fb0 sp=0xc823d01fa8
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2137 +0x44e

goroutine 1 [chan receive, 10 minutes]:
github.com/rcrowley/goagain.Wait(0x7f6a6c869858, 0xc820022240, 0x7f6a6c869858, 0x0, 0x0)
	/home/tyk/go/src/github.com/rcrowley/goagain/goagain.go:207 +0x488
main.main()
	/home/tyk/go/src/github.com/lonelycode/tyk/main.go:1302 +0x2414

goroutine 17 [syscall, 10 minutes, locked to thread]:
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 5 [chan receive, 10 minutes]:
github.com/getsentry/raven-go.(*Client).worker(0xc8200ff8c0)
	/home/tyk/go/src/github.com/getsentry/raven-go/client.go:415 +0x60
created by github.com/getsentry/raven-go.newClient
	/home/tyk/go/src/github.com/getsentry/raven-go/client.go:304 +0x193

goroutine 6 [syscall, 10 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 7 [select]:
github.com/pmylund/go-cache.(*janitor).Run(0xc82013b0f0, 0xc8201277a0)
	/home/tyk/go/src/github.com/pmylund/go-cache/cache.go:946 +0x13e
created by github.com/pmylund/go-cache.runJanitor
	/home/tyk/go/src/github.com/pmylund/go-cache/cache.go:964 +0x81

goroutine 8 [sleep]:
time.Sleep(0x2540be400)
	/usr/local/go/src/runtime/time.go:59 +0xf9
main.(*HostCheckerManager).CheckActivePollerLoop(0x11f1da0)
	/home/tyk/go/src/github.com/lonelycode/tyk/host_checker_manager.go:120 +0x48b
created by main.(*HostCheckerManager).Start
	/home/tyk/go/src/github.com/lonelycode/tyk/host_checker_manager.go:81 +0x3f

goroutine 106 [select, 10 minutes]:
github.com/lonelycode/gorpc.clientWriter(0xc8201a9680, 0x7f6a6c85b088, 0xc820022668, 0xc8202ede60, 0xc8202afaa0, 0xc82025bd40, 0xc82025bda0)
	/home/tyk/go/src/github.com/lonelycode/gorpc/client.go:613 +0xa6b
created by github.com/lonelycode/gorpc.clientHandleConnection
	/home/tyk/go/src/github.com/lonelycode/gorpc/client.go:560 +0x5de

goroutine 81 [select, 10 minutes]:
github.com/lonelycode/gorpc.clientWriter(0xc8201a9680, 0x7f6a6c85b088, 0xc8202bc1f0, 0xc8202d9b30, 0xc8202d6bd0, 0xc820390360, 0xc8203903c0)
	/home/tyk/go/src/github.com/lonelycode/gorpc/client.go:613 +0xa6b
created by github.com/lonelycode/gorpc.clientHandleConnection
	/home/tyk/go/src/github.com/lonelycode/gorpc/client.go:560 +0x5de

goroutine 20 [select]:
github.com/pmylund/go-cache.(*janitor).Run(0xc8202d6490, 0xc8202d86c0)
	/home/tyk/go/src/github.com/pmylund/go-cache/cache.go:946 +0x13e
created by github.com/pmylund/go-cache.runJanitor
	/home/tyk/go/src/github.com/pmylund/go-cache/cache.go:964 +0x81

goroutine 21 [chan send, 10 minutes]:
github.com/jeffail/tunny.(*workerWrapper).Loop(0xc8202d8720)
	/home/tyk/go/src/github.com/jeffail/tunny/worker.go:51 +0xbc
created by github.com/jeffail/tunny.(*workerWrapper).Open
	/home/tyk/go/src/github.com/jeffail/tunny/worker.go:80 +0x182

goroutine 22 [chan send, 10 minutes]:
github.com/jeffail/tunny.(*workerWrapper).Loop(0xc8202d8750)
	/home/tyk/go/src/github.com/jeffail/tunny/worker.go:51 +0xbc
created by github.com/jeffail/tunny.(*workerWrapper).Open
	/home/tyk/go/src/github.com/jeffail/tunny/worker.go:80 +0x182

goroutine 23 [chan send, 10 minutes]:
github.com/jeffail/tunny.(*workerWrapper).Loop(0xc8202d8780)
	/home/tyk/go/src/github.com/jeffail/tunny/worker.go:51 +0xbc
created by github.com/jeffail/tunny.(*workerWrapper).Open
	/home/tyk/go/src/github.com/jeffail/tunny/worker.go:80 +0x182
1 Like
7

Awesome, thank you - that’s been fixed now and pushed to our hybrid container (and github repo), if you pull and rebuild you should be able to continue