Crahsing: Concurrent Map Access

I am using GitHub - TykTechnologies/tyk-hybrid-docker: Tyk Hybrid Mode Docker Image as the basis for a hybrid tyk deployment. I have modified the Dockerfile to install package lua-nginx-redis and modified the nginx configs to suit my employer’s needs. The nginx configuration is tested and functional. I also set “enable_geo_ip” to false in the tyk.conf. The container is otherwise unmodified.

I have built and deployed this container along with linked redis and api containers using docker 1.7.1 on an Ubuntu 14.04 host. When testing the deployment using 30 concurrent client connections, I encountered repeated crashes in the docker container. Upon examining the logs, I encountered the following message immediately before a very verbose golang stack trace: “fatal error: concurrent map read and map write” the stack trace makes it cleat that this error is occurring in the tyk code.

I know that by default, golang maps are not safe for concurrent access. I have not examined the tyk source code.

The api I hope to deploy behind tyk is expected to handle many more than 30 concurrent connections. Is there a way for me to avoid crashes like this?

The stack trace would really help here :slight_smile:

We have hybrid setups pushing 500k requests per day, so we’re very sure about it handling traffic.

I’ve tried to replicate this both with the latest Tyk build (dev branch) throwing about 1k concurrents at it (geo_ip disabled) and our hybrid container and can;t seem to get the same issue you are getting, which implies it’s a tricky, nasty little problem that we’ll need the stack trace to pin down.

So it would really help to get it.

On the other hand - we can’t support anything outside of our own official docker hybrid build and setup - we build our docker image automatically on docker hub and that is the one we test and work with.

But I’d still like to get my hands on that stack trace :slight_smile:

Martin, thanks so much for your attention! I am not able to recover the stack trace at this time, but I will have it posted by tomorrow morning.

No worries - I’m testing the system with the race detector now - there are a few but they are minor and shouldn’t affect throughput or stability.

@Martin Thanks for your patience. The host machine was actually unplugged and moved between my first post and your response. I’ve got it running again now. :slight_smile:

Here are the first 106 lines of the stack trace. An additional 5500+ lines of goroutine traces were also produced. I can post some or all of those if needs be.

fatal error: concurrent map read and map write

goroutine 3632 [running]:
runtime.throw(0xe158a0, 0x21)
	/usr/local/go/src/runtime/panic.go:530 +0x90 fp=0xc823d016e0 sp=0xc823d016c8
runtime.mapaccess2_faststr(0xb1f380, 0xc820127860, 0xc8246e4c00, 0x18, 0xc826b81ea0, 0x1)
	/usr/local/go/src/runtime/hashmap_fast.go:307 +0x5b fp=0xc823d01740 sp=0xc823d016e0
main.(*OrganizationMonitor).ProcessRequestOffThread(0xc824701d60, 0x7f6a6c86ecf8, 0xc826b92b60, 0xc826b81ea0, 0x0, 0x0, 0x0, 0x0, 0x10cad82d0962d12b)
	/home/tyk/go/src/ +0x26d fp=0xc823d01810 sp=0xc823d01740
main.(*OrganizationMonitor).ProcessRequest(0xc824701d60, 0x7f6a6c86ecf8, 0xc826b92b60, 0xc826b81ea0, 0x0, 0x0, 0x0, 0x0, 0x5e91810b3d3ef424)
	/home/tyk/go/src/ +0x75 fp=0xc823d01860 sp=0xc823d01810
main.CreateMiddleware.func1.1(0x7f6a6c86ecf8, 0xc826b92b60, 0xc826b81ea0)
	/home/tyk/go/src/ +0x178 fp=0xc823d01918 sp=0xc823d01860
net/http.HandlerFunc.ServeHTTP(0xc823e79680, 0x7f6a6c86ecf8, 0xc826b92b60, 0xc826b81ea0)
	/usr/local/go/src/net/http/server.go:1618 +0x3a fp=0xc823d01938 sp=0xc823d01918
main.CreateMiddleware.func1.1(0x7f6a6c86ecf8, 0xc826b92b60, 0xc826b81ea0)
	/home/tyk/go/src/ +0x26f fp=0xc823d019f0 sp=0xc823d01938
net/http.HandlerFunc.ServeHTTP(0xc823e796c0, 0x7f6a6c86ecf8, 0xc826b92b60, 0xc826b81ea0)
	/usr/local/go/src/net/http/server.go:1618 +0x3a fp=0xc823d01a10 sp=0xc823d019f0*Router).ServeHTTP(0xc82020cc80, 0x7f6a6c86ecf8, 0xc826b92b60, 0xc826b81ea0)
	/home/tyk/go/src/ +0x29e fp=0xc823d01b28 sp=0xc823d01a10
net/http.(*ServeMux).ServeHTTP(0xc820010cc0, 0x7f6a6c86ecf8, 0xc826b92b60, 0xc826b81ea0)
	/usr/local/go/src/net/http/server.go:1910 +0x17d fp=0xc823d01b80 sp=0xc823d01b28
net/http.serverHandler.ServeHTTP(0xc825bd2b00, 0x7f6a6c86ecf8, 0xc826b92b60, 0xc826b81ea0)
	/usr/local/go/src/net/http/server.go:2081 +0x19e fp=0xc823d01be0 sp=0xc823d01b80
	/usr/local/go/src/net/http/server.go:1472 +0xf2e fp=0xc823d01fa8 sp=0xc823d01be0
	/usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc823d01fb0 sp=0xc823d01fa8
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2137 +0x44e

goroutine 1 [chan receive, 10 minutes]:, 0xc820022240, 0x7f6a6c869858, 0x0, 0x0)
	/home/tyk/go/src/ +0x488
	/home/tyk/go/src/ +0x2414

goroutine 17 [syscall, 10 minutes, locked to thread]:
	/usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 5 [chan receive, 10 minutes]:*Client).worker(0xc8200ff8c0)
	/home/tyk/go/src/ +0x60
created by
	/home/tyk/go/src/ +0x193

goroutine 6 [syscall, 10 minutes]:
	/usr/local/go/src/runtime/sigqueue.go:116 +0x132
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 7 [select]:*janitor).Run(0xc82013b0f0, 0xc8201277a0)
	/home/tyk/go/src/ +0x13e
created by
	/home/tyk/go/src/ +0x81

goroutine 8 [sleep]:
	/usr/local/go/src/runtime/time.go:59 +0xf9
	/home/tyk/go/src/ +0x48b
created by main.(*HostCheckerManager).Start
	/home/tyk/go/src/ +0x3f

goroutine 106 [select, 10 minutes]:, 0x7f6a6c85b088, 0xc820022668, 0xc8202ede60, 0xc8202afaa0, 0xc82025bd40, 0xc82025bda0)
	/home/tyk/go/src/ +0xa6b
created by
	/home/tyk/go/src/ +0x5de

goroutine 81 [select, 10 minutes]:, 0x7f6a6c85b088, 0xc8202bc1f0, 0xc8202d9b30, 0xc8202d6bd0, 0xc820390360, 0xc8203903c0)
	/home/tyk/go/src/ +0xa6b
created by
	/home/tyk/go/src/ +0x5de

goroutine 20 [select]:*janitor).Run(0xc8202d6490, 0xc8202d86c0)
	/home/tyk/go/src/ +0x13e
created by
	/home/tyk/go/src/ +0x81

goroutine 21 [chan send, 10 minutes]:*workerWrapper).Loop(0xc8202d8720)
	/home/tyk/go/src/ +0xbc
created by*workerWrapper).Open
	/home/tyk/go/src/ +0x182

goroutine 22 [chan send, 10 minutes]:*workerWrapper).Loop(0xc8202d8750)
	/home/tyk/go/src/ +0xbc
created by*workerWrapper).Open
	/home/tyk/go/src/ +0x182

goroutine 23 [chan send, 10 minutes]:*workerWrapper).Loop(0xc8202d8780)
	/home/tyk/go/src/ +0xbc
created by*workerWrapper).Open
	/home/tyk/go/src/ +0x182

1 Like

Awesome, thank you - that’s been fixed now and pushed to our hybrid container (and github repo), if you pull and rebuild you should be able to continue