Imported Google Group message. Original thread at: Redirecting to Google Groups Import Date: 2016-01-19 21:18:20 +0000.
Sender:[email protected].
Date:Thursday, 9 July 2015 08:47:46 UTC+1.
Hi !
Every day Tyk is blocked.
I still have the same error in the log tyk.log :
level = info msg = “http : Proxy error: dial tcp: hubmed lookup : no such host”
hubmed : my server name
I think Tyk reached the maximum number of open files (ulimit 1024) .
How can I verify this ?
Or is it something else?
Tyk Version : 1.6
Thank you for the help.
Imported Google Group message.
Sender:Martin Buhr.
Date:Thursday, 9 July 2015 09:21:35 UTC+1.
Hi,
On the Tyk host, can you ping hubmed or medroc from the command line? Tyk is trying to reach the hubmed host and can’t see it.
It could be a file limit (though then you would be seeing an error saying “too many open file descriptors” - but just in case, you can change this using
ulimit -n 80000
Run this as the user that is running Tyk and restart Tyk.
Yo can also do this:
Add fs.file-max=80000 to /etc/sysctl.conf
Then logout / log in again (or restart the server), it should set up the file limit globally - but this depends on your distro.
However, I think it has more to do with Tyk being able to resolve the host name. How is it set up?
Thanks,
Martin
Imported Google Group message.
Sender:[email protected].
Date:Thursday, 9 July 2015 09:51:53 UTC+1.
Thank you for your reply.
The hubmed server is accessible when it happens.
I can ping the server.
So I will increase the ulimit value.
For information, here is the number of currently open files. I see that it increases and never decreases. Is this normal ?
[ newtprod hubmed @ ~] $ lsof -n | awk ’ {print $ 1} ’ | sort | uniq -c | sort -rn | head
1583 java
1037 tyk
42 bash
Imported Google Group message.
Sender:Martin Buhr.
Date:Thursday, 9 July 2015 10:11:29 UTC+1.
Hi,
It shouldn’t indefinetely increase, the file descriptors are usually the redis connection pool. What is the redis connection pool set to (optimisation_max_idle) and how many API’s are you proxying on the host?
Tyk will keep a pool of open connections to redis, though it should never “exhaust” the system (it will open new ones if it needs them - i.e. the pool is being fully utilised), so this might account for why it always goes up. You could try increasing the number in the pool, I think the default is 100, but you may want to bump it to 500 or more. We run some HA versions of Tyk andd don’t run into a file descriptor wall in a long-running process, ever.
You could try using v1.7.1 instead of 1.6, it has quite a few optimisations in plce that improve performance.
Will keep an eye on this, there may be an issue with connection recreation on a hot reload, but it shouldn’t really…
Good luck and let me know how it goes.
Cheers,
Martin
Imported Google Group message.
Sender:[email protected].
Date:Thursday, 9 July 2015 13:29:17 UTC+1.
There are 25 API.
optimisation_max_idle = 100. I increased to 500.
I noticed that the number of connections is incremented when I have this message:
level=warning msg=“Invalid key detected, not found in storage engine”
Imported Google Group message.
Sender:Martin Buhr.
Date:Thursday, 9 July 2015 13:35:53 UTC+1.
Hmm, will need to investigate this, you really shouldn’t be running out of file handles unless you are handling hundreds of requests per second.
Also, you would get a file descriptor error. Not a dial error.
Imported Google Group message.
Sender:Martin Buhr.
Date:Thursday, 9 July 2015 13:37:38 UTC+1.
Wait. 25 APIs could cause a problem, you will need to up the system limit. As we have connection pools on a per API basis, so you will quickly run out of descriptors if traffic goes up.
Will still look into it though.
Thanks,
Martin
Imported Google Group message.
Sender:Martin Buhr.
Date:Thursday, 9 July 2015 15:52:38 UTC+1.
Took a quick look to see what Tyk is up to… if you just do the lsof -a -p {pid} of the tyk process, can you send me the ouput (you don;t need to post it here if it is sensitive, just sned it to my personal email).
Been trying to recreate ever-increasing file descriptors but can’t seem to. Mind you I am using 1.7.1, the file descriptors do increase (but that is expected), but they also get released back to the OS eventually, there is a “stable limit” they return to.
Imported Google Group message.
Sender:Matthieu Nantern.
Date:Monday, 20 July 2015 14:59:50 UTC+1.
Version 1.7.2 seems to fix that, right ?
Le jeudi 9 juillet 2015 16:52:38 UTC+2, Martin Buhr a écrit :
Took a quick look to see what Tyk is up to… if you just do the lsof -a -p {pid} of the tyk process, can you send me the ouput (you don;t need to post it here if it is sensitive, just sned it to my personal email).
Been trying to recreate ever-increasing file descriptors but can’t seem to. Mind you I am using 1.7.1, the file descriptors do increase (but that is expected), but they also get released back to the OS eventually, there is a “stable limit” they return to.
Imported Google Group message.
Sender:Martin Buhr.
Date:Monday, 20 July 2015 15:08:00 UTC+1.
Yes it does, you need to manually ativate it though in the tyk.conf file by adding “close_connections”: true since it will no longer respect the keep-alive header.
Thanks,
Martin