License pool depleted error

nsidir · September 17, 2019, 9:26am

I have been using on-premises developer licence for more than 2 years. Recently (about a month or so) I received an error message in my dashboard ‘Nodes and Licenses’ page that my license pool is depleted.

I have upgraded tyk (2.8.4) and tyk-dashboard (1.8.5) to latest versions. I have also received my license updates but whenever I try to update my licence, I receive an ‘CSRF Failure’ in a blank page. Going back I see that my license has been updated (since expiration date has changed) but the ‘License Pool Depleted’ remains. My installation contains 11 active API’s that continue to work.
Thanks in advance,
Nikos S.

leon · September 17, 2019, 10:58am

First of all I recommend you to restart both dashboard and gateway.

nsidir · September 18, 2019, 5:56am

Hi Leon,

I have restarted both services. I enclose logs from both services after rebooting.
tyk-gateway logs
Sep 18 08:34:52 tyk.ilsp.gr systemd[1]: Stopped Tyk API Gateway.
Sep 18 08:37:00 tyk.ilsp.gr systemd[1]: Started Tyk API Gateway.
Sep 18 08:37:00 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:00” level=info msg=“Setting up analytics normaliser”
Sep 18 08:37:00 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:00” level=info msg=“PIDFile location set to: ./tyk-gateway.pid”
Sep 18 08:37:00 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:00” level=warning msg=“The control_api_port should be changed for production”
Sep 18 08:37:00 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:00” level=info msg=“Initialising Tyk REST API Endpoints”
Sep 18 08:37:00 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:00” level=warning msg=“File descriptor limit 1024 too low for production use. Min 80000 recommended.\n\tThis could have a significant negative i
Sep 18 08:37:00 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:00” level=warning msg=“Num CPUs 1 too low for production use. Min 2 recommended.\n\tThis could have a significant negative impact on performance
Sep 18 08:37:00 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:00” level=warning msg=“Default secret 352d20ee67be67f6340b4c0605b044b7 should be changed for production.”
Sep 18 08:37:00 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:00” level=warning msg=“Default node_secret 352d20ee67be67f6340b4c0605b044b7 should be changed for production.”
Sep 18 08:37:00 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:00” level=info msg=“Starting Poller”
Sep 18 08:37:00 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:00” level=info msg=“Redis connection pools are ready after number of retires” currRetry=0
Sep 18 08:37:00 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:00” level=info msg=“Redis connection pools are ready”
Sep 18 08:37:00 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:00” level=info msg=”–> Standard listener (http)” port=“:80”
Sep 18 08:37:00 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:00” level=info msg=“Setting up Server”
Sep 18 08:37:00 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:00” level=info msg=“Registering node.”
Sep 18 08:37:00 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:00” level=info msg=“Node registered” id=
Sep 18 08:37:00 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:00” level=info msg=“Gateway started (v2.8.4)”
Sep 18 08:37:00 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:00” level=info msg=“Initialising distributed rate limiter”
Sep 18 08:37:02 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:02” level=warning msg=“Node not registered yet, skipping DRL Notification”
Sep 18 08:37:02 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:02” level=info msg=“Stopped Heartbeat”
Sep 18 08:37:04 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:04” level=warning msg=“Node not registered yet, skipping DRL Notification”
Sep 18 08:37:05 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:05” level=error msg=“Response failed with code 404; retrying in 5s”
Sep 18 08:37:06 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:06” level=warning msg=“Node not registered yet, skipping DRL Notification”
Sep 18 08:37:08 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:08” level=warning msg=“Node not registered yet, skipping DRL Notification”
Sep 18 08:37:10 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:10” level=warning msg=“Node not registered yet, skipping DRL Notification”
Sep 18 08:37:10 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:10” level=info msg=“Node registered” id=
Sep 18 08:37:10 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:10” level=info msg=“Recovering configurations, reloading…”
Sep 18 08:37:10 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:10” level=info msg=“Policies found (0 total):”
Sep 18 08:37:10 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:10” level=info msg=“Reload queued”
Sep 18 08:37:10 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:10” level=info msg=“reload: initiating”
Sep 18 08:37:10 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:10” level=info msg=“Registering node (again).”
Sep 18 08:37:10 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:10” level=error msg=“Could not deregister: deregister request failed with status 404”
Sep 18 08:37:12 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:12” level=warning msg=“Node not registered yet, skipping DRL Notification”
Sep 18 08:37:12 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:12” level=info msg=“Stopped Heartbeat”
Sep 18 08:37:14 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:14” level=warning msg=“Node not registered yet, skipping DRL Notification”
Sep 18 08:37:15 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:15” level=error msg=“Response failed with code 404; retrying in 5s”
Sep 18 08:37:16 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:16” level=warning msg=“Node not registered yet, skipping DRL Notification”
Sep 18 08:37:18 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:18” level=warning msg=“Node not registered yet, skipping DRL Notification”
Sep 18 08:37:20 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:20” level=warning msg=“Node not registered yet, skipping DRL Notification”
Sep 18 08:37:20 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:20” level=info msg=“Node registered” id=
Sep 18 08:37:20 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:20” level=info msg=“Recovering configurations, reloading…”
Sep 18 08:37:20 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:20” level=error msg="failed to load API specs: login failure, Response was: {"Status":"Error","Message":"Authorization failed (No node ID
Sep 18 08:37:20 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:20” level=error msg="Error during syncing apis:login failure, Response was: {"Status":"Error","Message":"Authorization failed (No node ID
Sep 18 08:37:20 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:20” level=info msg=“Reload already queued”
Sep 18 08:37:20 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:20” level=info msg=“Loading policies”
Sep 18 08:37:20 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:20” level=info msg=“Using Policies from Dashboard Service”
Sep 18 08:37:20 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:20” level=info msg=“Mutex lock acquired… calling”
Sep 18 08:37:20 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:20” level=info msg=“Calling dashboard service for policy list”
Sep 18 08:37:20 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:20” level=error msg="Policy request login failure, Response was: {"Status":"Error","Message":"Authorization failed (No node ID Found)",
Sep 18 08:37:20 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:20” level=info msg=“Registering node (again).”
Sep 18 08:37:20 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:20” level=error msg=“Could not deregister: deregister request failed with status 404”
Sep 18 08:37:22 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:22” level=warning msg=“Node not registered yet, skipping DRL Notification”
Sep 18 08:37:22 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:22” level=info msg=“Stopped Heartbeat”
Sep 18 08:37:24 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:24” level=warning msg=“Node not registered yet, skipping DRL Notification”
Sep 18 08:37:25 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:25” level=error msg=“Response failed with code 404; retrying in 5s”
Sep 18 08:37:26 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:26” level=warning msg=“Node not registered yet, skipping DRL Notification”
Sep 18 08:37:28 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:28” level=warning msg=“Node not registered yet, skipping DRL Notification”
Sep 18 08:37:30 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:30” level=warning msg=“Node not registered yet, skipping DRL Notification”
Sep 18 08:37:30 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:30” level=info msg=“Node registered” id=
Sep 18 08:37:30 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:30” level=info msg=“Recovering configurations, reloading…”
Sep 18 08:37:30 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:30” level=info msg=“Policies found (0 total):”
Sep 18 08:37:30 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:30” level=info msg=“Reload already queued”
Sep 18 08:37:30 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:30” level=info msg=“Registering node (again).”
Sep 18 08:37:30 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:30” level=error msg=“Could not deregister: deregister request failed with status 404”
Sep 18 08:37:32 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:32” level=warning msg=“Node not registered yet, skipping DRL Notification”
Sep 18 08:37:32 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:32” level=info msg=“Stopped Heartbeat”
Sep 18 08:37:34 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:34” level=warning msg=“Node not registered yet, skipping DRL Notification”
Sep 18 08:37:35 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:35” level=error msg=“Response failed with code 404; retrying in 5s”
Sep 18 08:37:36 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:36” level=warning msg=“Node not registered yet, skipping DRL Notification”
Sep 18 08:37:38 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:38” level=warning msg=“Node not registered yet, skipping DRL Notification”
Sep 18 08:37:40 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:40” level=warning msg=“Node not registered yet, skipping DRL Notification”
Sep 18 08:37:40 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:40” level=info msg=“Node registered” id=
Sep 18 08:37:40 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:40” level=info msg=“Recovering configurations, reloading…”
Sep 18 08:37:40 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:40” level=error msg="failed to load API specs: login failure, Response was: {"Status":"Error","Message":"Authorization failed (No node ID
Sep 18 08:37:40 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:40” level=error msg="Error during syncing apis:login failure, Response was: {"Status":"Error","Message":"Authorization failed (No node ID
Sep 18 08:37:40 tyk.ilsp.gr tyk[49591]: time=“Sep 18 08:37:40” level=info msg=“reload: complete”

and Tyk Dashboard Logs
Sep 18 08:35:35 tyk.ilsp.gr systemd[1]: Stopped Tyk API Dashboard.
Sep 18 08:36:29 tyk.ilsp.gr systemd[1]: Started Tyk API Dashboard.
Sep 18 08:36:29 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:36:29” level=info msg=“Using /opt/tyk-dashboard/tyk_analytics.conf for configuration”
Sep 18 08:36:29 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:36:29” level=warning msg=“Default tyk_api_config.secret 352d20ee67be67f6340b4c0605b044b7 should be changed for production use.”
Sep 18 08:36:29 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:36:29” level=warning msg=“Default admin_secret 12345 should be changed for production use.”
Sep 18 08:36:29 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:36:29” level=warning msg=“Default shared_node_secret 352d20ee67be67f6340b4c0605b044b7 should be changed for production use.”
Sep 18 08:36:29 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:36:29” level=info msg=“Connecting to MongoDB: [127.0.0.1:27017]”
Sep 18 08:36:29 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:36:29” level=info msg=“Mongo connection established”
Sep 18 08:36:29 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:36:29” level=info msg=“Creating new Redis connection pool”
Sep 18 08:36:29 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:36:29” level=info msg=“Creating new Redis connection pool”
Sep 18 08:36:29 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:36:29” level=info msg=“Creating new Redis connection pool”
Sep 18 08:36:29 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:36:29” level=info msg=“Creating new Redis connection pool”
Sep 18 08:36:29 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:36:29” level=info msg=“Adding available nodes…”
Sep 18 08:36:29 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:36:29” level=info msg=“Audit log is disabled in config”
Sep 18 08:36:29 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:36:29” level=info msg=“Tyk Analytics Dashboard v1.8.5”
Sep 18 08:36:29 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:36:29” level=info msg=“Copyright Tyk Technologies Ltd 2019”
Sep 18 08:36:29 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:36:29” level=info msg=“https://www.tyk.io”
Sep 18 08:36:29 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:36:29” level=info msg=“Listening on port: 3000”
Sep 18 08:36:29 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:36:29” level=info msg=“Registering nodes…”
Sep 18 08:36:29 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:36:29” level=info msg=“Adding available nodes…”
Sep 18 08:36:29 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:36:29” level=info msg=“Creating new Redis connection pool”
Sep 18 08:36:29 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:36:29” level=info msg=“Socket server started”
Sep 18 08:36:29 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:36:29” level=info msg=“–> Standard listener (http) for UI notifications” addr=“:5000”
Sep 18 08:36:29 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:36:29” level=info msg=“–> Standard listener (http) for dashboard and API”
Sep 18 08:36:29 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:36:29” level=info msg=“Starting zeroconf heartbeat”
Sep 18 08:36:29 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:36:29” level=info msg=“Starting notification handler for gateway cluster”
Sep 18 08:36:29 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:36:29” level=info msg=“Loading routes…”
Sep 18 08:36:29 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:36:29” level=info msg=“Generating portal on the same domain as dashboard: tyk.ilsp.gr”
Sep 18 08:36:29 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:36:29” level=info msg=“Generating portal on the custom domain: developers.ilsp.gr”
Sep 18 08:37:00 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:37:00” level=info msg=“Got configuration for nodeID: |tyk.ilsp.gr”
Sep 18 08:37:03 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:37:03” level=info msg=“Sending config request for node: -tyk.ilsp.gr”
Sep 18 08:37:03 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:37:03” level=info msg=“Got configuration for nodeID: |tyk.ilsp.gr”
Sep 18 08:37:05 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:37:05” level=error msg=“No nodes available”
Sep 18 08:37:13 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:37:13” level=info msg=“Sending config request for node: -tyk.ilsp.gr”
Sep 18 08:37:13 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:37:13” level=info msg=“Got configuration for nodeID: |tyk.ilsp.gr”
Sep 18 08:37:15 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:37:15” level=error msg=“No nodes available”
Sep 18 08:37:23 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:37:23” level=info msg=“Sending config request for node: -tyk.ilsp.gr”
Sep 18 08:37:23 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:37:23” level=info msg=“Got configuration for nodeID: |tyk.ilsp.gr”
Sep 18 08:37:25 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:37:25” level=error msg=“No nodes available”
Sep 18 08:37:33 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:37:33” level=info msg=“Sending config request for node: -tyk.ilsp.gr”
Sep 18 08:37:33 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:37:33” level=info msg=“Got configuration for nodeID: |tyk.ilsp.gr”
Sep 18 08:37:35 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:37:35” level=error msg=“No nodes available”
Sep 18 08:37:43 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:37:43” level=info msg=“Sending config request for node: -tyk.ilsp.gr”
Sep 18 08:37:43 tyk.ilsp.gr tyk-analytics[49511]: time=“Sep 18 08:37:43” level=info msg=“Got configuration for nodeID: |tyk.ilsp.gr”

leon · September 18, 2019, 6:10am

Pls send message to [email protected] with your license key

nsidir · September 18, 2019, 6:31am

I will do so. I must let you know that before I posted this post, I had received new 3-month developer licence keys. But ‘CSRF Failure’ keeps appearing.

leon · September 18, 2019, 6:57am

You can overcome this issue by directly updating tyk_analytics.conf file with new license

nsidir · September 18, 2019, 7:37am

I tried entering the new license key directly to tyk_analytics.conf file and then rebooting, but i keep geting the same error.