100% CPU and Disk IO usage from Tyk Analytics

Imported Google Group message. Original thread at: Redirecting to Google Groups Import Date: 2016-01-19 21:31:34 +0000.
Sender:Marcelo.
Date:Tuesday, 17 November 2015 14:03:54 UTC.

Hi everyone,

When I open analytics, my CPU and DiskIO reaches 100% for many minutes and sometimes I can’t edit my Users, APIs or Keys because tyk analytics was still processing chart query. I have about 4.5 million records on tyk_analytics collection. Is MongoDB records cleaning a good solution? If yes, is there any practical way to clean old tyk analytics records from mongoDB? Is there another way to increase query performance instead cleaning mongoDB?

Thanks in advance!

Imported Google Group message.
Sender:Martin Buhr.
Date:Thursday, 19 November 2015 07:24:43 UTC.

Hi Marcelo,

Yes it’s safe, it will just be re-created.

I’d look into creating the index at some point though, it’s pretty straightforward.

Cheers,
Martin

  • show quoted text -

  • show quoted text -


You received this message because you are subscribed to the Google Groups “Tyk Community Support” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web, visit https://groups.google.com/d/msgid/tyk-community-support/b52681f5-6254-444f-8b25-c3d58df887ce%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Imported Google Group message.
Sender:Marcelo.
Date:Tuesday, 17 November 2015 14:58:56 UTC.

Hi Marcelo,

Yes it will be the size of the DB causing the issue - you have two options: up the RAM on your DB so there’s more ehad room for the analytics to load or b) as you suggest, clean the DB up and kill off some old data.

Tyk can do this automatically, it will, by default set an expireAt value for each record - those values are usually set to like 100 years if no override is set, but you can override it using an Organisation object, assuming you are not using custom tokens:

https://tyk.io/v1.8/rest-api/organisation_quotas/

You will need to modify the object though to include a “data_expires”: numSeconds field.

You will also need to enable “enforce_org_data_age”: true in your tyk.conf

Finally, you will need to create an index in your MongoDB to expire the data:

Once all that is in place and you’ve restarted Tyk you should see data expiring as per the clock time set in expireAt - you’ll still need to manually get rid of some of the older stuff in your DB. It’s all timestamped through so you should be able to select and remove it quite easily.

Cheers,
Martin

  • show quoted text -

Imported Google Group message.
Sender:Martin Buhr.
Date:Thursday, 19 November 2015 02:49:50 UTC.

Hi Martin,
Thanks for reply. Suppose that analytics data are not so relevant to me. Is it safe just delete entire tyk_analytics collection? Eg execute “db.tyk_analytics.remove()” weekly?

Thanks.

Imported Google Group message.
Sender:Marcelo.
Date:Thursday, 19 November 2015 07:24:43 UTC.

Hi Marcelo,

Yes it’s safe, it will just be re-created.

I’d look into creating the index at some point though, it’s pretty straightforward.

Cheers,
Martin

  • show quoted text -

  • show quoted text -


You received this message because you are subscribed to the Google Groups “Tyk Community Support” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web, visit https://groups.google.com/d/msgid/tyk-community-support/b52681f5-6254-444f-8b25-c3d58df887ce%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Imported Google Group message.
Sender:Martin Buhr.
Date:Saturday, 21 November 2015 05:03:50 UTC.

Hi Martin, I deleted entire tyk_analytics collection except the current month. Tyk Analytics works pretty fine now! Thanks!