Losing ClientIds frequently

Imported Google Group message. Original thread at: https://groups.google.com/forum/#!topic/tyk-community-support/wUTAgGflPQI Import Date: 2016-01-19 21:37:45 +0000.
Sender:[email protected].
Date:Wednesday, 16 December 2015 12:27:45 UTC.

Hey Martin,

I’ve deployed Tyk on AWS Successfully but for some reason it doesn’t persist the client Ids.

It would be great if you could elaborate as to how tyk internally uses redis and mongo and what it stores in each of those.
I made changes to redis.conf to persist all data but the client ids still vanish after a point.

Looking forward to your guidance.

Imported Google Group message.
Sender:Martin Buhr.
Date:Wednesday, 16 December 2015 12:30:15 UTC.

Hi Balaji,

By clients do you mean OAuth client IDs or Tokens you’ve created in Tyk?

Many thanks.
Martin

Imported Google Group message.
Sender:Balaji Iyengar.
Date:Wednesday, 16 December 2015 13:16:43 UTC.

OAuth Client Ids.

If you could elaborate on the internal mechanism of tyk key storage and management.

Imported Google Group message.
Sender:Martin Buhr.
Date:Wednesday, 16 December 2015 13:25:28 UTC.

Only OAuth client IDs or have you also lost regular keys?

OAuth clients are stored in Redis, and they do not expire, so they should not drop.

Thanks,
Martin

Imported Google Group message.
Sender:Balaji Iyengar.
Date:Wednesday, 16 December 2015 13:31:43 UTC.

For some reason the client ids vanish and we have to create new client ids everyday.
There are no crash logs on redis’ side as well.

Clueless as to how and why this is happening.

Imported Google Group message.
Sender:Martin Buhr.
Date:Wednesday, 16 December 2015 13:32:48 UTC.

Again - are you losing any other credentials? Regular access tokens? OAuth tokens?

  • show quoted text -

  • show quoted text -


You received this message because you are subscribed to the Google Groups “Tyk Community Support” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web, visit https://groups.google.com/d/msgid/tyk-community-support/18c33c77-ab8b-4d10-803c-049d920920f3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Imported Google Group message.
Sender:Balaji Iyengar.
Date:Wednesday, 16 December 2015 13:34:59 UTC.

Only Auth Client Ids.

We haven’t used the basic keys.

Imported Google Group message.
Sender:Martin Buhr.
Date:Wednesday, 16 December 2015 13:35:52 UTC.

And OAuth tokens generated by the clients don’t get lost?

On Wed, Dec 16, 2015 at 5:35 AM -0800, [email protected] wrote:

Only Auth Client Ids.

We haven’t used the basic keys.

You received this message because you are subscribed to the Google Groups “Tyk Community Support” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web, visit https://groups.google.com/d/msgid/tyk-community-support/2d9b9bdc-280a-4b67-828b-acb8657a43b1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Imported Google Group message.
Sender:Balaji Iyengar.
Date:Wednesday, 16 December 2015 13:50:11 UTC.

Yes. Only Client Ids are being lost.

Regards,
Balaji Iyengar
Backend - MyGlamm
Ph : +91 9769974346

  • show quoted text -

Imported Google Group message.
Sender:Martin Buhr.
Date:Wednesday, 16 December 2015 14:10:46 UTC.

That’s quite weird:

It’s not a software bug, unless they are somehow being deleted - Tyk stores it’s OAuth clients in a set delimited by the organisation ID and as individual keys, so the set key looks something like this: oauth-data.2583f6ddd8c243b3664c8eb2bc0a3b2d.oauth-clientset.oauth-clientid.

Lets take a look at what’s in this set:

127.0.0.1:6379> SMEMBERS oauth-data.2583f6ddd8c243b3664c8eb2bc0a3b2d.oauth-clientset.oauth-clientid.

  1. “{“Id”:“c34756cef6d1483c5aad79ae70d1870f”,“Secret”:“YzhmZDZlYTEtYzE5OS00OWMxLTZhYzctYzJiYTQ4N2EzNzZi”,“RedirectUri”:“http://boo.com”}”
  2. “{“Id”:“f8c1acfaace2443c61c12b52de69d7fd”,“Secret”:“ZDAxMzIzNzUtZDVhZC00MWZiLTcwYzktYWRlYjE1ZGQwOWJi”,“RedirectUri”:“http://boo3.com”}”
  3. “{“Id”:“6d86abcef8c34f506987cb1d98531bf3”,“Secret”:“MjQ1NWJkZTctMzI1OC00YzMyLTU1MzEtMDBiY2U2NGU2NDI5”,“RedirectUri”:“http://boo2.com”}”

So, there’s our clients, all stored so you can list them in the dashboard

Let’s see if the set expires at all:

127.0.0.1:6379> TTL oauth-data.2583f6ddd8c243b3664c8eb2bc0a3b2d.oauth-clientset.oauth-clientid.
(integer) -1

TTL of -1, great, it will live forever, or until the redis instance restarts

Now, this is a reference list of clients, lets check the individual keys to make sure they don’t expire, let’s take client 6d86abcef8c34f506987cb1d98531bf3:

127.0.0.1:6379> GET oauth-data.2583f6ddd8c243b3664c8eb2bc0a3b2d.oauth-clientid.6d86abcef8c34f506987cb1d98531bf3
“{“Id”:“6d86abcef8c34f506987cb1d98531bf3”,“Secret”:“MjQ1NWJkZTctMzI1OC00YzMyLTU1MzEtMDBiY2U2NGU2NDI5”,“RedirectUri”:“http://boo2.com”}”

Ok, now the TTL:

127.0.0.1:6379> TTL oauth-data.2583f6ddd8c243b3664c8eb2bc0a3b2d.oauth-clientid.6d86abcef8c34f506987cb1d98531bf3
(integer) -1

Ok, so those won’t vanish either.

Now, let’s take a look at the deletion, the only way to delete a client is to call the DeleteClient call on our OAuth server storage interface, lets see where that gets used in the codebase:

Only two places where it gets mentioned, once in the storage handler, we can safely ignore that, and secondly in the API file, if you look at the relevant section, it’s just an API handler, so you can call it with the OAuth Tyk API, it doesn’t get invoked any other way.

The only way these get deleted is if you call the Tyk REST API with a delete call.

I’m really not sure how else to say that the software isn’t magicking away your OAuth clients except that maybe you have configured redis persistence wrongly, maybe take a look at the persistence docs, and in particularly appendonly

I’ve got a test running now just to make sure the entries above don’t magic away (that would be worrying) just to be thorough, but will need to wait a few days to see if it really is losing the data. In the meantime, I’d suggest looking at your redis config (maybe add a few test keys/values manually, then restart the instance to see if they still exist?)

Cheers,
Martin

Imported Google Group message.
Sender:Balaji Iyengar.
Date:Wednesday, 16 December 2015 17:42:43 UTC.

Hey Martin,

This is the error log that we could trace.

ERRO[507824] Request failed, trying again in 10s. Error was: Post http://identity-staging.myglamm.com:8080/oauth/notify/: dial tcp: lookup identi
ty-staging.myglamm.com on 172.31.0.2:53: no such host
ERRO[507824] Too many notification attempts, aborting.
INFO[507882] Attempted access from expired key. key=qmy41XqERIesqxwhzXEOlA origin=125.99.85.118:52146 path=/v1/subscriptions
INFO[507882] [OAuth] Notifying client host
ERRO[507882] Request failed, trying again in 10s. Error was: Post http://identity-staging.myglamm.com:8080/oauth/notify/: dial tcp: lookup identi
ty-staging.myglamm.com on 172.31.0.2:53: no such host
ERRO[507892] Request failed, trying again in 10s. Error was: Post http://identity-staging.myglamm.com:8080/oauth/notify/: dial tcp: lookup identi
ty-staging.myglamm.com on 172.31.0.2:53: no such host
ERRO[507896] Failure retreiving client ID key
ERRO[507896] Key not found
ERROR: Key not found
ERRO[507896] [OAuth] OAuth response marked as error: &{0 403 The authorization server encountered an unexpected condition that prevented it from
fulfilling the request. 403 map[error:server_error error_description:The authorization server encountered an unexpected condition that prevented
it from fulfilling the request.] map[Cache-Control:[no-store]] true Key not found false {0xc8206010a0 0xc82019a140}}
ERRO[507902] Request failed, trying again in 10s. Error was: Post http://identity-staging.myglamm.com:8080/oauth/notify/: dial tcp: lookup identi
ty-staging.myglamm.com on 172.31.0.2:53: no such host
ERRO[507902] Too many notification attempts, aborting.
INFO[508160] [OAuth] Notifying client host
ERRO[508160] Request failed, trying again in 10s. Error was: Post http://identity-staging.myglamm.com:8080/oauth/notify/: dial tcp: lookup identi
ty-staging.myglamm.com on 172.31.0.2:53: no such host
INFO[508160] [OAuth] Notifying client host
ERRO[508170] Request failed, trying again in 10s. Error was: Post http://identity-staging.myglamm.com:8080/oauth/notify/: dial tcp: lookup identi
ty-staging.myglamm.com on 172.31.0.2:53: no such host
ERRO[508180] Request failed, trying again in 10s. Error was: Post http://identity-staging.myglamm.com:8080/oauth/notify/: dial tcp: lookup identi
ty-staging.myglamm.com on 172.31.0.2:53: no such host

Regards,
Balaji Iyengar
Backend - MyGlamm
Ph : +91 9769974346

  • show quoted text -

Imported Google Group message.
Sender:Martin Buhr.
Date:Wednesday, 16 December 2015 17:58:57 UTC.

That said that the key is indeed missing, or the request is malformed (unless it worked before).

Have you actually looked in your Redis DB?

Thanks,
Martin

  • show quoted text -

Imported Google Group message.
Sender:Martin Buhr.
Date:Wednesday, 16 December 2015 18:10:45 UTC.

Is this key:

qmy41XqERIesqxwhzXEOlA

On the same client ID?

Why are your notifications not working?

  • show quoted text -

Imported Google Group message.
Sender:Balaji Iyengar.
Date:Tuesday, 22 December 2015 07:25:43 UTC.

Hey Martin,

When i run redis-cli command, it always gives me segmentation fault(core dumped) and we’ve not been to locate the core dump file.

Also We’re running tyk,redis and mongo on a t2.micro instance(1 Core, 1 GB).

Do you have any minimum server configuration required?

Regards,
Balaji Iyengar
Backend - MyGlamm
Ph : +91 9769974346

  • show quoted text -