User identifier in analytic logs?

tom.coates · May 16, 2017, 4:32pm

Hi all,

We are currently using pump to store analytical data in ElasticSearch. We would like to present this data to our users but the only identifier that we can see being passed through which will help us link the data to our users is the “api_key”.

As API keys expire, can be revoked etc it would be nice to have an identifier that does not change.

Is there a way to pass any other data though to ElasticSearch such as our own user id to help us link the data to our users?

The only other field that may possibly help us is “oauth_id” but that is currently blank. What does this value map to in Tyk? I’m struggling to find anything about this key in the docs.

Happy to provide more information if needed.

Cheers!
Tom

tom.coates · May 17, 2017, 10:19am

After looking at the data stored in Redis, it would appear that the alias is actually available but does not appear to be pumped in to Elasticsearch. Is they any way of passing the keys alias to Elasticsearch?

I have opened an issue on GitHub. tyk_analytics Alias mapping · Issue #38 · TykTechnologies/tyk-pump · GitHub

Cheers
Tom

Martin · May 17, 2017, 1:11pm

The alias, or tags, would be the way to go here. The ES pump is a community supplied pump, so it’s not usually part of our main roadmap, the relevant part to extend is here:

github.com

TykTechnologies/tyk-pump/blob/master/pumps/elasticsearch.go#L175


      
          	case "3":
          		op := new(Elasticsearch3Operator)
          		op.esClient, err = elasticv3.NewClient(elasticv3.SetURL(urls...), elasticv3.SetSniff(conf.EnableSniffing), elasticv3.SetBasicAuth(conf.Username, conf.Password), elasticv3.SetHttpClient(httpClient))
          
          		if err != nil {
          			return op, err
          		}
          
          		// Setup a bulk processor
          		p := op.esClient.BulkProcessor().Name("TykPumpESv3BackgroundProcessor")
          		if conf.BulkConfig.Workers != 0 {
          			p = p.Workers(conf.BulkConfig.Workers)
          		}
          
          		if conf.BulkConfig.FlushInterval != 0 {
          			p = p.FlushInterval(time.Duration(conf.BulkConfig.FlushInterval) * time.Second)
          		}
          
          		if conf.BulkConfig.BulkActions != 0 {
          			p = p.BulkActions(conf.BulkConfig.BulkActions)
          		}

If you are using the OAuth flows this will map to a client ID.

The ES pump is one of the more popular community plugins, so this may get picked up - though we can’t guarantee that we’ll address it, but it seems like an easy fix if you’;d like to submit a PR?

Cheers,
Martin

tom.coates · May 17, 2017, 2:52pm

Thanks @Martin!

I’ll a take a look. I’m having a few problems with Go moaning about decencies at the moment (use of internal package not allowed) when trying to build. Once I can build and updated the ES pump I will submit a PR.

Tom

cvaram · August 8, 2019, 12:15pm

@tom.coates
by a workaround, Our team has got that you can track the clients in ElasticSearch by disabling the hashing in tyk.
It is possible to disable key hashing in Tyk using hash_keys set to false in the tyk.conf file and the tyk_analytics.conf file. Before doing these things, read this doc.
otherwise you can do it in this way also. statics have a api_key field but this key was hashed and sent it by the gateway. you can get the key by curling this request

http://www.tyk-test.com:8080/tyk/keys/${api_key}?hashed=true

cvaram · August 23, 2019, 4:49am

Hi @Martin @tom.coates
I have sent the PR. I did not see any PR about this. Alias can be used for multiple purposes. In our case, We have to identify the users/clients. Alias can be used to easily identify the users.

Best Regards,
Sivaram