Viewing Tyk Gateway data in Redis

Anup_Rai · April 13, 2021, 4:46pm

Tyk uses Redis to store API tokens and OAuth clients as described here.

I have installed a Redis client to view this data but not able to see anything. How can we view the data which is stored by Tyk Gateway in Redis?

sedky · April 13, 2021, 6:40pm

Simply connect to the Redis server and do a keys * on the default database, that should show all the entries. by default, Tyk connects to “localhost:6379”, so make sure you are connected there as well.

Then, create a key in Tyk, and you should see an “apikey-****” entry in Redis

ahmet · April 13, 2021, 6:57pm

@Anup_Rai to provide a more concrete example for what Sedky suggested:

kubectl exec -it -n tykpro redis-db9784c88-2fn8v -- redis-cli
127.0.0.1:6379> keys *
 1) "tyk-admin-api-607582f6477fb5b377f59ab2"
 2) "tyk-node-prefix34b3d133-8bd6-45bd-77cd-b4a811c82311"
 3) "tyk-node-prefix697286bf-9b47-4a76-7215-e8370baca7a9"
 4) "redis-test-b05c27d5-2737-4d98-9e86-a6036daac446"
 5) "tyk-liveness-probe"
 6) "tyk-node-prefix373c1979-85f8-4edb-46a7-027a274100c9"
 7) "redis-test-2f492030-cd1b-49c0-8cc6-07d1495b6ddf"
 8) "redis-test-c0a2e085-6cd3-4621-a03b-e3739c54318e"
 9) "tyk-admin-api-444c9fbd-05aa-4d62-5fb4-f9a4d6da80cb"
10) "apikey-d631ef32221dd6b5d56890055411569c"
11) "tyk-nodeid-prefixnode-ids"
12) "tyk-admin-api-1660cea0b0b84a5e42344205f87ff6b2"
13) "tyk-node-prefix0fda95b7-329e-4c26-4fc1-3bf226e61905"
14) "host-checker:PollerActiveInstanceID"
15) "tyk-admin-api-f359d9b6-9a68-4d2f-719c-f3b691d70162"
16) "version-check-pump"

Please however be careful doing this in a production system. Consider KEYS as a command that should only be used in production environments with extreme care.

Anup_Rai · April 15, 2021, 6:31am

Thanks @sedky , @ahmet , I am able to see the keys and token in Redis but not the Analytics data. Below is the tyk.conf file and gateway(oss) container logs -

apiVersion: v1
data:
tyk.conf: |
{
“listen_port”: 8080,
“secret”: “352d20ee67be67f6340b4c0605b044b7”,
“template_path”: “/opt/tyk-gateway/templates”,
“tyk_js_path”: “/opt/tyk-gateway/js/tyk.js”,
“middleware_path”: “/opt/tyk-gateway/middleware”,
“use_db_app_configs”: false,
“app_path”: “/opt/tyk-gateway/apps/”,
“storage”: {
“type”: “redis”,
“host”: “redis”,
“port”: 6379,
“username”: “”,
“password”: “”,
“database”: 0,
“optimisation_max_idle”: 2000,
“optimisation_max_active”: 4000
},
“enable_analytics”: true,
“analytics_config”: {
“enable_detailed_recording”: true,
“type”: “redis”,
“ignored_ips”: ,
“enable_geo_ip”: false,
“geo_ip_db_path”: “”,
“normalise_urls”: {
“enabled”: true,
“normalise_uuids”: true,
“normalise_numbers”: true,
“custom_patterns”:
}
},
“health_check”: {
“enable_health_checks”: true,
“health_check_value_timeouts”: 60
},
“optimisations_use_async_session_write”: true,
“enable_non_transactional_rate_limiter”: true,
“enable_sentinel_rate_limiter”: false,
“enable_redis_rolling_limiter”: false,
“allow_master_keys”: false,
“policies”: {
“policy_source”: “file”,
“policy_record_name”: “/opt/tyk-gateway/policies/policies.json”
},
“hash_keys”: true,
“enable_hashed_keys_listing”: true,
“close_connections”: false,
“http_server_options”: {
“enable_websockets”: true
},
“allow_insecure_configs”: true,
“coprocess_options”: {
“enable_coprocess”: true,
“coprocess_grpc_server”: “”
},
“enable_bundle_downloader”: true,
“bundle_base_url”: “”,
“global_session_lifetime”: 100,
“force_global_session_lifetime”: false,
“max_idle_connections_per_host”: 500
}
kind: ConfigMap
metadata:
name: tyk-gateway-conf

time=“Apr 15 06:19:16” level=debug msg=“Using /opt/tyk-gateway/tyk.conf for configuration” prefix=main
time=“Apr 15 06:19:16” level=info msg=“Tyk API Gateway v3.1.0” prefix=main
time=“Apr 15 06:19:16” level=warning msg=“Insecure configuration allowed” config.allow_insecure_configs=true prefix=checkup
time=“Apr 15 06:19:16” level=warning msg=“Health Checker is deprecated and not recommended” prefix=checkup
time=“Apr 15 06:19:16” level=warning msg=“AnalyticsConfig.PoolSize unset. Defaulting to number of available CPUs” prefix=checkup runtime.NumCPU=8
time=“Apr 15 06:19:16” level=warning msg=“AnalyticsConfig.RecordsBufferSize < minimum - Overriding” minRecordsBufferSize=1000 prefix=checkup
time=“Apr 15 06:19:16” level=debug msg=“Session update async pool size” pool_size=50
time=“Apr 15 06:19:16” level=debug msg=“Setting up analytics DB connection” prefix=main
time=“Apr 15 06:19:16” level=debug msg=“No Primary instance found, assuming control” prefix=host-check-mgr
time=“Apr 15 06:19:16” level=info msg=“Starting Poller” prefix=host-check-mgr
time=“Apr 15 06:19:16” level=debug msg=“—> Initialising checker” prefix=host-check-mgr
time=“Apr 15 06:19:16” level=debug msg=“[HOST CHECKER] Config:TriggerLimit: 3”
time=“Apr 15 06:19:16” level=debug msg=“[HOST CHECKER] Config:Timeout: ~10”
time=“Apr 15 06:19:16” level=debug msg=“Analytics pool worker buffer size” workerBufferSize=125
time=“Apr 15 06:19:16” level=debug msg=“[HOST CHECKER] Config:WorkerPool: 8”
time=“Apr 15 06:19:16” level=debug msg=“Notifier will not work in hybrid mode” prefix=main
time=“Apr 15 06:19:16” level=info msg=“Setting up analytics normaliser” prefix=main
time=“Apr 15 06:19:16” level=info msg=“PIDFile location set to: /var/run/tyk/tyk-gateway.pid” prefix=main
time=“Apr 15 06:19:16” level=debug msg=“[HOST CHECKER] Init complete”
time=“Apr 15 06:19:16” level=debug msg=“—> Starting checker” prefix=host-check-mgr
time=“Apr 15 06:19:16” level=debug msg=“[HOST CHECKER] Starting…”
time=“Apr 15 06:19:16” level=debug msg=“[HOST CHECKER] Check loop started…”
time=“Apr 15 06:19:16” level=debug msg=“[HOST CHECKER] Host reporter started…”
time=“Apr 15 06:19:16” level=debug msg=“—> Checker started.” prefix=host-check-mgr
time=“Apr 15 06:19:16” level=warning msg=“The control_api_port should be changed for production” prefix=main
time=“Apr 15 06:19:16” level=debug msg=“Initialising default org store” prefix=main
time=“Apr 15 06:19:16” level=debug msg=“Starting routine for flushing network analytics” prefix=main
time=“Apr 15 06:19:16” level=info msg=“Initialising Tyk REST API Endpoints” prefix=main
time=“Apr 15 06:19:16” level=debug msg=“Connecting to redis cluster”
time=“Apr 15 06:19:16” level=debug msg=“Creating new Redis connection pool”
time=“Apr 15 06:19:16” level=info msg=“–> [REDIS] Creating single-node client”
time=“Apr 15 06:19:16” level=debug msg=“Loaded API Endpoints” prefix=main
time=“Apr 15 06:19:16” level=info msg=“–> Standard listener (http)” port=“:8080” prefix=main
time=“Apr 15 06:19:16” level=debug msg=“Connecting to redis cluster”
time=“Apr 15 06:19:16” level=debug msg=“Creating new Redis connection pool”
time=“Apr 15 06:19:16” level=info msg=“–> [REDIS] Creating single-node client”
time=“Apr 15 06:19:16” level=warning msg=“Starting HTTP server on:0.0.0.0:8080” prefix=main
time=“Apr 15 06:19:16” level=info msg=“Initialising distributed rate limiter” prefix=main
time=“Apr 15 06:19:16” level=debug msg=“DRL: Setting node ID: solo-6e116c3f-5259-4fcd-aa94-08234abf40b8|tyk-gtw-78996b597f-h2xxc”
time=“Apr 15 06:19:16” level=info msg=“Tyk Gateway started (v3.1.0)” prefix=main
time=“Apr 15 06:19:16” level=info msg=“–> Listening on address: (open interface)” prefix=main
time=“Apr 15 06:19:16” level=info msg=“–> Listening on port: 8080” prefix=main
time=“Apr 15 06:19:16” level=info msg=“–> PID: 6” prefix=main
time=“Apr 15 06:19:16” level=info msg=“Loading policies” prefix=main
time=“Apr 15 06:19:16” level=info msg=“Starting gateway rate limiter notifications…”
time=“Apr 15 06:19:16” level=info msg=“Policies found (1 total):” prefix=main
time=“Apr 15 06:19:16” level=debug msg=" - default" prefix=main
time=“Apr 15 06:19:16” level=info msg=“Loading API Specification from /opt/tyk-gateway/apps/app_sample.json”
time=“Apr 15 06:19:16” level=debug msg=“Checking for transform paths…”
time=“Apr 15 06:19:16” level=debug msg=“Checking for transform paths…”
time=“Apr 15 06:19:16” level=info msg=“Detected 1 APIs” prefix=main
time=“Apr 15 06:19:16” level=info msg=“Loading API configurations.” prefix=main
time=“Apr 15 06:19:16” level=info msg=“Tracking hostname” api_name=“Tyk Test API” domain=“(no host)” prefix=main
time=“Apr 15 06:19:16” level=info msg=“Initialising Tyk REST API Endpoints” prefix=main
time=“Apr 15 06:19:16” level=debug msg=“Loaded API Endpoints” prefix=main
time=“Apr 15 06:19:16” level=info msg=“API bind on custom port:0” prefix=main
time=“Apr 15 06:19:16” level=info msg=“Initializing HealthChecker”
time=“Apr 15 06:19:16” level=debug msg=“Initializing API” api_id=1 api_name=“Tyk Test API” org_id=default
time=“Apr 15 06:19:16” level=debug msg=“Batch requests enabled for API” prefix=main
time=“Apr 15 06:19:16” level=debug msg=Init api_id=1 api_name=“Tyk Test API” mw=VersionCheck org_id=default
time=“Apr 15 06:19:16” level=debug msg=Init api_id=1 api_name=“Tyk Test API” mw=RateCheckMW org_id=default
time=“Apr 15 06:19:16” level=info msg=“Checking security policy: Token” api_id=1 api_name=“Tyk Test API” org_id=default
time=“Apr 15 06:19:16” level=debug msg=Init api_id=1 api_name=“Tyk Test API” mw=AuthKey org_id=default
time=“Apr 15 06:19:16” level=debug msg=Init api_id=1 api_name=“Tyk Test API” mw=KeyExpired org_id=default
time=“Apr 15 06:19:16” level=debug msg=Init api_id=1 api_name=“Tyk Test API” mw=AccessRightsCheck org_id=default
time=“Apr 15 06:19:16” level=debug msg=Init api_id=1 api_name=“Tyk Test API” mw=GranularAccessMiddleware org_id=default
time=“Apr 15 06:19:16” level=debug msg=Init api_id=1 api_name=“Tyk Test API” mw=RateLimitAndQuotaCheck org_id=default
time=“Apr 15 06:19:16” level=debug msg=Init api_id=1 api_name=“Tyk Test API” mw=VersionCheck org_id=default
time=“Apr 15 06:19:16” level=debug msg=Init api_id=1 api_name=“Tyk Test API” mw=KeyExpired org_id=default
time=“Apr 15 06:19:16” level=debug msg=Init api_id=1 api_name=“Tyk Test API” mw=AccessRightsCheck org_id=default
time=“Apr 15 06:19:16” level=debug msg=“Rate limit endpoint is: /tyk-api-test/tyk/rate-limits/” api_id=1 api_name=“Tyk Test API” org_id=default
time=“Apr 15 06:19:16” level=debug msg=“Setting Listen Path: /tyk-api-test/” api_id=1 api_name=“Tyk Test API” org_id=default
time=“Apr 15 06:19:16” level=info msg=“API Loaded” api_id=1 api_name=“Tyk Test API” org_id=default prefix=gateway server_name=-- user_id=-- user_ip=–
time=“Apr 15 06:19:16” level=debug msg=“Checker host list” prefix=main
time=“Apr 15 06:19:16” level=info msg=“Loading uptime tests…” prefix=host-check-mgr
time=“Apr 15 06:19:16” level=debug msg=“— Setting tracking list up” prefix=host-check-mgr
time=“Apr 15 06:19:16” level=debug msg=“Reset initiated” prefix=host-check-mgr
time=“Apr 15 06:19:16” level=debug msg=“[HOST CHECKER] Checker reset queued!”
time=“Apr 15 06:19:16” level=debug msg=“Checker host Done” prefix=main
time=“Apr 15 06:19:16” level=info msg=“Initialised API Definitions” prefix=main
time=“Apr 15 06:19:16” level=info msg=“API reload complete” prefix=main
time=“Apr 15 06:19:26” level=debug msg=“Active Instance is: 9d245084-0471-42d7-9a51-196b744611ab” prefix=host-check-mgr
time=“Apr 15 06:19:26” level=debug msg=“— I am: 0b81fc50-c046-4469-8dae-185120e475eb” prefix=host-check-mgr
time=“Apr 15 06:19:26” level=debug msg=“New master found, no tests running” prefix=host-check-mgr
time=“Apr 15 06:19:26” level=info msg=“[HOST CHECKER] Stopping poller”
time=“Apr 15 06:19:26” level=warning msg=“Reconnecting storage: Redis is either down or ws not configured” prefix=pub-sub
time=“Apr 15 06:19:28” level=debug msg=“[HOST CHECKER] Host list reset”
time=“Apr 15 06:19:36” level=debug msg=“Active Instance is: 9d245084-0471-42d7-9a51-196b744611ab” prefix=host-check-mgr
time=“Apr 15 06:19:36” level=debug msg=“— I am: 0b81fc50-c046-4469-8dae-185120e475eb” prefix=host-check-mgr
time=“Apr 15 06:19:36” level=debug msg=“New master found, no tests running” prefix=host-check-mgr
time=“Apr 15 06:19:46” level=debug msg=“Active Instance is: 9d245084-0471-42d7-9a51-196b744611ab” prefix=host-check-mgr
time=“Apr 15 06:19:46” level=debug msg=“— I am: 0b81fc50-c046-4469-8dae-185120e475eb” prefix=host-check-mgr
time=“Apr 15 06:19:46” level=debug msg=“New master found, no tests running” prefix=host-check-mgr
time=“Apr 15 06:19:56” level=debug msg=“Error trying to get value:redis: nil”
time=“Apr 15 06:19:56” level=debug msg=“No Primary instance found, assuming control” prefix=host-check-mgr
time=“Apr 15 06:19:56” level=info msg=“Starting Poller” prefix=host-check-mgr
time=“Apr 15 06:19:56” level=debug msg=“—> Initialising checker” prefix=host-check-mgr
time=“Apr 15 06:19:56” level=debug msg=“[HOST CHECKER] Config:TriggerLimit: 3”
time=“Apr 15 06:19:56” level=debug msg=“[HOST CHECKER] Config:Timeout: ~10”
time=“Apr 15 06:19:56” level=debug msg=“[HOST CHECKER] Config:WorkerPool: 8”
time=“Apr 15 06:19:56” level=debug msg=“[HOST CHECKER] Init complete”
time=“Apr 15 06:19:56” level=debug msg=“—> Starting checker” prefix=host-check-mgr
time=“Apr 15 06:19:56” level=debug msg=“[HOST CHECKER] Starting…”
time=“Apr 15 06:19:56” level=debug msg=“[HOST CHECKER] Check loop started…”
time=“Apr 15 06:19:56” level=debug msg=“[HOST CHECKER] Host reporter started…”
time=“Apr 15 06:19:56” level=debug msg=“—> Checker started.” prefix=host-check-mgr
time=“Apr 15 06:20:06” level=debug msg=“Primary instance set, I am master” prefix=host-check-mgr

Are we missing something here?

ahmet · April 15, 2021, 6:46am

Could you check your api definition?

do_not_not_track: false

I believe this will explicitly enable analytics for the api.

Anup_Rai · April 16, 2021, 6:02am

Thanks @ahmet, After changing this parameter in API I can see the Analytics in Redis.