Tyk.conf's enable_analytics: true not working

Hi there,

this seems to me like the 101 of the Tyk Gateway but I can’t seem to get the gateway storing analytics data on Redis. A connection to Redis is established as the GW is storing the apikey-xxxxxx keys correctly but there’s no trace of analytics. Therefore, Tyk Pump is unable to pump data to the dashboard.

I’m using a dockerised version of the GW:

FROM tykio/tyk-gateway:latest
COPY tyk.conf /opt/tyk-gateway

And my tyk.conf:

 {
  "listen_port": 8080,
  "secret": "352d20ee67be67f6340b4c0605b044b7",
  "template_path": "/opt/tyk-gateway/templates",
  "use_db_app_configs": false,
  "app_path": "/opt/tyk-gateway/apps",
  "middleware_path": "/opt/tyk-gateway/middleware",
  "storage": {
    "type": "redis",
    "host": "redis",
    "port": 6379,
    "username": "",
    "password": "",
    "database": 0,
    "optimisation_max_idle": 2000,
    "optimisation_max_active": 4000
  },
  "enable_analytics": true,
  "analytics_config": {
    "enable_detailed_recording": true
    "type": "redis",
    "ignored_ips": [],
    "enable_geo_ip": false,
    "geo_ip_db_path": "",
    "normalise_urls": {
      "enabled": true,
      "normalise_uuids": true,
      "normalise_numbers": true,
      "custom_patterns": []
    }
  },
  "health_check": {
    "enable_health_checks": false,
    "health_check_value_timeouts": 60
  },
  "dns_cache": {
    "enabled": false,
    "ttl": 3600
  },
  "policies": {
    "policy_source": "file",
    "policy_record_name": "/opt/tyk-gateway/policies/policies.json"
  },
  "optimisations_use_async_session_write": true,
  "allow_master_keys": false,
  "hash_keys": true,
  "suppress_redis_signal_reload": false,
  "close_connections": false,
  "enable_non_transactional_rate_limiter": true,
  "enable_sentinel_rate_limiter": false,
  "local_session_cache": {
    "disable_cached_session_state": false
  },
  "uptime_tests": {
    "disable": false,
    "config": {
      "enable_uptime_analytics": false,
      "failure_trigger_sample_size": 2,
      "time_wait": 10,
      "checker_pool_size": 50
    }
  },
  "http_server_options": {
    "enable_http2": true,
    "enable_websockets": true
  },
  "hostname": "",
  "enable_custom_domains": true,
  "proxy_enable_http2": true,
  "enable_jsvm": true,
  "oauth_redirect_uri_separator": ";",
  "coprocess_options": {
    "enable_coprocess": false,
    "coprocess_grpc_server": "",
    "python_path_prefix": "/opt/tyk-gateway"
  },
  "pid_file_location": "./tyk-gateway.pid",
  "allow_insecure_configs": true,
  "public_key_path": "",
  "close_idle_connections": false,
  "allow_remote_config": false,
  "enable_bundle_downloader": true,
  "bundle_base_url": "",
  "global_session_lifetime": 100,
  "force_global_session_lifetime": false,
  "max_idle_connections_per_host": 500
}

I’ve got 4 working API definition, one keyless and the rest with JWT auth (couldn’t get the key signature validation working either but that’s another topic).
After a while hitting some API endpoints, this is how the redis database 0 looks like:

image

Appreciate any help to get the analytics stored on Redis so Pump can do its job.
Thanks!

Hi

You mention the dashboard but this config is for the CE gateway which does not use the dashboard so wondering if you have some confusion there.

For this analytics issue can you share logs from gateway restart with debug logging enabled.

I also noticed that the JSON isn’t valid there is a missing comma after "enable_detailed_recording": true

Cheers
Josh

Thanks Josh for your response,

I meant the CE gateway. First thing I am trying to accomplish is to set the GW up with analytics and then use Pump to feed MongoDB for the Dashboard but that would be the second step.

Regarding the missing comma, thanks for pointing that out I was tweaking the config file just before copying and pasting it into this post. With the comma in, it doesn’t make any difference though.

How can I restart the gateway with --debug when using docker? I took a look at the entrypoint.sh from https://github.com/TykTechnologies/tyk-gateway-docker/blob/master/entrypoint.sh and there isn’t actually any env variable I could potentially use.

You can use TYK_LOGLEVEL as an env var. alternatively "log_level": true in the config works too.

Funny thing is i just copied your config and it works fine for me so wondering if there is something else going on here too.

What do you mean by “it works for you”? The fact about the missing comma?

Anyway, here’s the startup logs where I can see a redis related error:

apigw            | time="May 19 15:09:34" level=debug msg="Error trying to get value:redis: nil"


apigw            | time="May 19 15:09:34" level=debug msg="Using /opt/tyk-gateway/tyk.conf for configuration" prefix=main
apigw            | time="May 19 15:09:34" level=info msg="Tyk API Gateway v2.9.4.1" prefix=main
apigw            | time="May 19 15:09:34" level=debug msg="Connecting to redis cluster"
apigw            | time="May 19 15:09:34" level=debug msg="Creating new Redis connection pool"
apigw            | time="May 19 15:09:34" level=info msg="--> [REDIS] Creating single-node client"
apigw            | time="May 19 15:09:34" level=debug msg="Session update async pool size" pool_size=50
apigw            | time="May 19 15:09:34" level=info msg="Rich plugins are disabled" prefix=coprocess
apigw            | time="May 19 15:09:34" level=debug msg="Notifier will not work in hybrid mode" prefix=main
apigw            | time="May 19 15:09:34" level=debug msg="Storage Engine already initialised..."
apigw            | time="May 19 15:09:34" level=info msg="Setting up analytics normaliser" prefix=main
apigw            | time="May 19 15:09:34" level=info msg="PIDFile location set to: ./tyk-gateway.pid" prefix=main
apigw            | time="May 19 15:09:34" level=warning msg="The control_api_port should be changed for production" prefix=main
apigw            | time="May 19 15:09:34" level=warning msg="Insecure configuration allowed: allow_insecure_configs: true" prefix=checkup
apigw            | time="May 19 15:09:34" level=warning msg="Default secret `352d20ee67be67f6340b4c0605b044b7` should be changed for production." prefix=checkup
apigw            | time="May 19 15:09:34" level=debug msg="Initialising default org store" prefix=main
apigw            | time="May 19 15:09:34" level=debug msg="Storage Engine already initialised..."
apigw            | time="May 19 15:09:34" level=debug msg="Storage Engine already initialised..."
apigw            | time="May 19 15:09:34" level=debug msg="Storage Engine already initialised..."
apigw            | time="May 19 15:09:34" level=debug msg="Storage Engine already initialised..."
apigw            | time="May 19 15:09:34" level=debug msg="Error trying to get value:redis: nil"
apigw            | time="May 19 15:09:34" level=debug msg="No Primary instance found, assuming control" prefix=host-check-mgr
apigw            | time="May 19 15:09:34" level=info msg="Redis connection pools are ready after number of retires" currRetry=0
apigw            | time="May 19 15:09:34" level=info msg="Redis connection pools are ready" prefix=main
apigw            | time="May 19 15:09:34" level=info msg="Initialising Tyk REST API Endpoints" prefix=main
apigw            | time="May 19 15:09:34" level=info msg="Starting Poller" prefix=host-check-mgr
apigw            | time="May 19 15:09:34" level=debug msg="---> Initialising checker" prefix=host-check-mgr
apigw            | time="May 19 15:09:34" level=debug msg="[HOST CHECKER] Config:TriggerLimit: 2"
apigw            | time="May 19 15:09:34" level=debug msg="[HOST CHECKER] Config:Timeout: ~10"
apigw            | time="May 19 15:09:34" level=debug msg="[HOST CHECKER] Config:WorkerPool: 50"
apigw            | time="May 19 15:09:34" level=debug msg="[HOST CHECKER] Init complete"
apigw            | time="May 19 15:09:34" level=debug msg="---> Starting checker" prefix=host-check-mgr
apigw            | time="May 19 15:09:34" level=debug msg="[HOST CHECKER] Starting..."
apigw            | time="May 19 15:09:34" level=debug msg="[HOST CHECKER] Check loop started..."
apigw            | time="May 19 15:09:34" level=debug msg="[HOST CHECKER] Host reporter started..."
apigw            | time="May 19 15:09:34" level=debug msg="---> Checker started." prefix=host-check-mgr
apigw            | time="May 19 15:09:34" level=debug msg="Loaded API Endpoints" prefix=main
apigw            | time="May 19 15:09:34" level=info msg="--> Standard listener (http)" port=":8080" prefix=main
apigw            | time="May 19 15:09:34" level=warning msg="Starting HTTP server on:[::]:8080" prefix=main
apigw            | time="May 19 15:09:34" level=info msg="Initialising distributed rate limiter" prefix=main
apigw            | time="May 19 15:09:34" level=debug msg="DRL: Setting node ID: solo-78492e01-3d00-4b52-8390-eb2ac754b7b2|f05a2abb4817"
apigw            | time="May 19 15:09:34" level=info msg="Tyk Gateway started (v2.9.4.1)" prefix=main
apigw            | time="May 19 15:09:34" level=info msg="--> Listening on address: (open interface)" prefix=main
apigw            | time="May 19 15:09:34" level=info msg="--> Listening on port: 8080" prefix=main
apigw            | time="May 19 15:09:34" level=info msg="--> PID: 6" prefix=main
apigw            | time="May 19 15:09:34" level=info msg="Starting gateway rate limiter notifications..."
apigw            | time="May 19 15:09:34" level=debug msg="Default JSVM timeout used: 5s" prefix=jsvm
apigw            | time="May 19 15:09:34" level=info msg="Loading policies" prefix=main
apigw            | time="May 19 15:09:34" level=info msg="Policies found (1 total):" prefix=main
apigw            | time="May 19 15:09:34" level=info msg=" - 5e189590801287e42a6cf5ce" prefix=main
apigw            | time="May 19 15:09:34" level=info msg="Loading API Specification from /opt/tyk-gateway/apps/1000.json"
apigw            | time="May 19 15:09:34" level=debug msg="Checking for transform paths..."
apigw            | time="May 19 15:09:34" level=debug msg="Checking for transform paths..."
apigw            | time="May 19 15:09:34" level=info msg="Loading API Specification from /opt/tyk-gateway/apps/1001.json"
apigw            | time="May 19 15:09:34" level=debug msg="Checking for transform paths..."
apigw            | time="May 19 15:09:34" level=debug msg="Checking for transform paths..."
apigw            | time="May 19 15:09:34" level=info msg="Loading API Specification from /opt/tyk-gateway/apps/1002.json"
apigw            | time="May 19 15:09:34" level=debug msg="Checking for transform paths..."
apigw            | time="May 19 15:09:34" level=debug msg="Checking for transform paths..."
apigw            | time="May 19 15:09:34" level=info msg="Loading API Specification from /opt/tyk-gateway/apps/1003.json"
apigw            | time="May 19 15:09:34" level=debug msg="Checking for transform paths..."
apigw            | time="May 19 15:09:34" level=debug msg="Checking for transform paths..."
apigw            | time="May 19 15:09:34" level=info msg="Loading API Specification from /opt/tyk-gateway/apps/1004.json"
apigw            | time="May 19 15:09:34" level=debug msg="Checking for transform paths..."
apigw            | time="May 19 15:09:34" level=debug msg="Checking for transform paths..."
apigw            | time="May 19 15:09:34" level=info msg="Detected 5 APIs" prefix=main
apigw            | time="May 19 15:09:34" level=info msg="Loading API configurations." prefix=main
apigw            | time="May 19 15:09:34" level=info msg="Tracking hostname" api_name="Discovery API" domain="(no host)" prefix=main
apigw            | time="May 19 15:09:34" level=info msg="Tracking hostname" api_name="Inventory API" domain="(no host)" prefix=main
apigw            | time="May 19 15:09:34" level=info msg="Tracking hostname" api_name="Admin API" domain="(no host)" prefix=main
apigw            | time="May 19 15:09:34" level=info msg="Tracking hostname" api_name="Runny API" domain="(no host)" prefix=main
apigw            | time="May 19 15:09:34" level=info msg="Tracking hostname" api_name="Auth API" domain="(no host)" prefix=main
apigw            | time="May 19 15:09:34" level=info msg="Initialising Tyk REST API Endpoints" prefix=main
apigw            | time="May 19 15:09:34" level=debug msg="Loaded API Endpoints" prefix=main
apigw            | time="May 19 15:09:34" level=debug msg="Storage Engine already initialised..."
apigw            | time="May 19 15:09:34" level=info msg="API bind on custom port:0" prefix=main
apigw            | time="May 19 15:09:34" level=debug msg="Storage Engine already initialised..."
apigw            | time="May 19 15:09:34" level=debug msg="Storage Engine already initialised..."
apigw            | time="May 19 15:09:34" level=debug msg="Storage Engine already initialised..."
apigw            | time="May 19 15:09:34" level=debug msg="Initializing API" api_id=1001 api_name="Discovery API" org_id=1
apigw            | time="May 19 15:09:34" level=debug msg="Connecting to redis cluster"
apigw            | time="May 19 15:09:34" level=debug msg="Creating new Redis connection pool"
apigw            | time="May 19 15:09:34" level=info msg="--> [REDIS] Creating single-node client"
apigw            | time="May 19 15:09:34" level=info msg="Checking security policy: Open" api_id=1001 api_name="Discovery API" org_id=1
apigw            | time="May 19 15:09:34" level=debug msg="CORS ENABLED" prefix=main
apigw            | time="May 19 15:09:34" level=debug msg=Init api_id=1001 api_name="Discovery API" mw=VersionCheck org_id=1
apigw            | time="May 19 15:09:34" level=debug msg=Init api_id=1001 api_name="Discovery API" mw=RateCheckMW org_id=1
apigw            | time="May 19 15:09:34" level=debug msg=Init api_id=1001 api_name="Discovery API" mw=MiddlewareContextVars org_id=1
apigw            | time="May 19 15:09:34" level=debug msg=Init api_id=1001 api_name="Discovery API" mw=TransformHeaders org_id=1
apigw            | time="May 19 15:09:34" level=debug msg=Init api_id=1001 api_name="Discovery API" mw=RedisCacheMiddleware org_id=1
apigw            | time="May 19 15:09:34" level=debug msg="Setting Listen Path: /auth-discovery/" api_id=1001 api_name="Discovery API" org_id=1
apigw            | time="May 19 15:09:34" level=info msg="API Loaded" api_id=1001 api_name="Discovery API" org_id=1 prefix=gateway server_name=-- user_id=-- user_ip=--
apigw            | time="May 19 15:09:34" level=info msg="API bind on custom port:0" prefix=main
apigw            | time="May 19 15:09:34" level=debug msg="Storage Engine already initialised..."
apigw            | time="May 19 15:09:34" level=debug msg="Storage Engine already initialised..."
apigw            | time="May 19 15:09:34" level=debug msg="Storage Engine already initialised..."
apigw            | time="May 19 15:09:34" level=debug msg="Initializing API" api_id=1000 api_name="Inventory API" org_id=1
apigw            | time="May 19 15:09:34" level=debug msg="Storage Engine already initialised..."
apigw            | time="May 19 15:09:34" level=info msg="Checking security policy: Open" api_id=1000 api_name="Inventory API" org_id=1
apigw            | time="May 19 15:09:34" level=debug msg="CORS ENABLED" prefix=main
apigw            | time="May 19 15:09:34" level=debug msg=Init api_id=1000 api_name="Inventory API" mw=VersionCheck org_id=1
apigw            | time="May 19 15:09:34" level=debug msg=Init api_id=1000 api_name="Inventory API" mw=RateCheckMW org_id=1
apigw            | time="May 19 15:09:34" level=debug msg=Init api_id=1000 api_name="Inventory API" mw=MiddlewareContextVars org_id=1
apigw            | time="May 19 15:09:34" level=debug msg=Init api_id=1000 api_name="Inventory API" mw=TransformHeaders org_id=1
apigw            | time="May 19 15:09:34" level=debug msg=Init api_id=1000 api_name="Inventory API" mw=RedisCacheMiddleware org_id=1
apigw            | time="May 19 15:09:34" level=debug msg="Setting Listen Path: /inventory-api/" api_id=1000 api_name="Inventory API" org_id=1
apigw            | time="May 19 15:09:34" level=info msg="API Loaded" api_id=1000 api_name="Inventory API" org_id=1 prefix=gateway server_name=-- user_id=-- user_ip=--
apigw            | time="May 19 15:09:34" level=info msg="API bind on custom port:0" prefix=main
apigw            | time="May 19 15:09:34" level=debug msg="Storage Engine already initialised..."
apigw            | time="May 19 15:09:34" level=debug msg="Storage Engine already initialised..."
apigw            | time="May 19 15:09:34" level=debug msg="Storage Engine already initialised..."
apigw            | time="May 19 15:09:34" level=debug msg="Initializing API" api_id=1002 api_name="Admin API" org_id=1
apigw            | time="May 19 15:09:34" level=debug msg="Storage Engine already initialised..."
apigw            | time="May 19 15:09:34" level=info msg="Checking security policy: Open" api_id=1002 api_name="Admin API" org_id=1
apigw            | time="May 19 15:09:34" level=debug msg="CORS ENABLED" prefix=main
apigw            | time="May 19 15:09:34" level=debug msg=Init api_id=1002 api_name="Admin API" mw=VersionCheck org_id=1
apigw            | time="May 19 15:09:34" level=debug msg=Init api_id=1002 api_name="Admin API" mw=RateCheckMW org_id=1
apigw            | time="May 19 15:09:34" level=debug msg=Init api_id=1002 api_name="Admin API" mw=MiddlewareContextVars org_id=1
apigw            | time="May 19 15:09:34" level=debug msg=Init api_id=1002 api_name="Admin API" mw=TransformHeaders org_id=1
apigw            | time="May 19 15:09:34" level=debug msg=Init api_id=1002 api_name="Admin API" mw=RedisCacheMiddleware org_id=1
apigw            | time="May 19 15:09:34" level=debug msg="Setting Listen Path: /auth-admin/" api_id=1002 api_name="Admin API" org_id=1
apigw            | time="May 19 15:09:34" level=info msg="API Loaded" api_id=1002 api_name="Admin API" org_id=1 prefix=gateway server_name=-- user_id=-- user_ip=--
apigw            | time="May 19 15:09:34" level=info msg="API bind on custom port:0" prefix=main
apigw            | time="May 19 15:09:34" level=debug msg="Storage Engine already initialised..."
apigw            | time="May 19 15:09:34" level=debug msg="Storage Engine already initialised..."
apigw            | time="May 19 15:09:34" level=debug msg="Storage Engine already initialised..."
apigw            | time="May 19 15:09:34" level=debug msg="Initializing API" api_id=1004 api_name="Runny API" org_id=1
apigw            | time="May 19 15:09:34" level=debug msg="Storage Engine already initialised..."
apigw            | time="May 19 15:09:34" level=info msg="Checking security policy: Open" api_id=1004 api_name="Runny API" org_id=1
apigw            | time="May 19 15:09:34" level=debug msg="CORS ENABLED" prefix=main
apigw            | time="May 19 15:09:34" level=debug msg=Init api_id=1004 api_name="Runny API" mw=VersionCheck org_id=1
apigw            | time="May 19 15:09:34" level=debug msg=Init api_id=1004 api_name="Runny API" mw=RateCheckMW org_id=1
apigw            | time="May 19 15:09:34" level=debug msg=Init api_id=1004 api_name="Runny API" mw=MiddlewareContextVars org_id=1
apigw            | time="May 19 15:09:34" level=debug msg=Init api_id=1004 api_name="Runny API" mw=TransformHeaders org_id=1
apigw            | time="May 19 15:09:34" level=debug msg=Init api_id=1004 api_name="Runny API" mw=RedisCacheMiddleware org_id=1
apigw            | time="May 19 15:09:34" level=debug msg="Setting Listen Path: /runny-api/" api_id=1004 api_name="Runny API" org_id=1
apigw            | time="May 19 15:09:34" level=info msg="API Loaded" api_id=1004 api_name="Runny API" org_id=1 prefix=gateway server_name=-- user_id=-- user_ip=--
apigw            | time="May 19 15:09:34" level=info msg="API bind on custom port:0" prefix=main
apigw            | time="May 19 15:09:34" level=debug msg="Storage Engine already initialised..."
apigw            | time="May 19 15:09:34" level=debug msg="Storage Engine already initialised..."
apigw            | time="May 19 15:09:34" level=debug msg="Storage Engine already initialised..."
apigw            | time="May 19 15:09:34" level=debug msg="Initializing API" api_id=1003 api_name="Auth API" org_id=1
apigw            | time="May 19 15:09:34" level=debug msg="Storage Engine already initialised..."
apigw            | time="May 19 15:09:34" level=info msg="Checking security policy: Open" api_id=1003 api_name="Auth API" org_id=1
apigw            | time="May 19 15:09:34" level=debug msg="CORS ENABLED" prefix=main
apigw            | time="May 19 15:09:34" level=debug msg=Init api_id=1003 api_name="Auth API" mw=VersionCheck org_id=1
apigw            | time="May 19 15:09:34" level=debug msg=Init api_id=1003 api_name="Auth API" mw=RateCheckMW org_id=1
apigw            | time="May 19 15:09:34" level=debug msg=Init api_id=1003 api_name="Auth API" mw=TransformHeaders org_id=1
apigw            | time="May 19 15:09:34" level=debug msg=Init api_id=1003 api_name="Auth API" mw=RedisCacheMiddleware org_id=1
apigw            | time="May 19 15:09:34" level=debug msg="Setting Listen Path: /auth-api/" api_id=1003 api_name="Auth API" org_id=1
apigw            | time="May 19 15:09:34" level=info msg="API Loaded" api_id=1003 api_name="Auth API" org_id=1 prefix=gateway server_name=-- user_id=-- user_ip=--
apigw            | time="May 19 15:09:34" level=debug msg="Checker host list" prefix=main
apigw            | time="May 19 15:09:34" level=info msg="Loading uptime tests..." prefix=host-check-mgr
apigw            | time="May 19 15:09:34" level=debug msg="--- Setting tracking list up" prefix=host-check-mgr
apigw            | time="May 19 15:09:34" level=debug msg="Reset initiated" prefix=host-check-mgr
apigw            | time="May 19 15:09:34" level=debug msg="[HOST CHECKER] Checker reset queued!"
apigw            | time="May 19 15:09:34" level=debug msg="Checker host Done" prefix=main
apigw            | time="May 19 15:09:34" level=info msg="Initialised API Definitions" prefix=main
apigw            | time="May 19 15:09:34" level=info msg="API reload complete" prefix=main
apigw            | time="May 19 15:09:44" level=debug msg="Primary instance set, I am master" prefix=host-check-mgr
apigw            | time="May 19 15:09:45" level=debug msg="[HOST CHECKER] Host list reset"
apigw            | time="May 19 15:09:54" level=debug msg="Primary instance set, I am master" prefix=host-check-mgr
apigw            | time="May 19 15:10:04" level=debug msg="Primary instance set, I am master" prefix=host-check-mgr
apigw            | time="May 19 15:10:14" level=debug msg="Primary instance set, I am master" prefix=host-check-mgr
apigw            | time="May 19 15:10:24" level=debug msg="Primary instance set, I am master" prefix=host-check-mgr
apigw            | time="May 19 15:10:34" level=debug msg="Primary instance set, I am master" prefix=host-check-mgr
apigw            | time="May 19 15:10:44" level=debug msg="Primary instance set, I am master" prefix=host-check-mgr
apigw            | time="May 19 15:10:54" level=debug msg="Primary instance set, I am master" prefix=host-check-mgr
apigw            | time="May 19 15:11:04" level=debug msg="Primary instance set, I am master" prefix=host-check-mgr
apigw            | time="May 19 15:11:15" level=debug msg="Primary instance set, I am master" prefix=host-check-mgr
apigw            | time="May 19 15:11:25" level=debug msg="Primary instance set, I am master" prefix=host-check-mgr

I took your config and used it with my local docker gateway. Then fired requests and checked for the analytics key.

From your logs this all looks good if you check redis there should be a analytics-tyk-system-analytics key up to 30 seconds after you make requests to the gateway. It will be purged after that interval.

EDIT: I checked and it will expire the record after 60s

Thanks Josh for your kind support.

I recreated all my environment and now it is working properly enabling Pump to fetch analytics from Redis.

Cheers

No problem, glad to hear!