Tyk Kubernetes Configuration

Support Installation and Deployment
, ,
1

I am struggling with the configuration of Tyk pump for prometheus on Kubernetes,
I saw on the documentation there is just a conf for Docker, Anyone Can help with Kubernetes ?
(Tyk Open source)
The big issue is to set it up i think, My actual config with the pump.conf is:

{
  "analytics_storage_type": "redis",
  "analytics_storage_config": {
    "type": "redis",
    "host": "tyk-redis",
    "port": 6379,
    "hosts": null,
    "username": "",
    "password": "",
    "database": 0,
    "optimisation_max_idle": 100,
    "optimisation_max_active": 100,
    "enable_cluster": false
  },
  "purge_delay": 2,
  "pumps": {
    "prometheus": {
      "type": "prometheus",
      "meta": {
		    "listen_address": ":8084",
		    "path": "/metrics",
		    "custom_metrics":[
          {
              "name":"tyk_http_requests_total",
              "description":"Total of API requests",
              "metric_type":"counter",
              "labels":["response_code","api_name","method","api_key","alias","path"]
          },
          {
              "name":"tyk_http_latency",
              "description":"Latency of API requests",
              "metric_type":"histogram",
              "labels":["type","response_code","api_name","method","api_key","alias","path"]
          }
      ]
      }
    }
  },
  "dont_purge_uptime_data": true,
  "statsd_connection_string": "graphite:8125"
}

Can Someone help :cold_face:

2

Hello @labi we have a very easy way to set this up with our new charts :slight_smile:

Let me know if that helps!

3

@zaid ,
I checked, let me show you the result.

global:
  components:
    # Determines whether the pump component should be installed.
    pump: true

  secrets:
    # APISecret sets secret in tyk.conf
    APISecret: CHANGEME
    # useSecretName can be used if you don't want to store plaintext secrets in the Helm value file and would
    # rather provide the k8s Secret externally.
    # You can set following fields in the secret
    # - APISecret - Sets node_secret and secret in tyk.conf
    # - redisPass - Sets Redis password
    # - redisSentinelPass - Sets Redis Password. Set it only if you are using Redis Sentinel
    useSecretName: ""

  redis:
    # The addrs value will allow you to set your Redis addresses.
    #
    # If you are using Redis (e.g. Bitnami Redis at bitnami/redis) then enter single
    # endpoint. If using sentinel connection mode for Redis, please update the port number (typically 26379).
    #
    # If using a Redis Cluster (e.g. bitnami/redis-cluster), you can list
    # the endpoints of the redis instances or use the cluster configuration endpoint.
    #
    # Default value: redis.{{ .Release.Namespace }}.svc.cluster.local:6379
    addrs:
    #   Example using tyk simple redis chart
    #   - redis.tyk.svc.cluster.local:6379
    #   Example using bitnami/redis
      - tyk-redis-master.tyk.svc.cluster.local:6379
    #   Example using bitnami/redis with sentinel
    #   - tyk-redis.tyk.svc.cluster.local:26379
    #   Example using bitnami/redis-cluster
    #   - tyk-redis-redis-cluster.tyk.svc.cluster.local:6379

    # Redis password
    # If you're using Bitnami Redis chart (e.g. bitnami/redis) please input
    # your password in the field below
    pass: "password"

    # Enables SSL for Redis connection. Redis instance will have to support that.
    # Default value: false
    # useSSL: true

    # If using "Redis Cluster" set enableCluster to true
    # (e.g. if using bitnami/redis-cluster)
    # enableCluster: true

    # Enables sentinel connection mode for Redis. If enabled, provide both
    # mandatory values for sentinelPass and masterName.
    # enableSentinel: false

    # Redis sentinel password, only required while enableSentinel is true.
    # For bitnami/redis the same password as Redis above
    # sentinelPass: ""

    # Redis sentinel master name, only required while enableSentinel is true.
    # For bitnami/redis typically redis-master
    # masterName: ""

    # By default the database index is 0. Setting the database index is not
    # supported with redis cluster. As such, if you have enableCluster: true,
    # then this value should be omitted or explicitly set to 0.
    storage:
      database: 0

  remoteControlPlane:
    enabled: false

tyk-gateway:
  ## Default values for tyk-gateway chart.
  ## This is a YAML-formatted file.
  ## Declare variables to be passed into your templates.
  ## See Tyk Helm documentation for installation details:
  ## https://tyk.io/docs/tyk-oss/ce-helm-chart/
  ## Registry for all Tyk images - https://hub.docker.com/u/tykio

  # nameOverride overrides the Chart name. It is truncated to 63 characters.
  # Default value: tyk-gateway.name
  nameOverride: ""

  # fullnameOverride overrides App name. It is truncated to 63 characters.
  # Default value: tyk-gateway.fullname
  fullnameOverride: ""

  gateway:
    # The hostname to bind the Gateway to.
    hostName: tyk-gw.local

    tls:
      # When true, sets the gateway protocol to HTTPS.
      enabled: false
      # When true, it will install the certificate present in the templates folder, set to false when using
      # a custom TLS certificate to avoid overwriting yours
      useDefaultTykCertificate: true
      # The name of the secret which should contain the TLS certificate you want to use with the gateway deployment
      secretName: tyk-default-tls-secret

    # kind is type of k8s object to be created for gateway.
    kind: Deployment

    # replicaCount specifies number of replicas to be created if kind is Deployment.
    replicaCount: 1

    # containerPort for gateway
    containerPort: 8080

    image:
      # image repository for Tyk Gateway
      repository: docker.tyk.io/tyk-gateway/tyk-gateway

      # image tag for Tyk Gateway
      tag: v5.0.0

      # image pull policy for Tyk Gateway
      pullPolicy: IfNotPresent

    service:
      # type of service
      type: NodePort

      # service port
      port: 443

      # external traffic policy of the service. Set it only if you are using LoadBalancer service type
      externalTrafficPolicy: Local

      # annotations for service
      annotations: {}

    control:
      # If enabled, exposes control port of the gateway
      enabled: false

      # control port of gateway
      containerPort: 9696

      # port number for control port service
      port: 9696

      # service type for control port service
      type: ClusterIP

      # annotations for control port service
      annotations: {}

    # Creates an ingress object in k8s. Will require an ingress-controller and
    # annotation to that ingress controller.
    ingress:
      # if enabled, creates an ingress resource for the gateway
      enabled: false

      # specify ingress controller class name
      className: ""

      # annotations for ingress
      annotations: {}

      # ingress rules
      hosts:
        - host: chart-example.local
          paths:
            - path: /
              pathType: ImplementationSpecific

      # tls configuration for ingress
      #  - secretName: chart-example-tls
      #    hosts:
      #      - chart-example.local
      tls: []

    # We usually recommend not to specify default resources and to leave this
    # as a conscious choice for the user. This also increases chances charts
    # run on environments with little resources, such as Minikube. If you do
    # want to specify resources, uncomment the following lines, adjust them
    # as necessary, and remove the curly braces after 'resources:'.
    # limits:
    #  cpu: 100m
    #  memory: 128Mi
    # requests:
    #  cpu: 100m
    #  memory: 128Mi
    resources: {}

    # securityContext values for gateway pod
    securityContext:
      runAsUser: 1000
      fsGroup: 2000

    # node labels for gateway pod assignment
    nodeSelector: {}

    # tolerations for gateway pod assignment
    tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule

    # affinity for gateway pod assignment
    affinity: {}

    # extraEnvs is used to set gateway env variables
    # - name: TYK_GW_HTTPSERVEROPTIONS_SSLINSECURESKIPVERIFY
    #   value: "true"
    extraEnvs: []

    ## extraVolumes is a list of volumes to be added to the pod
    ## extraVolumes:
    ##   - name: ca-certs
    ##     secret:
    ##       defaultMode: 420
    ##       secretName: ca-certs
    extraVolumes: []

    ## extraVolumeMounts is a list of volume mounts to be added to the pod
    ## extraVolumeMounts:
    ##   - name: ca-certs
    ##     mountPath: /etc/ssl/certs/ca-certs.crt
    ##     readOnly: true
    extraVolumeMounts: []

    # used to mount files to the gateway containers
    #  - name: test
    #    filename: test.txt
    #    mountPath: /opt/test
    mounts: []

tyk-pump:
  ## Default values for tyk-pump chart.
  ## This is a YAML-formatted file.
  ## Declare variables to be passed into your templates.
  ## See Tyk Helm documentation for installation details:
  ## https://tyk.io/docs/tyk-oss/ce-helm-chart/
  ## Registry for all Tyk images - https://hub.docker.com/u/tykio

  # Overrides chart name. It is truncated to 63 characters.
  # Default value: tyk-pump.name
  nameOverride: ""

  # Overrides app name. IT is truncated to 63 characters.
  # Default value: tyk-pump.fullname
  fullnameOverride: ""

  # If pump is enabled the Gateway will create and collect analytics data to send
  # to a data store of your choice. These can be set up in the pump config. The
  # possible pump configs can be found here:
  # https://github.com/TykTechnologies/tyk-pump#configuration
  pump:
    # number for replicas for pump deployment
    replicaCount: 1

    image:
      # image repository for Tyk pump
      repository: docker.tyk.io/tyk-pump/tyk-pump

      # tag for Tyk pump
      tag: v1.8.0

      # image pull policy
      pullPolicy: IfNotPresent

    # backend defines the pumps to be created by default, as an array of string.
    # Supported backends are ["mongo", "postgres", "prometheus"]
    # If you would like to use other backends such as ElasticSearch, please
    # configure the backend via environment variables.
    # By default, Tyk Pump uses Prometheus.
    backend:
      - "prometheus"

    # uptimePumpBackend configures uptime Tyk Pump. ["", "mongo", "postgres"].
    # Set it to "" for disabling uptime Tyk Pump. By default, uptime pump is disabled.
    uptimePumpBackend: ""

    # prometheusPump configures Tyk Pump to expose Prometheus metrics.
    # Please add "prometheus" to .Values.pump.backend in order to enable Prometheus Pump.
    prometheusPump:
      # host represents the host without port, where Tyk Pump serve the metrics for Prometheus.
      host: "localhost"
      # port represents the port where Tyk Pump serve the metrics for Prometheus.
      port: 9090
      # path represents the path to the Prometheus metrics endpoint. For example /metrics.
      path: /metrics
      # customMetrics allows defining custom Prometheus metrics for Tyk Pump.
      # It accepts a string that represents a JSON object. For instance,
      #
      customMetrics: '[{"name":"tyk_http_requests_total","description":"Total of API requests","metric_type":"counter","labels":["response_code","api_name","method","api_key","alias","path"]},          {              "name":"tyk_http_latency",              "description":"Latency of API requests",              "metric_type":"histogram",              "labels":["type","response_code","api_name","method","api_key","alias","path"]          }]'
      # customMetrics: ""
      # If you are using prometheus Operator, set the fields in the section below.
      prometheusOperator:
        # enabled determines whether the Prometheus Operator is in use or not. By default,
        # it is disabled.
        # Tyk Pump can be monitored with PodMonitor Custom Resource of Prometheus Operator.
        # If enabled, PodMonitor resource is created based on .Values.pump.prometheusPump.prometheusOperator.podMonitorSelector
        # for Tyk Pump.
        enabled: false
        # podMonitorSelector represents a podMonitorSelector of your Prometheus resource. So that
        # your Prometheus resource can select PodMonitor objects based on selector defined here.
        # Please set this field to the podMonitorSelector field of your monitoring.coreos.com/v1
        # Prometheus resource's spec.
        #
        # You can check the podMonitorSelector via:
        #   kubectl describe prometheuses.monitoring.coreos.com <PROMETHEUS_POD>
        podMonitorSelector:
          release: prometheus-stack

    # annotations for pump deployment
    annotations: {}

    # We usually recommend not to specify default resources and to leave this
    # as a conscious choice for the user. This also increases chances charts
    # run on environments with little resources, such as Minikube. If you do
    # want to specify resources, uncomment the following lines, adjust them
    # as necessary, and remove the curly braces after 'resources:'.
    # limits:
    #  cpu: 100m
    #  memory: 128Mi
    # requests:
    #  cpu: 100m
    #  memory: 128Mi
    resources: {}

    # securityContext values for pump pod
    securityContext:
      runAsUser: 1000
      fsGroup: 2000

    # node labels for pump pod assignment
    nodeSelector: {}

    # tolerations for pump pod assignment
    tolerations: []

    # affinity for pump pod assignment
    affinity: {}

    # extraEnvs is used to set environment variables in pump container
    # - name: TYK_PMP_PURGEDELAY
    #   value: 30
    extraEnvs: []

    ## extraVolumes is a list of volumes to be added to the pod
    ## extraVolumes:
    ##   - name: ca-certs
    ##     secret:
    ##       defaultMode: 420
    ##       secretName: ca-certs
    extraVolumes: []

    ## extraVolumeMounts is a list of volume mounts to be added to the pod
    ## extraVolumeMounts:
    ##   - name: ca-certs
    ##     mountPath: /etc/ssl/certs/ca-certs.crt
    ##     readOnly: true
    extraVolumeMounts: []

    # mounts can be used to mount files on pump container
    # - name: test
    #   filename: test.csv
    #   mountPath: /opt/test
    mounts: []

I have the pump-tyk-oss-tyk-pump-854b6964cc-fdm7w pod and also gateway-tyk-oss-tyk-gateway-6b74b568d-q8f68 pod

Here is the log of the tyk-pump:

Screenshot 2023-06-14 at 8.25.47 PM
Screenshot 2023-06-14 at 8.25.47 PM1095Ă—288 55.3 KB

But in an still not getting the metrics, even tyk_http_requests_total or tyk_http_latency
I just install tyk-oss you showed me, and also redis and update the value with redis.

4

The host value for prometheus should be 0.0.0.0 to allow prometheus to connect. The host is what hostname do you want the service to bind to.

Can you try that and let me know. Also can you please do a port-forward to the pump on port 9090 and and share the results?

5

ok; @zaid I change the host with this:

Screenshot 2023-06-15 at 12.55.34 AM
Screenshot 2023-06-15 at 12.55.34 AM1007Ă—232 37.4 KB

An then the

Screenshot 2023-06-15 at 12.56.54 AM
Screenshot 2023-06-15 at 12.56.54 AM1101Ă—67 8.46 KB

and the result on
localhost:9090/metrics is:


# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 5.4881e-05
go_gc_duration_seconds{quantile="0.25"} 0.000102968
go_gc_duration_seconds{quantile="0.5"} 0.000142151
go_gc_duration_seconds{quantile="0.75"} 0.000165738
go_gc_duration_seconds{quantile="1"} 0.00186975
go_gc_duration_seconds_sum 0.00349122
go_gc_duration_seconds_count 14
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 12
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.16.15"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 3.073824e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 3.4183808e+07
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.453667e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 443223
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 2.4099699019355685e-05
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 5.071472e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 3.073824e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 6.2185472e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 4.399104e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 27351
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 6.1759488e+07
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.6584576e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.686790631303133e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 470574
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 2400
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 76024
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 114688
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 4.194304e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 537397
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 491520
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 491520
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.4269704e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 8
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.99
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 11
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 3.3005568e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.68678997849e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 7.70048e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 6
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0

Maybe i am wrong, somewhere :cold_face:

Just to be sure, i need prometheus, tyk and redis, to make it work right ?

Do i need to install mongo and configure it ?
How can i be sure that the pump is connected to redis ?

6

Yes this is all you need.

First of all what you sent is the metrics from the prom /metrics endpoint. Are you port-forwarding both to your local 9090 by any chance?

Did you setup the scrapper on prometheus side?

7

No @zaid , i am just forwarding the pump, that’is the screen i showed you.

Now i stop forwarding the pum, and forward the prometheus-server, i am getting on localhost:9090/metrics :

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 9.2057e-05
go_gc_duration_seconds{quantile="0.25"} 0.000162908
go_gc_duration_seconds{quantile="0.5"} 0.000240094
go_gc_duration_seconds{quantile="0.75"} 0.000319364
go_gc_duration_seconds{quantile="1"} 0.004309754
go_gc_duration_seconds_sum 0.009638485
go_gc_duration_seconds_count 18
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 199
# HELP go_info Information about the Go environment.
net_conntrack_dialer_conn_failed_total{dialer_name="kubernetes-services",reason="refused"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="kubernetes-services",reason="resolution"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="kubernetes-services",reason="timeout"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="kubernetes-services",reason="unknown"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="prometheus",reason="refused"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="prometheus",reason="resolution"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="prometheus",reason="timeout"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="prometheus",reason="unknown"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="prometheus-pushgateway",reason="refused"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="prometheus-pushgateway",reason="resolution"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="prometheus-pushgateway",reason="timeout"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="prometheus-pushgateway",reason="unknown"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="traefik",reason="refused"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="traefik",reason="resolution"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="traefik",reason="timeout"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="traefik",reason="unknown"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="tyk",reason="refused"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="tyk",reason="resolution"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="tyk",reason="timeout"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="tyk",reason="unknown"} 0
# HELP net_conntrack_listener_conn_accepted_total Total number of connections opened to the listener of a given name.
# TYPE net_conntrack_listener_conn_accepted_total counter
net_conntrack_listener_conn_accepted_total{listener_name="http"} 92
# HELP net_conntrack_listener_conn_closed_total Total number of connections closed that were made to the listener of a given name.
# TYPE net_conntrack_listener_conn_closed_total counter
net_conntrack_listener_conn_closed_total{listener_name="http"} 86
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 14.42
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 59
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 5.12106496e+08
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.68679175835e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.803993088e+09
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
# HELP prometheus_api_remote_read_queries The current number of remote read queries being executed or waiting.
# TYPE prometheus_api_remote_read_queries gauge
prometheus_api_remote_read_queries 0
# HELP prometheus_build_info A metric with a constant '1' value labeled by version, revision, branch, goversion from which prometheus was built, and the goos and goarch for the build.
# TYPE prometheus_build_info gauge
prometheus_build_info{branch="HEAD",goarch="amd64",goos="linux",goversion="go1.20.4",revision="1ac5131f698ebc60f13fe2727f89b115a41f6558",tags="netgo,builtinassets,stringlabels",version="2.44.0"} 1
# HELP prometheus_config_last_reload_success_timestamp_seconds Timestamp of the last successful configuration reload.
# TYPE prometheus_config_last_reload_success_timestamp_seconds gauge
prometheus_config_last_reload_success_timestamp_seconds 1.6867917662674816e+09
# HELP prometheus_config_last_reload_successful Whether the last configuration reload attempt was successful.
# TYPE prometheus_config_last_reload_successful gauge
prometheus_config_last_reload_successful 1
# HELP prometheus_engine_queries The current number of queries being executed or waiting.
# TYPE prometheus_engine_queries gauge
prometheus_engine_queries 0
# HELP prometheus_engine_queries_concurrent_max The max number of concurrent queries.
# TYPE prometheus_engine_queries_concurrent_max gauge
prometheus_engine_queries_concurrent_max 20
# HELP prometheus_engine_query_duration_seconds Query timings
# TYPE prometheus_engine_query_duration_seconds summary
prometheus_engine_query_duration_seconds{slice="inner_eval",quantile="0.5"} 0.000947239
prometheus_engine_query_duration_seconds{slice="inner_eval",quantile="0.9"} 0.000947239
prometheus_engine_query_duration_seconds{slice="inner_eval",quantile="0.99"} 0.000947239
prometheus_engine_query_duration_seconds_sum{slice="inner_eval"} 0.000947239
prometheus_engine_query_duration_seconds_count{slice="inner_eval"} 1
prometheus_engine_query_duration_seconds{slice="prepare_time",quantile="0.5"} 1.3379e-05
prometheus_engine_query_duration_seconds{slice="prepare_time",quantile="0.9"} 1.3379e-05
prometheus_engine_query_duration_seconds{slice="prepare_time",quantile="0.99"} 1.3379e-05
prometheus_engine_query_duration_seconds_sum{slice="prepare_time"} 1.3379e-05
prometheus_engine_query_duration_seconds_count{slice="prepare_time"} 1
prometheus_engine_query_duration_seconds{slice="queue_time",quantile="0.5"} 0.00014286
prometheus_engine_query_duration_seconds{slice="queue_time",quantile="0.9"} 0.00014286
prometheus_engine_query_duration_seconds{slice="queue_time",quantile="0.99"} 0.00014286
prometheus_engine_query_duration_seconds_sum{slice="queue_time"} 0.00014286
prometheus_engine_query_duration_seconds_count{slice="queue_time"} 1
prometheus_engine_query_duration_seconds{slice="result_sort",quantile="0.5"} NaN
prometheus_engine_query_duration_seconds{slice="result_sort",quantile="0.9"} NaN
prometheus_engine_query_duration_seconds{slice="result_sort",quantile="0.99"} NaN
prometheus_engine_query_duration_seconds_sum{slice="result_sort"} 0
prometheus_engine_query_duration_seconds_count{slice="result_sort"} 0
# HELP prometheus_engine_query_log_enabled State of the query log.
# TYPE prometheus_engine_query_log_enabled gauge
prometheus_engine_query_log_enabled 0
# HELP prometheus_engine_query_log_failures_total The number of query log failures.
# TYPE prometheus_engine_query_log_failures_total counter
prometheus_engine_query_log_failures_total 0
# HELP prometheus_engine_query_samples_total The total number of samples loaded by all queries.
# TYPE prometheus_engine_query_samples_total counter
prometheus_engine_query_samples_total 0
# HELP prometheus_http_request_duration_seconds Histogram of latencies for HTTP requests.
# TYPE prometheus_http_request_duration_seconds histogram
prometheus_http_request_duration_seconds_bucket{handler="/",le="0.1"} 1
prometheus_http_request_duration_seconds_bucket{handler="/",le="0.2"} 1
prometheus_http_request_duration_seconds_bucket{handler="/",le="0.4"} 1
prometheus_http_request_duration_seconds_bucket{handler="/",le="1"} 1
prometheus_http_request_duration_seconds_bucket{handler="/",le="3"} 1
prometheus_http_request_duration_seconds_bucket{handler="/",le="8"} 1
prometheus_http_request_duration_seconds_bucket{handler="/",le="20"} 1
prometheus_http_request_duration_seconds_bucket{handler="/",le="60"} 1
prometheus_http_request_duration_seconds_bucket{handler="/",le="120"} 1
prometheus_http_request_duration_seconds_bucket{handler="/",le="+Inf"} 1
prometheus_http_request_duration_seconds_sum{handler="/"} 5.3323e-05
prometheus_http_request_duration_seconds_count{handler="/"} 1
prometheus_http_request_duration_seconds_bucket{handler="/-/healthy",le="0.1"} 21
prometheus_http_request_duration_seconds_bucket{handler="/-/healthy",le="0.2"} 21
prometheus_http_request_duration_seconds_bucket{handler="/-/healthy",le="0.4"} 21
prometheus_http_request_duration_seconds_bucket{handler="/-/healthy",le="1"} 21
prometheus_http_request_duration_seconds_bucket{handler="/-/healthy",le="3"} 21
prometheus_http_request_duration_seconds_bucket{handler="/-/healthy",le="8"} 21
prometheus_http_request_duration_seconds_bucket{handler="/-/healthy",le="20"} 21
prometheus_http_request_duration_seconds_bucket{handler="/-/healthy",le="60"} 21
prometheus_http_request_duration_seconds_bucket{handler="/-/healthy",le="120"} 21
prometheus_http_request_duration_seconds_bucket{handler="/-/healthy",le="+Inf"} 21
prometheus_http_request_duration_seconds_sum{handler="/-/healthy"} 0.000413369
prometheus_http_request_duration_seconds_count{handler="/-/healthy"} 21
prometheus_http_request_duration_seconds_bucket{handler="/-/ready",le="0.1"} 65
prometheus_http_request_duration_seconds_bucket{handler="/-/ready",le="0.2"} 65
prometheus_http_request_duration_seconds_bucket{handler="/-/ready",le="0.4"} 65
prometheus_http_request_duration_seconds_bucket{handler="/-/ready",le="1"} 65
prometheus_http_request_duration_seconds_bucket{handler="/-/ready",le="3"} 65
prometheus_http_request_duration_seconds_bucket{handler="/-/ready",le="8"} 65
prometheus_http_request_duration_seconds_bucket{handler="/-/ready",le="20"} 65
prometheus_http_request_duration_seconds_bucket{handler="/-/ready",le="60"} 65
prometheus_http_request_duration_seconds_bucket{handler="/-/ready",le="120"} 65
prometheus_http_request_duration_seconds_bucket{handler="/-/ready",le="+Inf"} 65
prometheus_http_request_duration_seconds_sum{handler="/-/ready"} 0.0017630629999999998
prometheus_http_request_duration_seconds_count{handler="/-/ready"} 65
prometheus_http_request_duration_seconds_bucket{handler="/-/reload",le="0.1"} 0
prometheus_http_request_duration_seconds_bucket{handler="/-/reload",le="0.2"} 0
prometheus_http_request_duration_seconds_bucket{handler="/-/reload",le="0.4"} 0
prometheus_http_request_duration_seconds_bucket{handler="/-/reload",le="1"} 0
prometheus_http_request_duration_seconds_bucket{handler="/-/reload",le="3"} 0
prometheus_http_request_duration_seconds_bucket{handler="/-/reload",le="8"} 1
prometheus_http_request_duration_seconds_bucket{handler="/-/reload",le="20"} 1
prometheus_http_request_duration_seconds_bucket{handler="/-/reload",le="60"} 1
prometheus_http_request_duration_seconds_bucket{handler="/-/reload",le="120"} 1
prometheus_http_request_duration_seconds_bucket{handler="/-/reload",le="+Inf"} 1
prometheus_http_request_duration_seconds_sum{handler="/-/reload"} 3.719684337
prometheus_http_request_duration_seconds_count{handler="/-/reload"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="0.1"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="0.2"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="0.4"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="1"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="3"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="8"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="20"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="60"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="120"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/label/:name/values",le="+Inf"} 1
prometheus_http_request_duration_seconds_sum{handler="/api/v1/label/:name/values"} 0.006404044
prometheus_http_request_duration_seconds_count{handler="/api/v1/label/:name/values"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="0.1"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="0.2"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="0.4"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="1"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="3"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="8"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="20"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="60"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="120"} 1
prometheus_http_request_duration_seconds_bucket{handler="/api/v1/query",le="+Inf"} 1
prometheus_http_request_duration_seconds_sum{handler="/api/v1/query"} 0.003447595
prometheus_http_request_duration_seconds_count{handler="/api/v1/query"} 1
prometheus_target_sync_length_seconds{scrape_job="traefik",quantile="0.5"} 4.7409e-05
prometheus_target_sync_length_seconds{scrape_job="traefik",quantile="0.9"} 0.000100573
prometheus_target_sync_length_seconds{scrape_job="traefik",quantile="0.99"} 0.00084105
prometheus_target_sync_length_seconds_sum{scrape_job="traefik"} 0.004709868999999999
prometheus_target_sync_length_seconds_count{scrape_job="traefik"} 68
prometheus_target_sync_length_seconds{scrape_job="tyk",quantile="0.01"} 2.604e-05
prometheus_target_sync_length_seconds{scrape_job="tyk",quantile="0.05"} 3.04e-05
prometheus_target_sync_length_seconds{scrape_job="tyk",quantile="0.5"} 4.7368e-05
prometheus_target_sync_length_seconds{scrape_job="tyk",quantile="0.9"} 0.000146353
prometheus_target_sync_length_seconds{scrape_job="tyk",quantile="0.99"} 0.01209354
prometheus_target_sync_length_seconds_sum{scrape_job="tyk"} 0.023592151999999995
prometheus_target_sync_length_seconds_count{scrape_job="tyk"} 68
# HELP prometheus_template_text_expansion_failures_total The total number of template text expansion failures.
# TYPE prometheus_template_text_expansion_failures_total counter
prometheus_template_text_expansion_failures_total 0
# HELP prometheus_template_text_expansions_total The total number of template text expansions.
# TYPE prometheus_template_text_expansions_total counter
prometheus_template_text_expansions_total 0
# HELP prometheus_treecache_watcher_goroutines The current number of watcher goroutines.
# TYPE prometheus_treecache_watcher_goroutines gauge
prometheus_treecache_watcher_goroutines 0
# HELP prometheus_treecache_zookeeper_failures_total The total number of ZooKeeper failures.
# TYPE prometheus_treecache_zookeeper_failures_total counter
prometheus_treecache_zookeeper_failures_total 0
# HELP prometheus_tsdb_blocks_loaded Number of currently loaded data blocks
# TYPE prometheus_tsdb_blocks_loaded gauge
prometheus_tsdb_blocks_loaded 7
# HELP prometheus_tsdb_checkpoint_creations_failed_total Total number of checkpoint creations that failed.
# TYPE prometheus_tsdb_checkpoint_creations_failed_total counter
prometheus_tsdb_checkpoint_creations_failed_total 0
# HELP prometheus_tsdb_checkpoint_creations_total Total number of checkpoint creations attempted.
# TYPE prometheus_tsdb_checkpoint_creations_total counter
prometheus_tsdb_checkpoint_creations_total 0
# HELP prometheus_tsdb_checkpoint_deletions_failed_total Total number of checkpoint deletions that failed.
# TYPE prometheus_tsdb_checkpoint_deletions_failed_total counter
prometheus_tsdb_checkpoint_deletions_failed_total 0
# HELP prometheus_tsdb_checkpoint_deletions_total Total number of checkpoint deletions attempted.
# TYPE prometheus_tsdb_checkpoint_deletions_total counter
prometheus_tsdb_checkpoint_deletions_total 0
# HELP prometheus_tsdb_clean_start -1: lockfile is disabled. 0: a lockfile from a previous execution was replaced. 1: lockfile creation was clean
# TYPE prometheus_tsdb_clean_start gauge
prometheus_tsdb_clean_start 1
# HELP prometheus_tsdb_compaction_chunk_range_seconds Final time range of chunks on their first compaction
# TYPE prometheus_tsdb_compaction_chunk_range_seconds histogram
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="100"} 0
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="400"} 0
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="1600"} 0
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="6400"} 0
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="25600"} 0
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="102400"} 0
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="409600"} 0
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="1.6384e+06"} 0
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="6.5536e+06"} 0
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="2.62144e+07"} 0
prometheus_tsdb_compaction_chunk_range_seconds_bucket{le="+Inf"} 0
prometheus_tsdb_compaction_chunk_range_seconds_sum 0
prometheus_tsdb_compaction_chunk_range_seconds_count 0
# HELP prometheus_tsdb_compaction_chunk_samples Final number of samples on their first compaction
# TYPE prometheus_tsdb_compaction_chunk_samples histogram
prometheus_tsdb_compaction_chunk_samples_bucket{le="4"} 0
prometheus_tsdb_compaction_chunk_samples_bucket{le="6"} 0
prometheus_tsdb_compaction_chunk_samples_bucket{le="9"} 0
prometheus_tsdb_compaction_chunk_samples_bucket{le="13.5"} 0
prometheus_tsdb_compaction_chunk_samples_bucket{le="20.25"} 0
prometheus_tsdb_compaction_chunk_samples_bucket{le="30.375"} 0
prometheus_tsdb_compaction_chunk_samples_bucket{le="45.5625"} 0
prometheus_tsdb_compaction_chunk_samples_bucket{le="68.34375"} 0
prometheus_tsdb_compaction_chunk_samples_bucket{le="102.515625"} 0
prometheus_tsdb_compaction_chunk_samples_bucket{le="153.7734375"} 0
prometheus_tsdb_compaction_chunk_samples_bucket{le="230.66015625"} 0
prometheus_tsdb_compaction_chunk_samples_bucket{le="345.990234375"} 0
prometheus_tsdb_compaction_chunk_samples_bucket{le="+Inf"} 0
prometheus_tsdb_compaction_chunk_samples_sum 0
prometheus_tsdb_compaction_chunk_samples_count 0
# HELP prometheus_tsdb_compaction_chunk_size_bytes Final size of chunks on their first compaction
# TYPE prometheus_tsdb_compaction_chunk_size_bytes histogram
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="32"} 0
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="48"} 0
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="72"} 0
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="108"} 0
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="162"} 0
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="243"} 0
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="364.5"} 0
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="546.75"} 0
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="820.125"} 0
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="1230.1875"} 0
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="1845.28125"} 0
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="2767.921875"} 0
prometheus_tsdb_compaction_chunk_size_bytes_bucket{le="+Inf"} 0
prometheus_tsdb_compaction_chunk_size_bytes_sum 0
prometheus_tsdb_compaction_chunk_size_bytes_count 0
# HELP prometheus_tsdb_compaction_duration_seconds Duration of compaction runs
# TYPE prometheus_tsdb_compaction_duration_seconds histogram
prometheus_tsdb_compaction_duration_seconds_bucket{le="1"} 0
prometheus_tsdb_compaction_duration_seconds_bucket{le="2"} 0
prometheus_tsdb_compaction_duration_seconds_bucket{le="4"} 0
prometheus_tsdb_compaction_duration_seconds_bucket{le="8"} 0
prometheus_tsdb_compaction_duration_seconds_bucket{le="16"} 0
prometheus_tsdb_compaction_duration_seconds_bucket{le="32"} 0
prometheus_tsdb_compaction_duration_seconds_bucket{le="64"} 0
prometheus_tsdb_compaction_duration_seconds_bucket{le="128"} 0
prometheus_tsdb_compaction_duration_seconds_bucket{le="256"} 0
prometheus_tsdb_compaction_duration_seconds_bucket{le="512"} 0
prometheus_tsdb_compaction_duration_seconds_bucket{le="1024"} 0
prometheus_tsdb_compaction_duration_seconds_bucket{le="2048"} 0
prometheus_tsdb_compaction_duration_seconds_bucket{le="4096"} 0
prometheus_tsdb_compaction_duration_seconds_bucket{le="8192"} 0
prometheus_tsdb_compaction_duration_seconds_bucket{le="+Inf"} 0
prometheus_tsdb_compaction_duration_seconds_sum 0
prometheus_tsdb_compaction_duration_seconds_count 0
# HELP prometheus_tsdb_compaction_populating_block Set to 1 when a block is currently being written to the disk.
# TYPE prometheus_tsdb_compaction_populating_block gauge
prometheus_tsdb_compaction_populating_block 0
# HELP prometheus_tsdb_compactions_failed_total Total number of compactions that failed for the partition.
# TYPE prometheus_tsdb_compactions_failed_total counter
prometheus_tsdb_compactions_failed_total 0
# HELP prometheus_tsdb_compactions_skipped_total Total number of skipped compactions due to disabled auto compaction.
# TYPE prometheus_tsdb_compactions_skipped_total counter
prometheus_tsdb_compactions_skipped_total 0
# HELP prometheus_tsdb_compactions_total Total number of compactions that were executed for the partition.
# TYPE prometheus_tsdb_compactions_total counter
prometheus_tsdb_compactions_total 0
# HELP prometheus_tsdb_compactions_triggered_total Total number of triggered compactions for the partition.
# TYPE prometheus_tsdb_compactions_triggered_total counter
prometheus_tsdb_compactions_triggered_total 5
# HELP prometheus_tsdb_data_replay_duration_seconds Time taken to replay the data on disk.
# TYPE prometheus_tsdb_data_replay_duration_seconds gauge
prometheus_tsdb_data_replay_duration_seconds 6.396156678
# HELP prometheus_tsdb_exemplar_exemplars_appended_total Total number of appended exemplars.
# TYPE prometheus_tsdb_exemplar_exemplars_appended_total counter
prometheus_tsdb_exemplar_exemplars_appended_total 0
# HELP prometheus_tsdb_exemplar_exemplars_in_storage Number of exemplars currently in circular storage.
# TYPE prometheus_tsdb_exemplar_exemplars_in_storage gauge
prometheus_tsdb_exemplar_exemplars_in_storage 0
# HELP prometheus_tsdb_exemplar_last_exemplars_timestamp_seconds The timestamp of the oldest exemplar stored in circular storage. Useful to check for what timerange the current exemplar buffer limit allows. This usually means the last timestampfor all exemplars for a typical setup. This is not true though if one of the series timestamp is in future compared to rest series.
# TYPE prometheus_tsdb_exemplar_last_exemplars_timestamp_seconds gauge
prometheus_tsdb_exemplar_last_exemplars_timestamp_seconds 0
# HELP prometheus_tsdb_exemplar_max_exemplars Total number of exemplars the exemplar storage can store, resizeable.
# TYPE prometheus_tsdb_exemplar_max_exemplars gauge
prometheus_tsdb_exemplar_max_exemplars 0
# HELP prometheus_tsdb_exemplar_out_of_order_exemplars_total Total number of out of order exemplar ingestion failed attempts.
# TYPE prometheus_tsdb_exemplar_out_of_order_exemplars_total counter
prometheus_tsdb_exemplar_out_of_order_exemplars_total 0
# HELP prometheus_tsdb_exemplar_series_with_exemplars_in_storage Number of series with exemplars currently in circular storage.
# TYPE prometheus_tsdb_exemplar_series_with_exemplars_in_storage gauge
prometheus_tsdb_exemplar_series_with_exemplars_in_storage 0
# HELP prometheus_tsdb_head_active_appenders Number of currently active appender transactions
# TYPE prometheus_tsdb_head_active_appenders gauge
prometheus_tsdb_head_active_appenders 0
# HELP prometheus_tsdb_head_chunks Total number of chunks in the head block.
# TYPE prometheus_tsdb_head_chunks gauge
prometheus_tsdb_head_chunks 87495
_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 16
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0

On prometheus side, yes i think it is ok,

Screenshot 2023-06-15 at 1.19.25 AM
Screenshot 2023-06-15 at 1.19.25 AM1111Ă—149 17.6 KB

and on the 0.0.0.0:9090/metrics, it is almost the same result.

8

The endpoint URL should be the be the pump service name not 0.0.0.0 in prom. You’re scraping proms own metrics endpoint :upside_down_face: thats why the status is up.

You should also be seeing some purge logs in the pump so their might another thing missing.

Can you send me the commands you used to deploy everything so I can test locally?

9

@zaid ,
First, Is that normal that the pump do not generate himself any services? I just have the gateway-svc-tyk-oss-tyk-gateway (tyk-gateway svc).
Do i have to do it manually by exposing the deployment ? with: ( kubectl expose deployment/pump-tyk-oss-tyk-pump -n tyk)

My commands:
kubectl create ns tyk
kubectl create ns monitoring
For redis:
helm install tyk-redis bitnami/redis -n tyk
For Tyk: With the values oss, i sent you before,
helm install tyk-oss tyk-helm/tyk-oss -n tyk -f values-oss.yaml --devel

For prometheus:
helm install prometheus prometheus-community/prometheus -n monitoring

10

Hello @zaid , @Olu :hot_face:
I try again to just follow the process and expose manually the tyk-pump by doing: k expose deployment/pump-tyk-oss-tyk-pump --type=LoadBalancer -n tyk
I didnt install prometheus on this one(just install tyk-oss and configure pump, choose backend as prometheus); but i am getting this on the port 9090:

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile=“0”} 4.649e-05
go_gc_duration_seconds{quantile=“0.25”} 8.2039e-05
go_gc_duration_seconds{quantile=“0.5”} 8.9075e-05
go_gc_duration_seconds{quantile=“0.75”} 0.000116346
go_gc_duration_seconds{quantile=“1”} 0.000124375
go_gc_duration_seconds_sum 0.000458325
go_gc_duration_seconds_count 5
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 13
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version=“go1.16.15”} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 2.068888e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.2213128e+07
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.449643e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 143544
# HELP go_memstats_gc_cpu_fraction The fraction of this program’s available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 2.7429333277110426e-05
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 4.997696e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 2.068888e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 6.234112e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 4.276224e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 12116
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 6.1579264e+07
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.6617344e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.686874087462315e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 155660
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 2400
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 77248
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 114688
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 4.194304e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 582429
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 491520
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 491520
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.4269704e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 7
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.35
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 11
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 3.2227328e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.68687389475e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 7.70048e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code=“200”} 0
promhttp_metric_handler_requests_total{code=“500”} 0
promhttp_metric_handler_requests_total{code=“503”} 0

I definitely don’t not know what i am doing wrong.

11

Hello @labi so first of all, you’re correct. You will need to expose the pump service using the kubectl expose command if you setup your own Prometheus. Can you please create an issue on the repo so that it can be added natively?

I also got it the setup working. Here are the commands I ran. I am using the Tyk Operator to deploy my API because its easier than using the API but its completely up to you.

Create namespaces

kubectl create ns tyk
kubectl create ns monitoring
kubectl create ns services

Install Redis and get Redis password

helm upgrade tyk-redis bitnami/redis -n tyk --install --wait -n tyk
redisPass=$(kubectl get secret --namespace tyk tyk-redis -o jsonpath="{.data.redis-password}" | base64 -d);

Install Tyk

helm upgrade tyk-oss tyk-helm/tyk-oss -n tyk --devel \
  --install --wait \
  --set global.components.pump=true \
  --set "global.redis.addrs[0]=tyk-redis-master.tyk.svc:6379" \
  --set global.redis.pass=$redisPass \
  --set global.tyk-pump.pump.prometheusPump.host=0.0.0.0 \
  --set global.tyk-pump.pump.prometheusPumpcustomMetrics="[{\"name\": \"tyk_http_requests_total\"\,\"description\": \"Total of API requests\"\,\"metric_type\": \"counter\"\,\"labels\": [\"response_code\"\,\"api_name\"\,\"method\"\,\"api_key\"\,\"alias\"\,\"path\"]}\,{\"name\": \"tyk_http_latency\"\,\"description\": \"Latency of API requests\"\,\"metric_type\": \"histogram\"\,\"labels\": [\"type\"\,\"response_code\"\,\"api_name\"\,\"method\"\,\"api_key\"\,\"alias\"\,\"path\"]}]"

Expose the pump service:

kubectl expose deployment/pump-tyk-oss-tyk-pump -n tyk

Install Prometheus with Tyk target and extra rules:

helm upgrade prometheus prometheus-community/prometheus -n monitoring --version 20.1.0 \
  --install --wait \
  --set "prometheus-pushgateway.enabled=false" \
  --set "prometheus-node-exporter.hostRootFsMount.enabled=false" \
  --set "server.global.scrape_interval=15s" \
  --set "server.global.evaluation_interval=15s" \
  --set "server.extraFlags={web.enable-lifecycle,web.enable-admin-api}" \
  --set "serverFiles.recording_rules\.yml.groups[0].name=tyk" \
  --set "serverFiles.recording_rules\.yml.groups[0].rules[0].record=task:http_response_error_count" \
  --set "serverFiles.recording_rules\.yml.groups[0].rules[0].expr=tyk_http_requests_total{response_code=~\"5\[0-9]{2}\"}" \
  --set "serverFiles.recording_rules\.yml.groups[0].rules[1].record=task:http_response_total_count" \
  --set "serverFiles.recording_rules\.yml.groups[0].rules[1].expr=tyk_http_requests_total{response_code=~\"\[0-9]{3}\"}" \
  --set "serverFiles.recording_rules\.yml.groups[0].rules[2].record=task:http_response_error_rate" \
  --set "serverFiles.recording_rules\.yml.groups[0].rules[2].expr=sum by (job\,api_name) (rate(task:http_response_error_count\[1m]))" \
  --set "serverFiles.recording_rules\.yml.groups[1].name=slo_metrics" \
  --set "serverFiles.recording_rules\.yml.groups[1].rules[0].record=job:slo_errors_per_request:ratio_rate10m" \
  --set "serverFiles.recording_rules\.yml.groups[1].rules[0].expr=sum by (job\,api_name) (rate(task:http_response_error_count\[10m])) / sum by (job\,api_name) (rate(task:http_response_total_count\[10m]))" \
  --set "serverFiles.recording_rules\.yml.groups[1].rules[1].record=job:error_budget:remaining" \
  --set "serverFiles.recording_rules\.yml.groups[1].rules[1].expr=(1 - job:slo_errors_per_request:ratio_rate10m) * 100" \
  --set "serverFiles.prometheus\.yml.scrape_configs[0].job_name=tyk" \
  --set "serverFiles.prometheus\.yml.scrape_configs[0].metrics_path=/metrics" \
  --set "serverFiles.prometheus\.yml.scrape_configs[0].static_configs[0].targets={pump-tyk-oss-tyk-pump.tyk.svc:9090}"

Create Tyk Operator secret, install Tyk Operator and its dependency cert-manager

kubectl create secret generic tyk-operator-conf -n tyk \
  --from-literal="TYK_MODE=ce" \
  --from-literal="TYK_URL=http://gateway-svc-tyk-oss-tyk-gateway.tyk.svc:443" \
  --from-literal="TYK_AUTH=CHANGEME" \
  --from-literal="TYK_ORG=tyk"

helm upgrade cert-manager jetstack/cert-manager --version v1.10.1 \
    --install --wait \
    --set "installCRDs=true" \
    --set "prometheus.enabled=false" \
    --namespace tyk

helm upgrade tyk-opertor tyk-helm/tyk-operator \
  --install --wait  \
  --namespace tyk

Create API and services in services namespace:

echo '
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin-svc
  labels:
    app: httpbin
spec:
  ports:
    - name: http
      port: 8000
      targetPort: 8080
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      containers:
        - image: zalbiraw/go-httpbin
          imagePullPolicy: IfNotPresent
          name: httpbin
          ports:
            - containerPort: 8080
          securityContext:
            runAsNonRoot: true
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            seccompProfile:
              type: RuntimeDefault
---
apiVersion: tyk.tyk.io/v1alpha1
kind: ApiDefinition
metadata:
  name: httpbin-keyless
spec:
  name: httpbin-keyless
  use_keyless: true
  protocol: http
  active: true
  proxy:
    target_url: http://httpbin-svc.services.svc:8000
    listen_path: /httpbin-keyless
    strip_listen_path: true
  enable_detailed_recording: true
' | kubectl apply -f - -n services

Port-forward and test:

kubectl port-forward svc/gateway-svc-tyk-oss-tyk-gateway -n tyk 8080:443 &
kubectl port-forward svc/prometheus-server 8090:80 -n monitoring &

curl localhost:8080/httpbin-keyless/get

You can see the pump logs here:
Screenshot 2023-06-16 at 1.32.28 PM

You can see prom here:

Screenshot 2023-06-16 at 1.26.28 PM
Screenshot 2023-06-16 at 1.26.28 PM1079Ă—551 30.7 KB

12

Working now ! :slightly_smiling_face:, Thanks. I will try to check why it was not with the file.
Thanks a lot, i will open an issue for the tyk-pump services.

1 Like
13

Hello @zaid ,
I have some others interrogation, i will put them here.
I do not have the metrics on the grafana. Also i can not see the list of API i have anc choose the one i wanted. Is that normal ?

Screenshot 2023-06-19 at 4.09.17 PM
Screenshot 2023-06-19 at 4.09.17 PM2866Ă—1494 184 KB

How can i have more metrics, also filter them by api_name ?
The one i have currently and are not loading in grafana :

Screenshot 2023-06-19 at 4.11.18 PM
Screenshot 2023-06-19 at 4.11.18 PM1920Ă—898 211 KB

But,
I Checked the official documentation, the demo with Docker, there are more. And the result is:

Screenshot 2023-06-19 at 4.14.15 PM
Screenshot 2023-06-19 at 4.14.15 PM1920Ă—1161 335 KB

14

Sorry not the most familiar with Grafana. But I do have a fully worked out example here. I hope it helps.

1 Like