Hello, I am trying to setup tyk with an external syslog.
I install the syslog on a VM.
Then i wanted to connect the tyk pump to the syslog outside the kubernetes server. (Tyk is on kubernetes)
I have this as configuration but i am not sure that it should be all.
Can you help with the full configuration ?
also notice that by default tyk is trying to use mongo as backend, i wanted to use syslog.
@labi Thanks for sharing a screenshot of the pump logs.
We can observe Redis connection error on port 6379. Pump needs to connect to Redis to purge analytics to your desired data sink. So you might want to verify there are no connection issues from your k8 server.
Additionally, can you share your pump configuration? You can strip out sanative info from it.
The pump conf is:
Hello @Olu
tyk-headless:
# These are your Tyk stack secrets will directly map to the following Tyk stack
# configuration:
secrets:
# APISecret: Password stored in the secret
useSecretName: "tyk-headless"
redis:
# The addrs value will allow you to set your Redis addresses.
addrs:
- tyk-redis-master.tyk.svc.cluster.local:6379
# pass: Password stored in the secret
gateway:
# The hostname to bind the Gateway to.
# hostName: i.rubyx.xyz
# When true, sets the gateway protocol to HTTPS.
# tls: false
kind: Deployment
replicaCount: 1
containerPort: 8080
image:
# repository: docker.tyk.io/tyk-gateway/tyk-gateway
repository: tykio/tyk-gateway
tag: v5.1
pullPolicy: IfNotPresent
extraEnvs:
- name: TYK_GW_ENABLEHASHEDKEYSLISTING
value: "true"
extraVolumes:
- name: tyk-app-storage
persistentVolumeClaim:
claimName: tyk-app-claim
- name: tyk-policies-storage
persistentVolumeClaim:
claimName: tyk-policies-claim
- name: tyk-middleware-storage
persistentVolumeClaim:
claimName: tyk-middleware-claim
extraVolumeMounts:
- name: tyk-app-storage
mountPath: /mnt/tyk-gateway/apps
- name: tyk-policies-storage
mountPath: /mnt/tyk-gateway/policies
- name: tyk-middleware-storage
mountPath: /mnt/tyk-gateway/middleware
service:
type: NodePort
port: 8080
externalTrafficPolicy: Local
annotations: {}
# Creates an ingress object in k8s. Will require an ingress-controller and
# annotation to that ingress controller.
ingress:
enabled: false
# The pumps backend for Tyk is postgres
backend: postgres
# Postgres connection string parameters.
postgres:
# ---------- ALL POSTRGES DATA IS STORED IN THE SECRET ----------
# host: 10.81.0.31
# port: 5432
# user: tyk_pump
# password:
# database: tyk_analytics
# sslmode:
# If pump is enabled the Gateway will create and collect analytics data to send
# to a data store of your choice. These can be set up in the pump config. The
# possible pump configs can be found here:
# https://github.com/TykTechnologies/tyk-pump#configuration
pump:
# Determines whether or not the pump component should be installed.
enabled: true
replicaCount: 1
image:
repository: docker.tyk.io/tyk-pump/tyk-pump
tag: v1.7.0
pullPolicy: IfNotPresent
annotations: {}
resources:
# We usually recommend not to specify default resources and to leave this
# as a conscious choice for the user. This also increases chances charts
# run on environments with little resources, such as Minikube. If you do
# want to specify resources, uncomment the following lines, adjust them
# as necessary, and remove the curly braces after 'resources:'.
limits:
cpu: 500m
memory: 256Mi
requests:
cpu: 500m
memory: 256Mi
# extraEnvs:
# - name: TYK_PMP_PUMPS_SYSLOG_TYPE
# value: syslog
# - name: TYK_PMP_PUMPS_SYSLOG_META_NETWORKADDR
# value: "34.123.247.207:8888"
# - name: TYK_PMP_PUMPS_SYSLOG_META_TRANSPORT
# value: "udp"
# - name: TYK_PMP_PUMPS_SYSLOG_LOG_LEVEL
# value: "6"
# - name: TYK_PMP_PUMPS_SYSLOG_META_TAG
# value: "syslog-pump"
rbac: true
where can i define the tyk.conf value, or it is not possible anymore.
But yes, how can i configure the syslog and also add add new imput for my logs, like api_name, api_keys…
It should be somewhere in the Config Maps. I am having some difficulty with my Kubernetes environment at the moment, so I can’t verify. But you actually don’t need the to locate the config file. You should be able to override config value by specifying environment variables. I was only asking for your settings to review cause of the Redis connection error
I see you’ve already gotten started with the commented out syslog environment vars.
ok ok. i found the config map.
Can you tell me how i can enable some content?
This rawresponse is empty everywhere,
Can you tell me how i can get more column ?
thanks
Seems like you got it working. Good job.
A common error of not seeing the detailed request and response is that detailed recording has been disabled. There are three scopes where this can be enabled or disabled:
You can verify which scope your environment has enabled by looking for the field enable_detailed_recording in the key definition, API definition or gateway configuration.
Note
Please be aware that enabling detailed analytics is an expensive process. We only recommend it during testing or debugging. Also, don’t enable detailed analytics for all the scopes. Each scope has different levels of priorities as shown above.
I don’t think this is possible. However, you could overload the tags field of the analytics record using either HTTP Headers or Analytics Plugin
Did any of my responses address this or is this something completely different?
Yes, it helped me a lot.
I notice that the raw_response is encrypted.
It do not really help to extract the log.
How Can i disable the encryption ?
It’s great to hear that
It’s a simple base 64 encoding. You can decode to see the content
Can you retype this. I am not sure I understand
Yes you can, by enabling the decode properties. Hope this helps
Hello @olu, i was trying to decode via tyk and it is not working , can you confirm ?
I used this:
extraEnvs:
- name: TYK_PMP_PUMPS_SYSLOG_TYPE
value: syslog
- name: TYK_PMP_PUMPS_SYSLOG_META_NETWORKADDR
value: syslog-fluent-bit.syslog.svc.cluster.local:5140
- name: TYK_PMP_PUMPS_SYSLOG_META_TRANSPORT
value: “tcp”
- name: TYK_PMP_PUMPS_SYSLOG_LOG_LEVEL
value: “6”
- name: TYK_PMP_PUMPS_SYSLOG_META_TAG
value: “tyk-pump”
> - name: TYK_PMP_DECODERAWREQUEST
> value: "true"
> - name: TYK_PMP_DECODERAWRESPONSE
> value: "true"
I also wanted to ask, i am using syslog to load the data from tyk but the format i am getting is not ok for me. I saw that syslog use some parser to parse the log.
I needed to get the original log so that i can write my own parser. Can you guide me on where i can find that ?
It seems that the decode properties (TYK_PMP_DECODERAWREQUEST and TYK_PMP_DECODERAWRESPONSE) are features that are availoable starting Pump version 1.8. I saw from your screenshot above, it looks like you are using Pump version 1.7 that’s why these variables will not work.
I saw that syslog use some parser to parse the log.
Can you show us where you saw this? You might be referring to this documentation where it mentioned about a parser. So when you are only using FluentD, you need to have a FluentD Parser based on the OS you are using so that FluentD can correctly read the logs.
You mentioned that you are not okay with the format you are getting. Can you show it to us?
As best effort, it seems you are using fluent-bit which has it’s default parser. Please note that this is out of our support scope though.
Hello @Olu ,@Page yes, you are right, i thought fluentbit will work as well.
I and to finally use Elasticsearch but i have an issue.You can see the image below.
my configuration is this:
- name: TYK_PMP_PUMPS_ELASTICSEARCH_TYPE
value: “elasticsearch”
- name: TYK_PMP_PUMPS_ELASTICSEARCH_META_INDEXNAME
value: “tyk_analytics”
- name: TYK_PMP_PUMPS_ELASTICSEARCH_META_ELASTICSEARCHURL
value: “http://elasticsearch-master.elk.svc.cluster.local:9200”
- name: TYK_PMP_PUMPS_ELASTICSEARCH_META_ENABLESNIFFING
value: “false”
- name: TYK_PMP_PUMPS_ELASTICSEARCH_META_DOCUMENTTYPE
value: “tyk_analytics”
- name: TYK_PMP_PUMPS_ELASTICSEARCH_META_ROLLINGINDEX
value: “false”
- name: TYK_PMP_PUMPS_ELASTICSEARCH_META_EXTENDEDSTATISTICS
value: “false”
- name: TYK_PMP_PUMPS_ELASTICSEARCH_META_VERSION
value: “5”
- name: TYK_PMP_PUMPS_ELASTICSEARCH_META_BULKCONFIG_WORKERS
value: “2”
- name: TYK_PMP_PUMPS_ELASTICSEARCH_META_BULKCONFIG_FLUSHINTERVAL
value: “60”
- name: TYK_PMP_DECODERAWRESPONSE
value: “true”
I am using tyk-gateway version: v5.2.1
tyk-pump: v1.8.3
elastic: 8.5.1
i was thinking maybe it is not working because in the docs pump support version 2-7 elasticsearch version. I am not able to instal old version of elk.
@labi
Yes, you are correct, we only support Elastic search versions 2.0 - 7.x.
I am also getting that issue when installing elasticsearch version 6. I tried and successfully installed version 7.17.3. Can you try and see if the Pump issue that you are seeing when using version 8 exists if you used version 7.x?
Hello @Page.
Elastic-search if finally ok,i am trying to work with fluentd now, but i am not able to print the log.
I show you my how i use it. i use helm chart for tyk and fkuentd.
tyk:
## Default values for tyk-headless chart.
## This is a YAML-formatted file.
## Declare variables to be passed into your templates.
## See Tyk Helm documentation for installation details:
## https://tyk.io/docs/tyk-oss/ce-helm-chart/
## Registry for all Tyk images - https://hub.docker.com/u/tykio
# Chart name override. Truncates to 63 characters.
# Default value: tyk-headless.name
nameOverride: "tyk"
# App name override. Truncates to 63 characters.
# Default value: tyk-headless.fullname
fullnameOverride: "tyk-h"
# These are your Tyk stack secrets will directly map to the following Tyk stack
# configuration:
secrets:
# tyk.conf node_secret
# tyk.conf secret
APISecret: CHANGEME
# If you don't want to store plaintext secrets in the Helm value file and would
# rather provide the k8s Secret externally please populate the value below
useSecretName: ""
redis:
addrs:
- tyk-redis-master.tyk.svc.cluster.local:6379
pass: "zk8hzkdx3r"
# Redis password
# If you're using Bitnami Redis chart (e.g. bitnami/redis) please input
# your password in the field below
# pass: ""
# Enables sentinel connection mode for Redis. If enabled, provide both
# mandatory values for sentinelPass and masterName.
# enableSentinel: false
# Redis sentinel password, only required while enableSentinel is true.
# For bitnami/redis the same password as Redis above
# sentinelPass: ""
# Redis sentinel master name, only required while enableSentinel is true.
# For bitnami/redis typically redis-master
# masterName: "redis-master"
# By default the database index is 0. Setting the database index is not
# supported with redis cluster. As such, if you have enableCluster: true,
# then this value should be omitted or explicitly set to 0.
storage:
database: 0
# Choose the pumps backend for Tyk. [ "", "mongo", "postgres" ]
backend: "postgres"
mongo:
# The mongoURL value will allow you to set your MongoDB address.
# Default value: mongodb://mongo.{{ .Release.Namespace }}.svc.cluster.local:27017/tyk_analytics
# mongoURL: mongodb://mongo.tyk.svc.cluster.local:27017/tyk_analytics
# If your MongoDB has a password you can add the username and password to the url
# mongoURL: mongodb://root:[email protected]:27017/tyk_analytics?authSource=admin
# Enables SSL for MongoDB connection. MongoDB instance will have to support that.
# Default value: false
useSSL: false
# Postgres connection string parameters.
postgres:
host: tyk-postgres-postgresql.tyk.svc.cluster.local
port: 5432
user: postgres
password: "YR7bUylTuD"
database: tyk_analytics
sslmode:
gateway:
# The hostname to bind the Gateway to.
hostName: tyk-gw.local
# When true, sets the gateway protocol to HTTPS.
tls: false
kind: Deployment
replicaCount: 1
containerPort: 8080
image:
repository: docker.tyk.io/tyk-gateway/tyk-gateway
tag: v5.2.1
pullPolicy: IfNotPresent
service:
type: NodePort
port: 8080
externalTrafficPolicy: Local
annotations: {}
# control:
# enabled: false
# containerPort: 9696
# port: 9696
# type: ClusterIP
# annotations: {}
# Creates an ingress object in k8s. Will require an ingress-controller and
# annotation to that ingress controller.
ingress:
enabled: false
# specify your ingress controller class name below
className: ""
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
hosts:
- host: chart-example.local
paths:
- path: /
pathType: ImplementationSpecific
tls: []
# - secretName: chart-example-tls
# hosts:
# - chart-example.local
resources: {}
# We usually recommend not to specify default resources and to leave this
# as a conscious choice for the user. This also increases chances charts
# run on environments with little resources, such as Minikube. If you do
# want to specify resources, uncomment the following lines, adjust them
# as necessary, and remove the curly braces after 'resources:'.
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
securityContext:
runAsUser: 1000
fsGroup: 2000
nodeSelector: {}
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
affinity: {}
extraEnvs:
- name: TYK_GW_ENABLEHASHEDKEYSLISTING
value: "true"
- name: TYK_GW_ANALYTICSCONFIG_ENABLEDETAILEDRECORDING
value: "true"
## extraVolumes A list of volumes to be added to the pod
## extraVolumes:
## - name: ca-certs
## secret:
## defaultMode: 420
## secretName: ca-certs
extraVolumes: []
## extraVolumeMounts A list of volume mounts to be added to the pod
## extraVolumeMounts:
## - name: ca-certs
## mountPath: /etc/ssl/certs/ca-certs.crt
## readOnly: true
extraVolumeMounts: []
mounts: []
# If pump is enabled the Gateway will create and collect analytics data to send
# to a data store of your choice. These can be set up in the pump config. The
# possible pump configs can be found here:
# https://github.com/TykTechnologies/tyk-pump#configuration
pump:
# Determines whether or not the pump component should be installed.
enabled: true
replicaCount: 1
image:
repository: docker.tyk.io/tyk-pump/tyk-pump
tag: v1.8.3
pullPolicy: IfNotPresent
annotations: {}
resources: {}
# We usually recommend not to specify default resources and to leave this
# as a conscious choice for the user. This also increases chances charts
# run on environments with little resources, such as Minikube. If you do
# want to specify resources, uncomment the following lines, adjust them
# as necessary, and remove the curly braces after 'resources:'.
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
securityContext:
runAsUser: 1000
fsGroup: 2000
nodeSelector: {}
tolerations: []
affinity: {}
extraEnvs:
- name: TYK_PMP_PUMPS_SYSLOG_TYPE
value: syslog
- name: TYK_PMP_PUMPS_SYSLOG_META_NETWORKADDR
value: my-fluentd-aggregator.fluentd.svc.cluster.local:5410
- name: TYK_PMP_PUMPS_SYSLOG_META_TRANSPORT
value: "udp"
- name: TYK_PMP_PUMPS_SYSLOG_LOG_LEVEL
value: "6"
- name: TYK_PMP_PUMPS_SYSLOG_META_TAG
value: "tyk-pump"
- name: TYK_PMP_DECODERAWREQUEST
value: "true"
- name: TYK_PMP_DECODERAWRESPONSE
value: "true"
extraVolumes: []
## extraVolumeMounts A list of volume mounts to be added to the pod
## extraVolumeMounts:
## - name: ca-certs
## mountPath: /etc/ssl/certs/ca-certs.crt
## readOnly: true
extraVolumeMounts: []
mounts: []
rbac: true
and for fluentd:
helm install -f ./values-fluentd.yaml my-fluentd bitnami/fluentd --version 5.9.6 -n fluentd
Can you please help me with the configuration values file of fluentd and tell me if the side tyk-pump configuration is ok ?
For fluentd i dont know if i need to use the aggregator or the forwarder, it is confusing and i test to use both but the are not working…
Thanks in advance !
Well, that looks okay, with the exception of the log_level. I think it should be LOGLEVEL. I haven’t tested with Kubernetes but my previous tests with docker were similar to that.
I do recall having some issues connecting to FluentD directly when I used a Syslog input. So I used a Syslog listening service like
You may just have better luck with defining a UDP input plugin. Here is a quick config to listen and spit out the analytics to stdout
# Global service wide config
<system>
log_level info
</system>
# Syslog UDP input from Tyk Pump
<source>
@type udp
@label @syslog.udp
tag pump.udp.syslog # required
<parse>
@type none
</parse>
port 5160
bind 0.0.0.0
tag syslog.udp
</source>
# StdOut output log to FluentD
<label @syslog.udp>
<match **>
@type stdout
</match>
</label>
It appears you may be trying to build an EFK observability stack. I would encourage you to visit the docs or the fluentd community for further assistance