Tyk elastic pump settings to allow header, raw_request and raw_response without encoding

Support Administration and Operation
, ,
1

Hi Support,

I successfully created a tyk pump to send the API analytics stats to elasticsearch then to kibana.

I wanted to see more information about the API, I wanted include the header, raw_reqeust without encoding, raw_response without encoding. In the header, we have the client-id that we want the separate stats, in the request and response, there are a geo location we need to aggregate.

I tried this settings to see this but I don’t seem them in kibana. Can you point me what went wrong in the elasticsearch pump settings?

    "elasticsearch": {
      "type": "elasticsearch",
      "meta": {
        "index_name": "tyk_analytics",
        "elasticsearch_url": "http://server694:9200",
        "enable_sniffing": false,
        "document_type": "tyk_analytics",
        "rolling_index": false,
        "extended_stats": true,
        "raw_request_decoded": true,
        "raw_response_decoded": true,
        "version": "7",
        "enable_http_header_capture": true,
        "header_filters": ["Client-Id"],        
        "bulk_config": {
          "workers": 2,
          "flush_interval": 60
        }
      }
    }

I got what I want in the csv pump though:

    "csv": {
      "type": "csv",
      "omit_detailed_recording": false,
      "raw_request_decoded": true,
      "raw_response_decoded": true,
      "meta": {
        "csv_dir": "/mnt",
        "include_detailed_recording": true
      }
    },

And this is tyk.conf for GW

  "enable_analytics": true,
  "analytics_config": {
    "enable_detailed_recording": true,
    "type": "",
    "ignored_ips": [],
    "forward_headers": ["Client-Id"]
  },
2

Hi @mike3dk

Welcome to the community :partying_face:

Please use decode_base64 to decode records. Also, some fields don’t appear to be valid and I have removed them below.

"elasticsearch": {
      "type": "elasticsearch",
      "meta": {
        "index_name": "tyk_analytics",
        "elasticsearch_url": "http://server694:9200",
        "enable_sniffing": false,
        "document_type": "tyk_analytics",
        "rolling_index": false,
        "extended_stats": true,
        "decode_base64 ": true,
        "version": "7",    
        "bulk_config": {
          "workers": 2,
          "flush_interval": 60
        }
      }
    }

Capturing headers this way isn’t quite supported. You should be able to achieve this using Custom Tags. Please let us know if this is okay for you.

Best regards,