Traffic analysis does not capture all requests

Support
, ,
1

Hello,
I am working on some proofs of concept and I have some doubts.
I have Tyk open source gateway with PUMP and OpenSearch (Elasticsearch) working.
I want to save in OpenSearch all the requests that TYK receives. How can I do it?
I have found that they are not captured in the following cases:

  • Case 1: Requests to “mocks” of TYK.
  • Case 2: Requests whose “listen_path” does not exist in the APIs.
  • Case 3: Requests to get the token (OAuth2 client_credentials). URL: “/oauth/token”

Maybe there are more cases.

Case 1: Example:

{
    "name": "test404",
    "api_id": "test404",
    "org_id": "TEST",
    "version_data": {
        "not_versioned": true,
        "versions": {
            "Default": {
                "name": "Default",
                "use_extended_paths": true,
                "extended_paths": {
                    "white_list": [
                        {
                            "path": "(.*)",
                            "method_actions": {
                                "GET": {
                                    "action": "reply",
                                    "code": 404,
                                    "data": "{\"status\":\"404\"}",
                                    "headers": {
                                        "Content-Type": "application/json"
                                    }
                                }
                            }
                        }
                    ]
                }
            }
        }
    },
    "enable_detailed_recording": true,
    "use_keyless": true,
    "proxy": {
        "listen_path": "/"
    }
}

Case 2: I can create an API with “listen_path”=“/” but I don’t know how to customize the output and also capture the request without having a real API that listens to everything.
Is there any better option?

Case 3: I need to capture successful and failed token requests but I can’t find any way.

I have also tried it with “looping” without getting what I want, and I have found some strange behavior.
If I have 3 APIs in TYK, A->B->C(TARGET_URL)
The registered “api_id” is C.
But if B is secured by OAuth2 client_credentials, the “api_id” registered is B.
What is the logic in traffic analysis when looping is used?

Thank you.

2

@soycreal Welcome to the community.

I will answer some of your questions in theory.

Case 1 : Requests to “mocks” of TYK

Yes, requests to mocks are not recorded by the analytics and don’t get pumped or purged to any external data sink. However, I think you could achieve showing mocks in analytics through building a custom gateway. We have some code within the gateway you could modify to get the records showing in analytics.

There is a RedisAnalyticsHandler within the analytics.go class that has a mockEnabled property. I haven’t tried it but my guess is that if this is true, then analytics for mocks would be recorded

In theory, if you were to add a new property (something like mockEnabled) in the AnalyticsConfig struct, then you could reference it in the same code as show above to record a mock response hit.

3

Can you explain more?

  • What do you mean by customize
  • Capture the request without having a real API

Putting a non-existence upstream should still record a 404 in the analytics.

I could give an answer based on my assumption. You could give each API their expected listen path and then use the root to record listen paths for all entries. You can mark each desired API as an internal API, and forward requests which match the first part of the path the desired internal API.

{
  "api_id": "urbsmktda0l6qbi3g4o064tde44vgjan",
  "internal": true
}

The URL rewrite and looping feature would come in handy here. For example, I have designed the root api to forward all requests starting with http_bin_api to the desired http bin API.

"extended_paths": {
          "url_rewrites": [
						{
							"path": "^/(http_bin_api){1}/?",
							"method": "GET",
							"match_pattern": "^/(http_bin_api){1}/?",
							"rewrite_to": "tyk://iashh9tmfek86wyegfcfadhis7ncq3m3",
							"triggers": []
						}
					]
				}

You just need to ensure that you have not overridden the liveliness endpoint (/hello)

4

The oauth/tokens are part of the REST API endpoints of the gateway and are not recorded in the analytics.

As for the logic when looping it used, it is that way by default. You could use full http scheme (instead of the tyk;// scheme) and call API B to ensure the api_id of A is still registered in the analytics call. This is at a cost of an extra network request.

5

Thank you very much Olu for your answers, they have been very useful to me.

About case 2.
I would like to return a 404 with a custom body, and record the request in analytics.

As you indicate in case 1, the Mocks are not recorded.

In my tests.

If I don’t fill the “target_url” field, I get:

Status code 500
{
     "error": "There was a problem proxying the request"
}

If I put a non-existent url in target_url, I get:

Status code 500
{
     "error": "Upstream host lookup failed"
}

or

{
     "error": "There was a problem proxying the request"
}

If target_url=tyk://noexist, I get:

Status code 500
{
     "error": "Couldn\'t detect target"
}

Using looping and URL rewrite, I get similar results.

With what you have told me, I think I should use an external “mock” to get what I want. Using Error Templates is also not a solution.
Thank you

6

Ah! This is true. What I meant to say is that a 404 would be the result if the base path of the upstream is valid but the resource path of the upstream is invalid or non-existent.

All the 500 errors as mentioned in the previous posts are valid for a completely non-existent upstream. The same would also work when using looping.

7

Do you mean using an upstream mock here?

8

Yes, this is how it behaves if there is a server listening, like a Tomcat. If I have a Tomcat with a working API, but the Tomcat or the server where the Tomcat is running crashes, Tyk will return a 500, not a 404.
It is not what I was looking for for my use.

9

I’m not sure. I mean, I would have to have a Tomcat or similar, that returns the 404 response with the body that I want, and create an API in Tyk with listen_path = ‘/’ that sends the requests to Tomcat.
This way, I can record the request in the analytics and I can return the response I want.

So far I haven’t found any other way to do it.