Url_rewriting with strip_listen_path and prometheus

[dszortyka]

Hello Folks,

Is there anyone using Prometheus with Tyk Gateway?
I can’t make it work and I see differences when I use strip_list_path true or false, which are not clear is they are related or not.

when “strip_listen_path”: true, here’s the logs. I cannot find a way to match the proxy path because it understands “/prometheus” as the api endpoint, but not the path.

This part of the log that I’m talking about:
time=“Feb 24 16:43:54” level=debug msg=“Stripping: /prometheus/”
time=“Feb 24 16:43:54” level=debug msg="Upstream Path is: "

When the strip_listen_path is true, I cannot pick up what is after the host because it’s cleaned up by the logic. In that way, looks like API Endpoint and “path” are two different things.

I need to be access to access tyk_host/prometheus and make sure it will be forwarded to the target host:9090/ (which will then be redirected by prometheus to :9090/graph)

time="Feb 24 16:43:54" level=debug msg=Finished api_id=9 api_name=Prometheus code=200 key="****OCJ9" mw=URLRewriteMiddleware ns=15265 org_id=1 origin=11.240.20.15 path="/prometheus/"
time="Feb 24 16:43:54" level=debug msg="Started proxy"
time="Feb 24 16:43:54" level=debug msg="Stripping: /prometheus/"
time="Feb 24 16:43:54" level=debug msg="Upstream Path is: "
time="Feb 24 16:43:54" level=debug msg=Started api_id=9 api_name=Prometheus mw=ReverseProxy org_id=1 ts=1677257034568844370
time="Feb 24 16:43:54" level=debug msg="Upstream request URL: " api_id=9 api_name=Prometheus mw=ReverseProxy org_id=1
time="Feb 24 16:43:54" level=debug msg="Outbound request URL: http://monitoring-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090" api_id=9 api_name=Prometheus mw=ReverseProxy org_id=1
time="Feb 24 16:43:54" level=debug msg="Creating new transport" api_id=9 api_name=Prometheus mw=ReverseProxy org_id=1
time="Feb 24 16:43:54" level=debug msg="Out request url: http://monitoring-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090" api_id=9 api_name=Prometheus mw=ReverseProxy org_id=1
time="Feb 24 16:43:54" level=debug msg=Finished api_id=9 api_name=Prometheus mw=ReverseProxy ns=54447322 org_id=1
time="Feb 24 16:43:54" level=debug msg="Upstream request took (ms): 54.487376"
time="Feb 24 16:43:54" level=debug msg="Done proxy"

When “strip_listen_path”: false, then I can match the regex and the subpath, but it always append the /prometheus in the target proxy server.

time="Feb 24 16:48:00" level=debug msg="Started proxy"
time="Feb 24 16:48:00" level=debug msg=Started api_id=9 api_name=Prometheus mw=ReverseProxy org_id=1 ts=1677257280569841655
time="Feb 24 16:48:00" level=debug msg="Upstream request URL: /prometheus/" api_id=9 api_name=Prometheus mw=ReverseProxy org_id=1
time="Feb 24 16:48:00" level=debug msg="Outbound request URL: http://monitoring-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090/prometheus/" api_id=9 api_name=Prometheus mw=ReverseProxy org_id=1
time="Feb 24 16:48:00" level=debug msg="Creating new transport" api_id=9 api_name=Prometheus mw=ReverseProxy org_id=1
time="Feb 24 16:48:00" level=debug msg="Out request url: http://monitoring-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090/prometheus/" api_id=9 api_name=Prometheus mw=ReverseProxy org_id=1
time="Feb 24 16:48:00" level=debug msg=Finished api_id=9 api_name=Prometheus mw=ReverseProxy ns=46466011 org_id=1
time="Feb 24 16:48:00" level=debug msg="Upstream request took (ms): 46.599425"

Prometheus API Definition:

{
      "name": "Prometheus",
      "slug": "prometheus",
      "api_id": "9",
      "org_id": "1",
      "use_basic_auth": true,
      "use_keyless": false,
      "auth": {
        "auth_header_name": "Authorization"
      },
      "definition": {
        "location": "header",
        "key": "x-api-version"
      },
      "version_data": {
        "not_versioned": true,
        "versions": {
          "Default": {
            "name": "v1",
            "use_extended_paths": true,
            "extended_paths": {
              "url_rewrites": [
                {
                  "path": "/graph",
                  "method": "GET",
                  "match_pattern": "(.*)",
                  "rewrite_to": "/prometheus/graph"
                }
              ]
            },
            "global_headers": {},
            "global_headers_remove": [],
            "global_size_limit": 0,
            "override_target": ""
          }
        }
      },
      "proxy": {
        "listen_path": "/prometheus/",
        "target_url": "http://monitoring-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090/",
        "strip_listen_path": true,
        "disable_strip_slash": false
      },
      "active": true
    }

Thanks for any help!!
-Daniel

[dszortyka]

Another update.

This is the closer I can get when using the following setup, however, looks like it works for /prometheus, but right after that, files from /static are loaded and then it fails.

"extended_paths": {
  "url_rewrites": [
    {
      "path": "",
      "method": "GET",
      "match_pattern": "",
      "rewrite_to": "/graph"
    }
  ]
},
{...}
"proxy": {
        "listen_path": "/prometheus/",
        "target_url": "http://monitoring-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090/",
        "strip_listen_path": false,
        "disable_strip_slash": false
},

time="Feb 24 16:58:21" level=debug msg=Started api_id=9 api_name=Prometheus key="****OCJ9" mw=URLRewriteMiddleware org_id=1 origin=11.240.20.15 path="/prometheus/static/js/main.4b3d0916.js" ts=1677257901336041195
time="Feb 24 16:58:21" level=debug msg="Rewriter active"
time="Feb 24 16:58:21" level=debug msg="/prometheus/static/js/main.4b3d0916.js"
time="Feb 24 16:58:21" level=debug msg="Inbound path: /prometheus/static/js/main.4b3d0916.js"
time="Feb 24 16:58:21" level=debug msg="Rewriter checking matches, len is: 39"
time="Feb 24 16:58:21" level=debug msg="[[] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] []]"
time="Feb 24 16:58:21" level=debug msg="[]"
time="Feb 24 16:58:21" level=debug msg="URL Re-written from: /prometheus/static/js/main.4b3d0916.js"
time="Feb 24 16:58:21" level=debug msg="URL Re-written to: /graph"
time="Feb 24 16:58:21" level=debug msg=Finished api_id=9 api_name=Prometheus code=200 key="****OCJ9" mw=URLRewriteMiddleware ns=3160903 org_id=1 origin=11.240.20.15 path="/prometheus/static/js/main.4b3d0916.js"
time="Feb 24 16:58:21" level=debug msg="Started proxy"
time="Feb 24 16:58:21" level=debug msg=Started api_id=9 api_name=Prometheus mw=ReverseProxy org_id=1 ts=1677257901339271749
time="Feb 24 16:58:21" level=debug msg="Upstream request URL: /graph" api_id=9 api_name=Prometheus mw=ReverseProxy org_id=1
time="Feb 24 16:58:21" level=debug msg="Outbound request URL: http://monitoring-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090/graph" api_id=9 api_name=Prometheus mw=ReverseProxy org_id=1
time="Feb 24 16:58:21" level=debug msg=Finished api_id=9 api_name=Prometheus mw=ReverseProxy ns=4246379 org_id=1
time="Feb 24 16:58:21" level=debug msg="Upstream request took (ms): 4.268931"
time="Feb 24 16:58:21" level=debug msg="Done proxy"
time="Feb 24 16:58:21" level=debug msg=Finished api_id=9 api_name=Prometheus mw=ReverseProxy ns=2567271 org_id=1
time="Feb 24 16:58:21" level=debug msg="Upstream request took (ms): 2.597213"
time="Feb 24 16:58:21" level=debug msg="Done proxy"

[Olu]

@dszortyka

I need to be access to access tyk_host/prometheus and make sure it will be forwarded to the target host:9090/ (which will then be redirected by prometheus to :9090/graph)

What are you using the Tyk gateway to do? I see you mention it above but is it for consuming APIs or for loading a page on a browser? If it is the later then I am sure you can do the same with a redirect. I have dropped a snippet below

@dszortyka

I cannot find a way to match the proxy path because it understands “/prometheus” as the api endpoint, but not the path.

Do you mean between the listen_path and the subpath?

@dszortyka

In that way, looks like API Endpoint and “path” are two different things.

If you mean the listen_path and subpath, then yes, they are both different things. Have you gone through our documentation about the proxy settings in the API definition?

From the logs, I don’t observe anything wrong. It looks to be doing what is specified in the configuration.

Snippet

{
	"api_id": "d6cdee2d7af948c17190580eb8b74d99",
	"definition": {
		"location": "header",
		"key": "x-api-version",
		"strip_path": false
	},
	"id": "63f9063c2e6d3a0001877c51",
	"name": "Prometheus",
	"proxy": {
		"preserve_host_header": false,
		"listen_path": "/prometheus/",
		"target_url": "http://host.docker.internal:9000",
		"disable_strip_slash": true,
		"strip_listen_path": true
	},
	"use_keyless": true,
	"use_standard_auth": false,
	"version_data": {
		"not_versioned": true,
		"default_version": "",
		"versions": {
			"Default": {
				"name": "Default",
				"use_extended_paths": true,
				"extended_paths": {
					"ignored": [
						{
							"path": "/graph/?",
							"ignore_case": false,
							"method_actions": {
								"GET": {
									"action": "reply",
									"code": 301,
									"data": "Redirecting...",
									"headers": {
										"Location": "http://localhost:9000/"
									}
								}
							}
						}
					]
				}
			}
		}
	}
}

[dszortyka]

Hey Olu, as usual, thanks for help!

I can make it work when I manually type <TYK_FQDN>/prometheus/graph, then, everything after /prometheus is added to target proxy like /prometheus/graph . I added the logs below in this message.

The problem happens when I hit /prometheus/ only, without any subpath. Then, the connection is proxied to <TARGET_HOST:9090>/prometheus.

According to Prometheus documentation, we expose it in /prometheus when using behind a reverse proxy.

I believe the solution should be something like this:

• Regex to validate that ONLY “/prometheus” or “/prometheus” is in the Path and then redirect to <target_host:9090>/prometheus/graph (and stop applying any other rule)
• If the URL has any suffix after /prometheus/ , then it should rewrite the URL to /prometheus/$1 (This is working).

# https://github.com/prometheus/prometheus/issues/4925
# https://blog.cubieserver.de/2020/configure-prometheus-on-a-sub-path-behind-reverse-proxy/
routePrefix: /prometheus/
externalUrl: "http://localhost:9090/prometheus/"

If I add the ignore parameters as per your suggestion, in most of the cases I got a too many redirects error.

prometheus.json: |
    {
      "name": "Prometheus",
      "slug": "prometheus",
      "api_id": "9",
      "org_id": "1",
      "use_basic_auth": true,
      "use_keyless": false,
      "auth": {
        "auth_header_name": "Authorization"
      },
      "definition": {
        "location": "header",
        "key": "x-api-version"
      },
      "version_data": {
        "not_versioned": true,
        "versions": {
          "Default": {
            "name": "v1",
            "use_extended_paths": true,
            "extended_paths": {
              "url_rewrites": [
                {
                  "path": "/prometheus/",
                  "method": "GET",
                  "match_pattern": "(.*)",
                  "rewrite_to": "$1"
                }
              ]
            },
            "global_headers": {},
            "global_headers_remove": [],
            "global_size_limit": 0,
            "override_target": ""
          }
        }
      },
      "proxy": {
        "listen_path": "/prometheus/",
        "target_url": "http://tyk-tests-kube-prometheus-prometheus.tyk-tests.svc.cluster.local:9090/prometheus/",
        "strip_listen_path": true,
        "disable_strip_slash": false
      },
      "active": true
    }

This ignore section was a try by redirecting the “/prometheus/” path to Tyk’s FQDN /prometheus/graph, assuming it would be matched in the Proxy rules and then redirected internally to Prometheus host.

ignored": [
	{
		"path": "/prometheus/",
		"ignore_case": false,
		"method_actions": {
			"GET": {
				"action": "reply",
				"code": 301,
				"data": "Redirecting...",
				"headers": {
					"Location": "https://tyk.our.fake.host/prometheus/graph"
				}
			}
		}
	}
]

** Logs when /prometheus/graph **

time="Feb 26 14:18:02" level=debug msg=Started api_id=9 api_name=Prometheus key="****OCJ9" mw=URLRewriteMiddleware org_id=1 origin=11.240.20.13 path="/prometheus/graph" ts=1677421082986760441
time="Feb 26 14:18:02" level=debug msg="/prometheus/graph"
time="Feb 26 14:18:02" level=debug msg="Inbound path: /prometheus/graph"
time="Feb 26 14:18:02" level=debug msg="[[/prometheus/graph /prometheus/graph]]"
time="Feb 26 14:18:02" level=debug msg="URL Re-written from: /prometheus/graph"
time="Feb 26 14:18:02" level=debug msg="URL Re-written to: /prometheus/graph"
time="Feb 26 14:18:02" level=debug msg=Finished api_id=9 api_name=Prometheus code=200 key="****OCJ9" mw=URLRewriteMiddleware ns=101878 org_id=1 origin=11.240.20.13 path="/prometheus/graph"
time="Feb 26 14:18:02" level=debug msg="Stripping: /prometheus/"
time="Feb 26 14:18:02" level=debug msg=Started api_id=9 api_name=Prometheus mw=ReverseProxy org_id=1 ts=1677421082986898942
time="Feb 26 14:18:02" level=debug msg="Upstream request URL: graph" api_id=9 api_name=Prometheus mw=ReverseProxy org_id=1
time="Feb 26 14:18:02" level=debug msg="Outbound request URL: http://tyk-tests-kube-prometheus-prometheus.tyk-tests.svc.cluster.local:9090/prometheus/graph" api_id=9 api_name=Prometheus mw=ReverseProxy org_id=1

Regards,
Daniel

[dszortyka]

This solution from Martin seems to be exactly what I need:

Redirect separately to two similar GET endpoints

Sorry for late reply. Can I suggest upgrading to the latest version, we’re on 2.3 at the moment and there’s been loads of routing fixes. Your rewrites are in the wrong order, the {a} will always match, try reversing them: "extended_paths": { "url_rewrites": [ { "path": "/all", "method": "GET", "match_pattern": "all", "rewrite_to": "measurement" }, { "path": "{a}", …

1- If only /prometheus is provided (which is originally the listen_path), then I need rewrite to /prometheus/graph
2- If anything else is provided, like /prometheus/graph or /prometheus/alerts-> rewrite to /prometheus/$1.

"proxy": {
        "listen_path": "/prometheus/",
        "preserve_host_header": false,
        "target_url": "http://tyk-tests-kube-prometheus-prometheus.tyk-tests.svc.cluster.local:9090/prometheus/",
        "strip_listen_path": true,
        "disable_strip_slash": false
      },

This is how I’m defining url_rewrites:

"extended_paths": {
  "url_rewrites": [
    {
      "path": "/prometheus/",
      "method": "GET",
      "match_pattern": "prometheus",
      "rewrite_to": "/graph/"
    },
    {
      "path": "/prometheus/",
      "method": "GET",
      "match_pattern": "/prometheus/(\\w+)",
      "rewrite_to": "$1"
    }
  ]
},

Then, if just /prometheus is typed, goes to too many redirects loop.
If I manually hit /prometheus/graph, it works fine.

[dszortyka]

Just another update. I got it working, with a few “ugly” workarounds…
I tried so many different ways to get this working, anyway, this idea was based on this post

Would be nice if you guys could have either some examples on tools like prometheus and grafana running behind Tyk out of the box. It was a bit challenging for me, unless I’m missing something… (:

I think we can close this topic. Thanks @Olu !

prometheus.json: |
    {
      "name": "Prometheus",
      "slug": "prometheus",
      "api_id": "9",
      "org_id": "1",
      "use_basic_auth": true,
      "use_keyless": false,
      "enable_context_vars": true,
      "auth": {
        "auth_header_name": "Authorization"
      },
      "definition": {
        "location": "header",
        "key": "x-api-version"
      },
      "version_data": {
        "not_versioned": true,
        "versions": {
          "Default": {
            "name": "v1",
            "use_extended_paths": true,
            "extended_paths": {
              "url_rewrites": [
                {
                  "path": "/graph",
                  "method": "GET",
                  "match_pattern": "prometheus\/(.*)",
                  "rewrite_to": "prometheus/$1"
                },
                {
                  "path": "/static",
                  "method": "GET",
                  "match_pattern": "prometheus[\\^prometheus\\/]*(\\/.*)",
                  "rewrite_to": "prometheus$1"
                },
                {
                  "path": "/api",
                  "method": "GET",
                  "match_pattern": "prometheus[\\^prometheus\\/]*(\\/.*)",
                  "rewrite_to": "prometheus$1"
                },
                {
                  "path": "/graph",
                  "method": "GET",
                  "match_pattern": "graph",
                  "rewrite_to": "prometheus/graph/"
                },
                {
                  "path": "{a}",
                  "method": "GET",
                  "match_pattern": "(.*)",
                  "rewrite_to": "https://<TYK_FQDN>/prometheus/graph/"
                }
              ]
            },
            "global_headers": {},
            "global_headers_remove": [],
            "global_size_limit": 0,
            "override_target": ""
          }
        }
      },
      "proxy": {
        "listen_path": "/prometheus/",
        "preserve_host_header": false,
        "target_url": "http://tyk-tests-kube-prometheus-prometheus.tyk-tests.svc.cluster.local:9090/",
        "strip_listen_path": false,
        "disable_strip_slash": false,
        "transport": {
          "ssl_insecure_skip_verify": true,
          "ssl_force_common_name_check": false
        }
      },
      "active": true
    }

[Olu]

@dszortyka

uld be nice if you guys could have either some examples on tools like prometheus and grafana running behind Tyk out of the box. It was a bit challenging for me, unless I’m missing so

Great to hear that you worked it out