Trouble with Tyk: URL Rewrite and Response Transform Not Working Together

Product Bug Reports
1

Branch/Environment/Version

  • Branch/Version: [Release 5.2.1]
  • Environment: [On-prem]

Describe the bug
I’m facing issues configuring API definition with both URL rewrite and response transform features. Specifically, I’ve set up rules for URL rewriting and response body transformation, but they don’t seem to work seamlessly together. The URL rewrite is functioning as expected, but the response body transformation is not working. The relative path used here is something like “/download$” as I am also white-listing this relative path (this will forbid additional segment e.g., /download/1).

Reproduction steps (see the second reply for more details)
Steps to reproduce the behaviour:

  1. Create an API definition
  2. Add URL rewrite
  3. Add response body transform

Actual behaviour
response body transform is not working, the response body is not transformed.

Expected behaviour
response body is transformed

2

@toto35711 Did you remember to add the response_processors?

    "response_processors": [
        {
            "name": "response_body_transform",
            "options": {}
        }
    ]

If you did, then can you share an API definition example?

3

Hi, below is my relevant API definition snippet. The current configuration involves two key processes. First, it rewrites the relative path “/download$” to “/reportDownload,” and white-listing the exact path “/download$” (so, it is forbidding any additional segments to be added after it (e.g., “/download/1” is forbidden). Second, it transforms the JSON body of the request to XML and subsequently transforms the XML body of the response back to JSON.

   "version_data": {
      "not_versioned": true,
      "default_version": "1.0.0",
      "versions": {
        "1.0.0": {
          "name": "1.0.0",
          "expires": "",
          "paths": {
            "ignored": [],
            "white_list": [],
            "black_list": []
          },
          "use_extended_paths": true,
          "global_headers": {},
          "global_headers_remove": [],
          "global_response_headers": {},
          "global_response_headers_remove": [],
          "ignore_endpoint_case": false,
          "global_size_limit": 0,
          "override_target": "",
          "extended_paths": {
            "white_list": [
              {
                "disabled": false,
                "path": "/reportDownload",
                "method": "",
                "ignore_case": false,
                "method_actions": {
                  "POST": {
                    "action": "no_action",
                    "code": 200,
                    "headers": {}
                  }
                }
              },
              {
                "disabled": false,
                "path": "/download$",
                "method": "",
                "ignore_case": true,
                "method_actions": {
                  "POST": {
                    "action": "no_action",
                    "code": 200,
                    "headers": {}
                  }
                }
              }
            ],
            "transform": [
              {
                "disabled": false,
                "template_data": {
                  "input_type": "json",
                  "template_mode": "blob",
                  "enable_session": false,
                  "template_source": "PHNvYXBlbnY6RW52ZWxvcGUgeG1sbnM6c29hcGVudj0iaHR0cDovL3NjaGVtYXMueG1sc29hcC5vcmcvc29hcC9lbnZlbG9wZS8iIHhtbG5zOnJlcD0iaHR0cDovL3NvYXAtdGVzdC5jb20vcmVwb3J0Ij4NCiAgICA8c29hcGVudjpIZWFkZXIvPg0KICAgIDxzb2FwZW52OkJvZHk+DQogICAgICAgIDxyZXA6cmVwb3J0RG93bmxvYWQ+DQogICAgICAgICAgICA/DQogICAgICAgIDwvcmVwOnJlcG9ydERvd25sb2FkPg0KICAgIDwvc29hcGVudjpCb2R5Pg0KPC9zb2FwZW52OkVudmVsb3BlPg==",
                  "input": "",
                  "output": ""
                },
                "path": "/reportDownload",
                "method": "POST"
              },
              {
                "disabled": false,
                "template_data": {
                  "input_type": "json",
                  "template_mode": "blob",
                  "enable_session": false,
                  "template_source": "PHNvYXBlbnY6RW52ZWxvcGUgeG1sbnM6c29hcGVudj0iaHR0cDovL3NjaGVtYXMueG1sc29hcC5vcmcvc29hcC9lbnZlbG9wZS8iIHhtbG5zOnJlcD0iaHR0cDovL3NvYXAtdGVzdC5jb20vcmVwb3J0Ij4NCiAgICA8c29hcGVudjpIZWFkZXIvPg0KICAgIDxzb2FwZW52OkJvZHk+DQogICAgICAgIDxyZXA6cmVwb3J0RG93bmxvYWQ+DQogICAgICAgICAgICA/DQogICAgICAgIDwvcmVwOnJlcG9ydERvd25sb2FkPg0KICAgIDwvc29hcGVudjpCb2R5Pg0KPC9zb2FwZW52OkVudmVsb3BlPg==",
                  "input": "",
                  "output": ""
                },
                "path": "/download$",
                "method": "POST"
              }
            ],
            "transform_response": [
              {
                "disabled": false,
                "template_data": {
                  "input_type": "xml",
                  "template_mode": "blob",
                  "enable_session": false,
                  "template_source": "e3sgJHJlc3BvbnNlIDo9IC5FbnZlbG9wZS5Cb2R5LnJlcG9ydERvd25sb2FkUmVzcG9uc2UgfCBqc29uTWFyc2hhbCB8IGZyb21Kc29uIH19e3sgb21pdCAkcmVzcG9uc2UgIi1ucyIgfCB0b1ByZXR0eUpzb24gfX0=",
                  "input": "",
                  "output": ""
                },
                "path": "/reportDownload",
                "method": "POST"
              },
              {
                "disabled": false,
                "template_data": {
                  "input_type": "xml",
                  "template_mode": "blob",
                  "enable_session": false,
                  "template_source": "e3sgJHJlc3BvbnNlIDo9IC5FbnZlbG9wZS5Cb2R5LnJlcG9ydERvd25sb2FkUmVzcG9uc2UgfCBqc29uTWFyc2hhbCB8IGZyb21Kc29uIH19e3sgb21pdCAkcmVzcG9uc2UgIi1ucyIgfCB0b1ByZXR0eUpzb24gfX0=",
                  "input": "",
                  "output": ""
                },
                "path": "/download$",
                "method": "POST"
              }
            ],
            "transform_headers": [
              {
                "delete_headers": [],
                "add_headers": {
                  "content-type": "text/xml"
                },
                "path": "/reportDownload",
                "method": "POST",
                "act_on": false
              },
              {
                "delete_headers": [],
                "add_headers": {
                  "content-type": "text/xml"
                },
                "path": "/download$",
                "method": "POST",
                "act_on": false
              }
            ],
            "transform_response_headers": [
              {
                "delete_headers": [],
                "add_headers": {
                  "content-type": "applications/json"
                },
                "path": "/reportDownload",
                "method": "POST",
                "act_on": false
              },
              {
                "delete_headers": [],
                "add_headers": {
                  "content-type": "application/json"
                },
                "path": "/download$",
                "method": "POST",
                "act_on": false
              }
            ],
            "url_rewrites": [
              {
                "path": "/download$",
                "method": "POST",
                "match_pattern": "/download$",
                "rewrite_to": "/reportDownload",
                "triggers": []
              }
            ]
          }
        }
      }
    },

    "response_processors": [
      {
        "name": "header_injector",
        "options": {}
      },
      {
        "name": "response_body_transform",
        "options": {}
      }
    ],

Upon further investigation, it has been observed that removing the “$” in the URL rewrite path enables the successful application of the body response transformation. However, it is crucial to note that this modification also permits additional segments to be added after “/download” (e.g., “/download/1” will not be forbidden). Is there any solution that allows for both the current whitelisting, preventing additional segments after “/download,” and the successful application of the body response transformation?

As for comparison, using the relative path /reportDownload directly (this one does not has the $ and URL rewrite), it works seamlessly.

Thanks in advance.

Below is a screenshot for some visual.

7
7821×473 24.6 KB

4

I am not sure I understand the full scope of the issue. Maybe a sample request and response would paint a better picture

However, if the $ character is the issue here, then can you point out if it’s either within the path property or the match_pattern property? Or maybe even both? For instance, do you have to remove the $ symbol on both fields or only just one?

5

Here is the sample request and response using the relative path “/download$”. Notice that the response body is still in XML format.

8
8942×542 34.5 KB

Using the direct relative path “/reportDownload”, here the response body transform works seamlessly (the response is in JSON format).

9
9945×437 24.3 KB
.

here is the snippets of transform-related properties of path “/download$”:

            "transform_response": [

              ...

              {
                "disabled": false,
                "template_data": {
                  "input_type": "xml",
                  "template_mode": "blob",
                  "enable_session": false,
                  "template_source": "e3sgJHJlc3BvbnNlIDo9IC5FbnZlbG9wZS5Cb2R5LnJlcG9ydERvd25sb2FkUmVzcG9uc2UgfCBqc29uTWFyc2hhbCB8IGZyb21Kc29uIH19e3sgb21pdCAkcmVzcG9uc2UgIi1ucyIgfCB0b1ByZXR0eUpzb24gfX0=",
                  "input": "",
                  "output": ""
                },
                "path": "/download$",
                "method": "POST"
              }
            ],
            "transform_response_headers": [
              
             ...

              {
                "delete_headers": [],
                "add_headers": {
                  "content-type": "application/json"
                },
                "path": "/download$",
                "method": "POST",
                "act_on": false
              }
            ],
            ...

            "url_rewrites": [
              {
                "path": "/download$",
                "method": "POST",
                "match_pattern": "/download$",
                "rewrite_to": "/reportDownload",
                "triggers": []
              }
            ]

Notice that both path and match_pattern properties are “/download$”.

6

I can reproduce this. It’s a bit weird that the $ character even makes it work. I’ll file a bug ticket internally.

As for a workaround, you could create a blacklist for /download(.+), which should match anything afterward.

"black_list": [
      {
        "disabled": false,
        "path": "download(.+)",
        "method": "",
        "ignore_case": false,
        "method_actions": {
          "POST": {
            "action": "no_action",
            "code": 200,
            "headers": {}
          }
        }
      }
    ]