Url rewrite plugin in combination with the whitelist plugin

Support
1

Hi all,

It looks like there is a bug in the endpoint designer with the url rewrite plugin in combination with the whitelist plugin. Please see the image below. I have defined the endpoint “/commercialproduct_get”. I have enabled the plugin whitelist and I have enabled the URL rewrite plugin for this endpoint with the following properties:

  • Match pattern: (?i)/commercialproduct_get
  • Rewrite to: /General_Product_API/rest/CommercialProduct_V1/CommercialProduct_Get

When I perform an HTTP request to this url: “https://HOST/general-services/CommercialProduct_Get” it looks like that the url rewrite doesn’t work correct because I receive a HTTP 404. The backend system returns the HTTP 404 response because the path doesn’t exist.

When I perform an HTTP request to this url: “https://HOST/general-services/commercialproduct_get” (lowercase) it works correctly. I’m able to reach the endpoint.

I’m not able to change the path to “CommercialProduct_Get” because the whitelist plugin requires a lowercase path. I have a feeling that this bug is related to the lowercases used in the path in combination with the lowercases match pattern in the URL rewrite plugin. The whitelist plugin accepts the path “CommercialProduct_Get” but the URL rewrite plugin tries to find the string “commercialproduct_get” and it cannot find the string due lowercases. Therefore the URL rewrite is not executed and the backend system does not have an endpoint “commercialproduct_get”.

I have already tried to use regex pattern (?i) to make the match pattern case-insensitive but this doesn’t work.

Could you please try to reproduce this bug in your own environment or do you have a workaround for this bug?

image

2

API definition:
{
“api_definition”: {
“api_id”: “dd4af40dbcea484050fc00b8e26f03fc”,
“upstream_certificates”: {},
“use_keyless”: true,
“enable_coprocess_auth”: false,
“custom_middleware”: {
“pre”: [],
“post”: [],
“post_key_auth”: [],
“auth_check”: {
“name”: “”,
“path”: “”,
“require_session”: false
},
“response”: [],
“driver”: “”,
“id_extractor”: {
“extract_from”: “”,
“extract_with”: “”,
“extractor_config”: {}
}
},
“disable_quota”: false,
“custom_middleware_bundle”: “”,
“cache_options”: {
“enable_cache”: false,
“enable_upstream_cache_control”: false,
“cache_timeout”: 60,
“cache_response_codes”: [],
“cache_all_safe_requests”: false
},
“enable_ip_blacklisting”: false,
“tag_headers”: [],
“pinned_public_keys”: {},
“domain”: “”,
“openid_options”: {
“providers”: [],
“segregate_by_client”: false
},
“active”: true,
“config_data”: {},
“notifications”: {
“oauth_on_keychange_url”: “”,
“shared_secret”: “”
},
“auth”: {
“auth_header_name”: “”,
“use_certificate”: false
},
“check_host_against_uptime_tests”: false,
“blacklisted_ips”: [],
“hmac_allowed_clock_skew”: -1,
“uptime_tests”: {
“check_list”: [],
“config”: {
“service_discovery”: {
“use_discovery_service”: false,
“query_endpoint”: “”,
“use_nested_query”: false,
“parent_data_path”: “”,
“data_path”: “”,
“cache_timeout”: 60
}
}
},
“enable_jwt”: false,
“name”: “General services”,
“slug”: “general-services”,
“oauth_meta”: {
“allowed_access_types”: [],
“allowed_authorize_types”: [],
“auth_login_redirect”: “”
},
“CORS”: {
“enable”: true,
“max_age”: 24,
“allow_credentials”: false,
“exposed_headers”: [],
“allowed_headers”: [
“"
],
“options_passthrough”: false,
“debug”: false,
“allowed_origins”: [
"”
],
“allowed_methods”: [
“GET”,
“POST”,
“PUT”,
“DELETE”
]
},
“event_handlers”: {
“events”: {}
},
“proxy”: {
“target_url”: “https://BACKEND/”,
“service_discovery”: {
“cache_timeout”: 0,
“parent_data_path”: “”,
“query_endpoint”: “”,
“use_discovery_service”: false,
“_sd_show_port_path”: false,
“target_path”: “”,
“use_nested_query”: false,
“data_path”: “”,
“port_data_path”: “”
},
“check_host_against_uptime_tests”: false,
“transport”: {
“ssl_ciphers”: [],
“ssl_min_version”: 0,
“proxy_url”: “”
},
“target_list”: [],
“preserve_host_header”: false,
“strip_listen_path”: true,
“enable_load_balancing”: false,
“listen_path”: “/general-services”
},
“client_certificates”: [],
“use_basic_auth”: false,
“version_data”: {
“not_versioned”: true,
“versions”: {
“Default”: {
“name”: “Default”,
“expires”: “”,
“override_target”: “”,
“extended_paths”: {
“white_list”: [
{
“method”: “”,
“timeout”: 0,
“path”: “/General_Company_API/rest/GeneralDebtor_v1/debtor_get”,
“add_headers”: {},
“samples”: 100,
“rewrite_to”: “”,
“template_data”: {
“input_type”: “”,
“template_mode”: “”,
“enable_session”: false,
“template_source”: “”
},
“function_source_uri”: “”,
“threshold_percent”: 0.1,
“delete_headers”: [],
“response_function_name”: “myVirtualHandler”,
“_lists”: [],
“match_pattern”: “”,
“function_source_type”: “blob”,
“return_to_service_after”: 60,
“size_limit”: 0,
“use_session”: false,
“method_actions”: {
“GET”: {
“action”: “no_action”,
“code”: 200,
“headers”: {}
}
}
},
{
“method”: “”,
“timeout”: 0,
“path”: “/commercialproduct_get”,
“add_headers”: {},
“samples”: 100,
“rewrite_to”: “”,
“template_data”: {
“input_type”: “”,
“template_mode”: “”,
“enable_session”: false,
“template_source”: “”
},
“function_source_uri”: “”,
“threshold_percent”: 0.1,
“delete_headers”: [],
“response_function_name”: “myVirtualHandler”,
“_lists”: [],
“match_pattern”: “”,
“function_source_type”: “blob”,
“return_to_service_after”: 60,
“size_limit”: 0,
“use_session”: false,
“method_actions”: {
“POST”: {
“action”: “no_action”,
“code”: 200,
“headers”: {}
}
}
},
{
“method”: “”,
“timeout”: 0,
“path”: “/debtor_get”,
“add_headers”: {},
“samples”: 100,
“rewrite_to”: “”,
“template_data”: {
“input_type”: “”,
“template_mode”: “”,
“enable_session”: false,
“template_source”: “”
},
“function_source_uri”: “”,
“threshold_percent”: 0.1,
“delete_headers”: [],
“response_function_name”: “myVirtualHandler”,
“_lists”: [],
“match_pattern”: “”,
“function_source_type”: “blob”,
“return_to_service_after”: 60,
“size_limit”: 0,
“use_session”: false,
“method_actions”: {
“GET”: {
“action”: “no_action”,
“code”: 200,
“headers”: {}
}
}
},
{
“method”: “”,
“timeout”: 0,
“path”: “/test”,
“add_headers”: {},
“samples”: 100,
“rewrite_to”: “”,
“template_data”: {
“input_type”: “”,
“template_mode”: “”,
“enable_session”: false,
“template_source”: “”
},
“function_source_uri”: “”,
“threshold_percent”: 0.1,
“delete_headers”: [],
“response_function_name”: “myVirtualHandler”,
“_lists”: [],
“match_pattern”: “”,
“function_source_type”: “blob”,
“return_to_service_after”: 60,
“size_limit”: 0,
“use_session”: false,
“method_actions”: {
“GET”: {
“action”: “no_action”,
“code”: 200,
“headers”: {}
}
}
}
],
“url_rewrites”: [
{
“method”: “POST”,
“timeout”: 0,
“path”: “/commercialproduct_get”,
“add_headers”: {},
“samples”: 100,
“rewrite_to”: “/General_Product_API/rest/CommercialProduct_V1/CommercialProduct_Get”,
“template_data”: {
“input_type”: “”,
“template_mode”: “”,
“enable_session”: false,
“template_source”: “”
},
“function_source_uri”: “”,
“threshold_percent”: 0.1,
“delete_headers”: [],
“response_function_name”: “myVirtualHandler”,
“_lists”: [],
“match_pattern”: “(?i)/commercialproduct_get”,
“function_source_type”: “blob”,
“triggers”: [],
“return_to_service_after”: 60,
“size_limit”: 0,
“use_session”: false,
“method_actions”: {}
},
{
“method”: “GET”,
“timeout”: 0,
“path”: “/debtor_get”,
“add_headers”: {},
“samples”: 100,
“rewrite_to”: “/General_Company_API/rest/GeneralDebtor_v1/debtor_get$1”,
“template_data”: {
“input_type”: “”,
“template_mode”: “”,
“enable_session”: false,
“template_source”: “”
},
“function_source_uri”: “”,
“threshold_percent”: 0.1,
“delete_headers”: [],
“response_function_name”: “myVirtualHandler”,
“_lists”: [],
“match_pattern”: “/debtor_get(.*)”,
“function_source_type”: “blob”,
“triggers”: [],
“return_to_service_after”: 60,
“size_limit”: 0,
“use_session”: false,
“method_actions”: {}
}
],
“track_endpoints”: [
{
“method”: “POST”,
“timeout”: 0,
“path”: “/commercialproduct_get”,
“add_headers”: {},
“samples”: 100,
“rewrite_to”: “”,
“template_data”: {
“input_type”: “”,
“template_mode”: “”,
“enable_session”: false,
“template_source”: “”
},
“function_source_uri”: “”,
“threshold_percent”: 0.1,
“delete_headers”: [],
“response_function_name”: “myVirtualHandler”,
“_lists”: [],
“match_pattern”: “”,
“function_source_type”: “blob”,
“return_to_service_after”: 60,
“size_limit”: 0,
“use_session”: false,
“method_actions”: {}
},
{
“method”: “GET”,
“timeout”: 0,
“path”: “/debtor_get”,
“add_headers”: {},
“samples”: 100,
“rewrite_to”: “”,
“template_data”: {
“input_type”: “”,
“template_mode”: “”,
“enable_session”: false,
“template_source”: “”
},
“function_source_uri”: “”,
“threshold_percent”: 0.1,
“delete_headers”: [],
“response_function_name”: “myVirtualHandler”,
“_lists”: [],
“match_pattern”: “”,
“function_source_type”: “blob”,
“return_to_service_after”: 60,
“size_limit”: 0,
“use_session”: false,
“method_actions”: {}
}
]
},
“global_headers”: {},
“global_headers_remove”: [],
“global_size_limit”: 0,
“use_extended_paths”: true
}
},
“default_version”: “”
},
“use_standard_auth”: false,
“disable_rate_limit”: false,
“definition”: {
“key”: “x-api-version”,
“location”: “header”
},
“use_oauth2”: false,
“allowed_ips”: [],
“org_id”: “58924c7c660450026f755e09”,
“enable_ip_whitelisting”: false,
“global_rate_limit”: {
“rate”: 0,
“per”: 0
},
“enable_context_vars”: false,
“tags”: [],
“strip_auth_data”: false,
“id”: “5b83c24fb8763d27f6f3f0a0”,
“enable_signature_checking”: false,
“use_openid”: false,
“enable_batch_request_support”: false,
“response_processors”: [],
“use_mutual_tls_auth”: false,
“auth_provider”: {
“name”: “”,
“storage_engine”: “”,
“meta”: {}
},
“do_not_track”: false,
“dont_set_quota_on_create”: false,
“expire_analytics_after”: 0,
“session_lifetime”: 0,
“jwt_disable_issued_at_validation”: false,
“jwt_disable_expires_at_validation”: false,
“jwt_disable_not_before_validation”: false
},
“hook_references”: [],
“is_site”: false,
“sort_by”: 0
}

3

Thanks Tim we’ll look into it and get back to you soon.

4

Hi Tim

From out docs //tyk.io/docs/advanced-configuration/transform-traffic/endpoint-designer/

NOTE: For security reasons, the whitelist plugin is case-insensitive when performing the whitelist check. For example, if path /DoSomething is added to a whitelist, the gateway will listen on the path /DoSomething but the path /dosomething would be whitelisted. Which means that the path will not be accessible and a 403 Forbidden response will always be returned.

So i think the uppercase url will not ever work for your case.

5

Josh,

Thank you for your reply. Please see this link: http - Are URIs case-insensitive? - Stack Overflow. Apparently there are some web servers which have case sensitive URI’s. Can this be considered as a bug in Tyk or do you have a different opinion about this?

Kind regards,

Tim

6

Josh,

I used a different regex pattern and with that pattern I’m able to perform a succesful http call to the url:
https://HOST/general-services/CommercialProduct_Get

image
image1550×565 35.6 KB

Nevertheless, it would be nicer if this bug could be fixed in Tyk.

Kind regards,

Tim

7

Glad you got it working Tim - i’ll look into it!

8

Hi @Josh,

Could you please give me an update about this issue?

9

Hi Tim sorry for the delay we’ve been having some internal discussion around this issue. The whitelist having the restriction is mainly for legacy reasons from the old white/blacklist functionality but we can make it configurable to change the way it functions.

We’ll get the change in our backlog for either a patch or full release so you wont require the workaround.

Thanks
Josh

10

Hi Josh,

Glad to hear that you are going to fix the bug. Can you please provide me the github ticket so we can monitor it.

Thanks
Tim