i am using tyk gateway 4.3.4 using grpc at the backend
earlier it was working fine with not utf-8 characetrs for older tyk version
for example if i pass a header with value Lagardère it is giving 403 error in post man
in the backend logs it says rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8
can someone please help me to get root cause behind this ?
Can you share the previous version it was working with?
with non utf 8 character i am gettinf error :–
REQUEST:
1 tyk[22902]: %s
: [441B blob data]
: [210B blob data]
: time=“Sep 18 03:59:41” level=error msg=“Dispatch error” api_id=***** api_name="" error=“rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8” mw=CoProcessMiddleware org_id=default origin=* path=*****
tyk[22902]: REQUEST:
tyk[22902]: %s
tyk[22902]: ProcessRequest %s &{GET **** HTTP/1.1 1 1 map[Accept:[/] Accept-Encoding:[gzip, deflate, br] Connection:[keep-alive] Hh:[jj] Postman-Token:[e2fe4a43-eb4e-42ff-aa8f-cebcbd407c33] User-Agent:[PostmanRuntime/7.32.3]] {} 0 false ********t:8081 map map map 10.38.254.243:53400 /****0xc0000e9290 0xc0011a32f0}
[22902]: ProcessRequestHeader %s map[Accept:[/] Accept-Encoding:[gzip, deflate, br] Connection:[keep-alive] Hh:[jj] Postman-Token:[e2fe4a43-eb4e-42ff-aa8f-cebcbd407c33] User-Agent:[PostmanRuntime/7.32.3]]
Also i am seeing weird behaviour if i see request hit from multiple platforms:–
if i hit request via Gibash i get error:— 403 keys not authroized
Whereas if i hit this via Ubuntu shell the request is passing through to backend
is there any dependency on the calling system OS i.e windows and linux may have different behaviour while calling backend
@trust_level_0 @trust_level_1 @trust_level_2 @trust_level_3 @trust_level_4 @Olu
can anyone help me on this ?
this has become a blocker for me
Hi @Yogesh_Tolani, sorry about the delay.
Can you share your gateway config and API definition along with the a sample of the http or curl request with the expected output
hi @Olu
Please find attached my api json file
{
“name”: “",
“slug”: “”,
“listen_port”: 0,
“protocol”: “”,
“enable_proxy_protocol”: false,
“api_id”: "”,
“org_id”: “default”,
“use_keyless”: false,
“use_oauth2”: false,
“external_oauth”: {
“enabled”: false,
“providers”: null
},
“use_openid”: false,
“openid_options”: {
“providers”: null,
“segregate_by_client”: false
},
“oauth_meta”: {
“allowed_access_types”: null,
“allowed_authorize_types”: null,
“auth_login_redirect”: “”
},
“auth”: {
“name”: “”,
“use_param”: false,
“param_name”: “”,
“use_cookie”: false,
“cookie_name”: “”,
“disable_header”: false,
“auth_header_name”: “authorization”,
“use_certificate”: false,
“validate_signature”: false,
“signature”: {
“algorithm”: “”,
“header”: “”,
“use_param”: false,
“param_name”: “”,
“secret”: “”,
“allowed_clock_skew”: 0,
“error_code”: 0,
“error_message”: “”
}
},
“auth_configs”: null,
“use_basic_auth”: false,
“basic_auth”: {
“disable_caching”: false,
“cache_ttl”: 0,
“extract_from_body”: false,
“body_user_regexp”: “”,
“body_password_regexp”: “”
},
“use_mutual_tls_auth”: false,
“client_certificates”: null,
“upstream_certificates”: null,
“pinned_public_keys”: null,
“enable_jwt”: false,
“use_standard_auth”: false,
“use_go_plugin_auth”: false,
“enable_coprocess_auth”: true,
“jwt_signing_method”: “”,
“jwt_source”: “”,
“jwt_identity_base_field”: “”,
“jwt_client_base_field”: “”,
“jwt_policy_field_name”: “”,
“jwt_default_policies”: null,
“jwt_issued_at_validation_skew”: 0,
“jwt_expires_at_validation_skew”: 0,
“jwt_not_before_validation_skew”: 0,
“jwt_skip_kid”: false,
“scopes”: {
“jwt”: {},
“oidc”: {}
},
“jwt_scope_to_policy_mapping”: null,
“jwt_scope_claim_name”: “”,
“notifications”: {
“shared_secret”: “”,
“oauth_on_keychange_url”: “”
},
“enable_signature_checking”: false,
“hmac_allowed_clock_skew”: 0,
“hmac_allowed_algorithms”: null,
“request_signing”: {
“is_enabled”: false,
“secret”: “”,
“key_id”: “”,
“algorithm”: “”,
“header_list”: null,
“certificate_id”: “”,
“signature_header”: “”
},
“base_identity_provided_by”: “”,
“definition”: {
“enabled”: false,
“name”: “”,
“default”: “”,
“location”: “header”,
“key”: “version”,
“strip_path”: false,
“strip_versioning_data”: false,
“versions”: null
},
“version_data”: {
“not_versioned”: true,
“default_version”: “”,
“versions”: {
“Default”: {
“name”: “Default”,
“expires”: “3000-01-02 15:04”,
“paths”: {
“ignored”: null,
“white_list”: null,
“black_list”: null
},
“use_extended_paths”: true,
“extended_paths”: {
“persist_graphql”: null
},
“global_headers”: null,
“global_headers_remove”: null,
“global_response_headers”: null,
“global_response_headers_remove”: null,
“ignore_endpoint_case”: false,
“global_size_limit”: 0,
“override_target”: “”
}
}
},
“uptime_tests”: {
“check_list”: null,
“config”: {
“expire_utime_after”: 0,
“service_discovery”: {
“use_discovery_service”: false,
“query_endpoint”: “”,
“use_nested_query”: false,
“parent_data_path”: “”,
“data_path”: “”,
“port_data_path”: “”,
“target_path”: “”,
“use_target_list”: false,
“cache_timeout”: 0,
“endpoint_returns_list”: false
},
“recheck_wait”: 0
}
},
“proxy”: {
“preserve_host_header”: false,
“listen_path”: “/",
“target_url”: "https://************",
“disable_strip_slash”: false,
“strip_listen_path”: true,
“enable_load_balancing”: false,
“target_list”: null,
“check_host_against_uptime_tests”: false,
“service_discovery”: {
“use_discovery_service”: false,
“query_endpoint”: “”,
“use_nested_query”: false,
“parent_data_path”: “”,
“data_path”: “”,
“port_data_path”: “”,
“target_path”: “”,
“use_target_list”: false,
“cache_timeout”: 0,
“endpoint_returns_list”: false
},
“transport”: {
“ssl_insecure_skip_verify”: false,
“ssl_ciphers”: null,
“ssl_min_version”: 0,
“ssl_max_version”: 0,
“ssl_force_common_name_check”: false,
“proxy_url”: “”
}
},
“disable_rate_limit”: false,
“disable_quota”: true,
“custom_middleware”: {
“pre”: null,
“post”: null,
“post_key_auth”: null,
“auth_check”: {
“name”: "RateLimitByIP(3000,60)|RecordRESTHeader(,)|RecordRESTHeader(,*)”,
“path”: “”,
“require_session”: true,
“raw_body_only”: false
},
“response”: null,
“driver”: “grpc”,
“id_extractor”: {
“extract_from”: “”,
“extract_with”: “”,
“extractor_config”: null
}
},
“custom_middleware_bundle”: “”,
“cache_options”: {
“cache_timeout”: 0,
“enable_cache”: false,
“cache_all_safe_requests”: false,
“cache_response_codes”: null,
“enable_upstream_cache_control”: false,
“cache_control_ttl_header”: “”,
“cache_by_headers”: null
},
“session_lifetime”: 10,
“active”: false,
“internal”: false,
“auth_provider”: {
“name”: “”,
“storage_engine”: “”,
“meta”: null
},
“session_provider”: {
“name”: “”,
“storage_engine”: “”,
“meta”: null
},
“event_handlers”: {
“events”: null
},
“enable_batch_request_support”: true,
“enable_ip_whitelisting”: false,
“allowed_ips”: null,
“enable_ip_blacklisting”: false,
“blacklisted_ips”: null,
“dont_set_quota_on_create”: false,
“expire_analytics_after”: 60,
“response_processors”: null,
“CORS”: {
“enable”: false,
“allowed_origins”: null,
“allowed_methods”: null,
“allowed_headers”: null,
“exposed_headers”: null,
“allow_credentials”: false,
“max_age”: 0,
“options_passthrough”: false,
“debug”: false
},
“domain”: “”,
“certificates”: null,
“do_not_track”: false,
“enable_context_vars”: false,
“config_data”: null,
“tag_headers”: null,
“global_rate_limit”: {
“rate”: 0,
“per”: 0
},
“strip_auth_data”: false,
“enable_detailed_recording”: false,
“graphql”: {
“enabled”: false,
“execution_mode”: “”,
“version”: “”,
“schema”: “”,
“type_field_configurations”: null,
“playground”: {
“enabled”: false,
“path”: “”
},
“engine”: {
“field_configs”: null,
“data_sources”: null
},
“proxy”: {
“auth_headers”: null
},
“subgraph”: {
“sdl”: “”
},
“supergraph”: {
“subgraphs”: null,
“merged_sdl”: “”,
“global_headers”: null,
“disable_query_batching”: false
}
},
“analytics_plugin”: {},
“tags”: null
}
@Olu
attached is my tyk.conf file
{
“allow_insecure_configs”: true,
“allow_master_keys”: true,
“allow_remote_config”: false,
“analytics_config”: {
“csv_dir”: “/tmp”,
“enable_detailed_recording”: false,
“enable_geo_ip”: false,
“geo_ip_db_path”: “”,
“ignored_ips”: [
],
“mongo_collection”: “”,
“mongo_db_name”: “”,
“mongo_url”: “”,
“normalise_urls”: {
“custom_patterns”: [
],
“enabled”: true,
“normalise_numbers”: true,
“normalise_uuids”: true
},
“pool_size”: 100,
“purge_delay”: 100,
“storage_expiration_time”: 60,
“type”: “"
},
“app_path”: “/opt/tyk-gateway/apps”,
“bundle_base_url”: “”,
“close_connections”: true,
“close_idle_connections”: false,
“coprocess_options”: {
“coprocess_grpc_server”: “unix:///tmp/grpc-go.sock”,
“enable_coprocess”: true
},
“db_app_conf_options”: {
“connection_string”: “”,
“node_is_segmented”: false,
“tags”: [
]
},
“disable_dashboard_zeroconf”: true,
“enable_analytics”: true,
“enable_bundle_downloader”: false,
“enable_custom_domains”: true,
“enable_jsvm”: false,
“enable_non_transactional_rate_limiter”: true,
“enable_sentinel_rate_limiter”: false,
“enforce_org_data_detail_logging”: false,
“enforce_org_quotas”: false,
“experimental_process_org_off_thread”: false,
“force_global_session_lifetime”: false,
“global_session_lifetime”: 100,
“hash_keys”: true,
“health_check”: {
“enable_health_checks”: false,
“health_check_value_timeouts”: 60
},
“hostname”: “”,
“http_server_options”: {
“certificates”: [
{
“cert_file”: "",
“domain_name”: "",
“key_file”: ""
}
],
“enable_http2”: true,
“enable_websockets”: true,
“flush_interval”: 1,
“read_timeout”: 2000,
“ssl_insecure_skip_verify”: true,
“use_ssl”: true,
“write_timeout”: 2000
},
“listen_port”: 8081,
“local_session_cache”: {
“disable_cached_session_state”: false
},
“max_idle_connections_per_host”: 100,
“middleware_path”: “/opt/tyk-gateway/middleware”,
“node_secret”: "",
“oauth_redirect_uri_separator”: “;”,
“optimisations_use_async_session_write”: true,
“pid_file_location”: “./tyk-gateway.pid”,
“policies”: {
“allow_explicit_policy_id”: true,
“policy_connection_string”: “”,
“policy_record_name”: “tyk_policies”,
“policy_source”: “”
},
“proxy_default_timeout”: 6000,
“public_key_path”: “”,
“secret”: "",
“storage”: {
“database”: 0,
“enable_cluster”: true,
“host”: “localhost”,
“hosts”: {
"",
"",
"",
"",
"",
"***”
},
“optimisation_max_active”: 4000,
“optimisation_max_idle”: 2000,
“password”: “”,
“port”: 6379,
“type”: “redis”,
“username”: “”
},
“suppress_redis_signal_reload”: false,
“template_path”: “/opt/tyk-gateway/templates”,
“tyk_js_path”: “/opt/tyk-gateway/js/tyk.js”,
“uptime_tests”: {
“config”: {
“checker_pool_size”: **,
“enable_uptime_analytics”: **,
“failure_trigger_sample_size”: 3,
“time_wait”: 1
},
“disable”: true
},
“use_db_app_configs”: false,
“use_logstash”: false,
“use_redis_log”: false
}
@Yogesh_Tolani Thanks for the config and the API definition. We understand the issue has to do with custom authentication with gRPC middleware.
We were hoping to get a sample of the curl or http request when the issue happens. However, we were able to reproduce the issue regardless with golang gRPC.
What we were not able to reproduce is a difference in behavior between v2.5.1 and v4.3.4. The same UTF 8 error occurs when we make the request via postman UI but not when we copy the exact curl request from postman via code. We suspect there may be an issue with postman converting and transferring the request to Tyk. But maybe you could help us find more information.
You had earlier mentioned
Would you mind sharing the sample of that request and the environment specifications. e.g. OS.
We are happy to look further into the issue but as it stands, were were unable to find a difference in functionality between the two versions.
@olu tyk gateway is running on Ubuntu 18.04
this is the curl command i am using
$ curl -kL ‘https://:8081/<listen_path_of_api>’ -H ‘R: vérifier’ -H ‘Hh: jj’ -v
issue is with header vérifier and thus creating this issue
Are you executing the curl command from the same Ubuntu machine?
We’re not able to reproduce this with curl, only Postman.
@Ubong client can hit from any platform windows, linux or android
we have observed with windows it was creating issue and on linux it was working fine.
this seems to be because character set limitations in windows operating system
Any API Proxy irrespective of calling client should return same response independent of calling system
@Olu @Ubong i have not found this issue in 2.5.1 version but found on 4.3.4 version
were you able to replicate same ?
Hi @Yogesh_Tolani, @Mohit_Kumar
Thank you for your patience.
While we’ve been able to reproduce the marshal error, we’re not able to confirm a difference in behaviour of the Gateway between v2.5.1 and 4.3.4.
We see the marshal error on both v2.5.1 and v4.3.1 when we use Gitbash on Windows. But there’s no error when using the Command Prompt.
This leads us to believe it’s an issue with Gitbash, not necessarily Windows - perhaps Gitbash isn’t properly utilising the character set available in the operating system. I’ve done some research and come across this post. Unfortunately, implementing one of the solutions mentioned made no difference in my test, but maybe you’d have a better experience.
I’ve created this repo containing the setup I’ve used to perform the tests across both versions. It contains gateway, API def and policy def. I’ve used one of our official grpc plugins for the grpc server. Please check through if I’ve missed anything.
We would generally recommend channelling research efforts towards Git Bash as it seems to be the outlier among calling clients. If that’s not the case, please let us know what other clients are experiencing the issue.
Let us know how you get on.