Request failed with status 403, Access to this API has been disallowed

Support
,
1

While using tyk gateway with high load getting below error also in the tyk logs, seeing timeout while connecting to redis.
tyk logs:-
time=“Apr 10 16:07:03” level=error msg=“Error trying to set value: dial tcp 10.0.78.10:6379: i/o timeout”
time=“Apr 10 16:07:03” level=error msg=“Error trying to set value: dial tcp 10.0.78.10:6379: i/o timeout”
time=“Apr 10 16:07:03” level=error msg=“Error trying to publish message: dial tcp 10.0.78.10:6379: i/o timeout”
time=“Apr 10 16:07:03” level=error msg=“Could not send notification: dial tcp 10.0.78.10:6379: i/o timeout” prefix=pub-sub
time=“Apr 10 16:07:03” level=error msg=“Can’t update session” error=“dial tcp 10.0.78.10:6379: i/o timeout”
time=“Apr 10 16:07:03” level=error msg=“Error trying to publish message: dial tcp 10.0.78.10:6379: i/o timeout”
time=“Apr 10 16:07:03” level=error msg=“Could not send notification: dial tcp 10.0.78.10:6379: i/o timeout” prefix=pub-sub
time=“Apr 10 16:07:03” level=error msg=“Can’t update session” error=“dial tcp 10.0.78.10:6379: i/o timeout”
time=“Apr 10 16:07:03” level=error msg=“Error trying to set value: dial tcp 10.0.78.10:6379: i/o timeout”
time=“Apr 10 16:07:03” level=error msg=“Error trying to set value: dial tcp 10.0.78.10:6379: i/o timeout”
time=“Apr 10 16:07:03” level=error msg=“Error trying to publish message: dial tcp 10.0.78.10:6379: i/o timeout”
time=“Apr 10 16:07:03” level=error msg=“Could not send notification: dial tcp 10.0.78.10:6379: i/o timeout” prefix=pub-sub
time=“Apr 10 16:07:03” level=error msg=“Can’t update session” api_id=uat-api api_name=uat-api error=“dial tcp 10.0.78.10:6379: i/o timeout” org_id=1
time=“Apr 10 16:07:03” level=error msg=“Can’t update session” api_id=uat-api api_name=uat-api error=“dial tcp 10.0.78.10:6379: i/o timeout” org_id=1
time=“Apr 10 16:07:04” level=error msg=“Error trying to append to set keys” error=“dial tcp 10.0.78.10:6379: i/o timeout”
time=“Apr 10 16:07:05” level=error msg=“Error trying to append to set keys” error=“dial tcp 10.0.78.10:6379: i/o timeout”

application logs:-
time=“2023-04-10T17:11:59Z” level=error msg=“Request failed with status 403. Response body: {\n "error": "Access to this API has been disallowed"\n}” source=console
time=“2023-04-10T17:11:59Z” level=error msg=“Request failed with status 403. Response body: {\n "error": "Access to this API has been disallowed"\n}” source=console
time=“2023-04-10T17:11:59Z” level=error msg=“Request failed with status 403. Response body: {\n "error": "Access to this API has been disallowed"\n}” source=console
time=“2023-04-10T17:11:59Z” level=error msg=“Request failed with status 403. Response body: {\n "error": "Access to this API has been disallowed"\n}” source=console
time=“2023-04-10T17:11:59Z” level=error msg=“Request failed with status 403. Response body: {\n "error": "Access to this API has been disallowed"\n}” source=console
time=“2023-04-10T17:11:59Z” level=error msg=“Request failed with status 403. Response body: {\n "error": "Access to this API has been disallowed"\n}” source=console

2

Hi Himanshu,

Thanks for writing in.
Can you provide your tyk.conf file to verify the config setting. Do you see any cpu/mem issue from redis side.

Regards,
Asha

3

@Himanshu_Bisht

There are a couple of errors in the logs shared, but I’ll just focus on 2.

403 errors “Access to this API has been disallowed” could be any of the possible issues:

  • Check if the key has access to the right API version
  • Check if the key has access to the right API definition
  • Check if the authentication key used is still valid
  • Check if the certificate used for authentication is present
  • Check if the authentication key is created and present in the database. You can use Gateway Keys APIs for confirmation
  • Check if API definition is using JWT auth and if auth header key and or value is empty or missing

Considering you mentioned you got this on high load, I would ask you to review our proxy error:context cancelled docs. I don’t see the errors shared in our docs but maybe sharing the debug logs could shed more light.

4

Hi @asha
here is my tyk.conf
{
“listen_port”: 8080,
“secret”: “352d20ee67be67f6340b4c0605b044b7”,
“template_path”: “/opt/tyk-gateway/templates”,
“tyk_js_path”: “/opt/tyk-gateway/js/tyk.js”,
“middleware_path”: “/opt/tyk-gateway/middleware”,
“use_db_app_configs”: false,
“logging”: {
“log_level”: “debug”,
“formats”: {
“json”: true
}
},
“enable_auth_header_logging”: true,
“app_path”: “/opt/tyk-gateway/apps/”,
“storage”: {
“type”: “redis”,
“host”: “tyk-redis”,
“port”: 6379,
“username”: “”,
“password”: “”,
“database”: 0,
“optimisation_max_idle”: 3000,
“optimisation_max_active”: 5000
},
“enable_analytics”: true,
“analytics_config”: {
“type”: “csv”,
“csv_dir”: “/tmp”,
“mongo_url”: “”,
“mongo_db_name”: “”,
“mongo_collection”: “”,
“purge_delay”: -1,
“ignored_ips”: []
},
“health_check”: {
“enable_health_checks”: true,
“health_check_value_timeouts”: 60
},
“optimisations_use_async_session_write”: true,
“enable_non_transactional_rate_limiter”: true,
“enable_sentinel_rate_limiter”: false,
“enable_redis_rolling_limiter”: false,
“allow_master_keys”: false,
“policies”: {
“policy_source”: “file”,
“policy_record_name”: “/opt/tyk-gateway/policies/policies.json”
},
“hash_keys”: true,
“close_connections”: false,
“http_server_options”: {
“enable_websockets”: true,
“enable_headers_all”: true
},
“allow_insecure_configs”: true,
“coprocess_options”: {
“enable_coprocess”: true,
“coprocess_grpc_server”: “”
},
“enable_bundle_downloader”: true,
“bundle_base_url”: “”,
“global_session_lifetime”: 100,
“force_global_session_lifetime”: false,
“max_idle_connections_per_host”: 500
}

no we are not seeing any cpu/mem issue on redis.

5
  • Check if the key has access to the right API version :- yes
  • Check if the key has access to the right API definition :- yes
  • Check if the authentication key used is still valid :- yes
  • Check if the certificate used for authentication is present:- not using any cert for auth
  • Check if the authentication key is created and present in the database. You can use Gateway Keys APIs for confirmation :- It is present
  • Check if API definition is using JWT auth and if auth header key and or value is empty or missing :- it is not missing.
6

I was suggesting maybe some OS system limits could be the issue. That’s why I shared this. The section below talks about file handles, opened sockets and the rest. If you are sure it works as expected on low or normal load then the file handles could simply be the issue here.

When it happens on the high load, it can be a lot of different reasons. For example your OS is running out of system limits, like number of opened sockets, and to validate it, you need to try your system limits. See this guide Planning for Production.