I am configuring 100 rate per 60 seconds with a python plugin.
When I am using a single gateway pod everything is working fine. but when we scale up number of pods we are seeing that
for first 60 seconds we are getting 200
but after that, rate never gets reset. We are constantly getting 429s
My middleware is like this
from tyk.decorators import *
from gateway import TykGateway as tyk
from policies import policy
import datetime
import os
import sys
parent_dir = os.path.abspath(os.path.dirname(__file__))
vendor_dir = os.path.join(parent_dir, 'vendor')
sys.path.append(vendor_dir)
@Hook
def MyAuthMiddleware(request, session, metadata, spec):
tyk.log("Starting Auth middleware", "info")
default_policy = policy["voyages"]["default"]
try:
tyk.log(f"request ip is {request.get_header('X-Real-Ip')}", "info")
if str(request.get_header('X-Real-Ip')) != '':
metadata["token"] = request.get_header('X-Real-Ip')
session.rate = default_policy["rate"]
session.per = default_policy["per"]
session.quota_max = default_policy["per"]
session.quota_renewal_rate = default_policy["quota_renewal_rate"]
session.quota_renews = int(datetime.datetime.now().timestamp()) + 60
session.session_lifetime = 300
except:
tyk.log("something went wrong", "info")
return request, session, metadata
Could you set log_level=“debug” mode and enable_key_logging. Then share the verbose logs of the resulting behaviour.
You should see something like this
level=debug msg="[RATELIMIT] Inbound raw key is: eyJvcmciOiJkZWZhdWx0IiwiaWQiOiI4ZWU3NGY3ZmRkMzM0YmQ4YWNlN2Q2YzIxYzhmM2Q1NSIsImgiOiJtdXJtdXI2NCJ9"
level=debug msg="[RATELIMIT] Rate limiter key is: rate-limit-eyJvcmciOiJkZWZhdWx0IiwiaWQiOiI4ZWU3NGY3ZmRkMzM0YmQ4YWNlN2Q2YzIxYzhmM2Q1NSIsImgiOiJtdXJtdXI2NCJ9"
level=debug msg="Incrementing raw key: rate-limit-eyJvcmciOiJkZWZhdWx0IiwiaWQiOiI4ZWU3NGY3ZmRkMzM0YmQ4YWNlN2Q2YzIxYzhmM2Q1NSIsImgiOiJtdXJtdXI2NCJ9"
level=debug msg="keyName is: rate-limit-eyJvcmciOiJkZWZhdWx0IiwiaWQiOiI4ZWU3NGY3ZmRkMzM0YmQ4YWNlN2Q2YzIxYzhmM2Q1NSIsImgiOiJtdXJtdXI2NCJ9"
You can use the key rate-limit-* to inspect Redis for the key and check the TTL.
On second look, you might just be hitting your quota limit. Kindly see token object details for more info
One things that I am observing is (for 100 rate per minutes setting), for first 100 calls, api call is successful where logs say ‘Returned: <some number between 1-99>’ for next 100 calls, I get 429 with logs saying ‘Returned: <some number between 101-199>’ but after that I am still getting 429 with logs ‘Returned: <199 or 200>’
While on redis side, I can see that tyk is creating key ‘rate-limit-*’ of type zset and every time the key getting updated a new key is getting created in zset
I am yet to try this with the structure of your plugin but when I test with round-robin loadbancer, I don’t experience any issues.
While on redis side, I can see that tyk is creating key ‘rate-limit-*’ of type zset and every time the key getting updated a new key is getting created in zset
From Redis what’s the Time To Live (TTL) of the key rate-limit-a76c8f79? It should start from the value set in per and count down every second
I think the issue here is that the redis rate limiter requires the traffic be below the ‘rate’ for a full ‘per’ period before it will allow more calls through. It isn’t a leaky bucket like the DRL (distributed rate limiter)
Another way to think of this is that 429s count towards the rate so they have to be below the rate limit for anything to be allowed through.
On second look, you might just be hitting your quota limit. Kindly see