Developer's API key not honoring associated policies

That will be it, all operations on APIs should be done within the context of an organisation, i.e. with an Organisation-level user, not an unbounded superuser.

Well, sorry to say but I just created another developer, and same thing: rate limit exceeded :frowning:

EDIT: debug log output from gateway

time="Apr 27 13:50:28" level=debug msg="Request size limiter active" 
time="Apr 27 13:50:28" level=debug msg="Global limit is: 0" 
time="Apr 27 13:50:28" level=debug msg="[STORE] Getting WAS: 56f3e529fb9231000a000001a8c686c17e3f49cb5a8b2a20b0c72c32" 
time="Apr 27 13:50:28" level=debug msg="Input key was: apikey-50f2c89f" 
time="Apr 27 13:50:28" level=debug msg="[STORE] Getting: apikey-50f2c89f" 
time="Apr 27 13:50:28" level=debug msg="Input key was: apikey-50f2c89f" 
time="Apr 27 13:50:28" level=debug msg="Session has policy, checking" 
time="Apr 27 13:50:28" level=debug msg="[RATELIMIT] Inbound raw key is: 56f3e529fb9231000a000001a8c686c17e3f49cb5a8b2a20b0c72c32" 
time="Apr 27 13:50:28" level=debug msg="[RATELIMIT] Rate limiter key is: rate-limit-50f2c89f" 
time="Apr 27 13:50:28" level=debug msg="Incrementing raw key: rate-limit-50f2c89f" 
time="Apr 27 13:50:28" level=debug msg="keyName is: rate-limit-50f2c89f" 
time="Apr 27 13:50:28" level=debug msg="Now is:2016-04-27 13:50:28.213211087 +0000 UTC" 
time="Apr 27 13:50:28" level=debug msg="Then is: 2016-04-27 13:50:28.213211087 +0000 UTC" 
time="Apr 27 13:50:28" level=debug msg="Returned: 0" 
time="Apr 27 13:50:28" level=debug msg="SessionState: {0 0 0 0 0 0 1461763929 0 0 map[7b393528bfca4549506b7083657d76e8:{Sendify Api 7b393528bfca4549506b7083657d76e8 [Default] []}] 56f3e529fb9231000a000001  map[] { } {} false  false 56f3e6affb9231000a000005 0 {[]} false map[tyk_developer_id:5720bf3b4477120008000001 tyk_key_request_fields:map[] tyk_user_fields:map[]] []}" 
time="Apr 27 13:50:28" level=debug msg="X-Forwarded-For set, remote IP: 192.168.10.1" 
time="Apr 27 13:50:28" level=info msg="Key rate limit exceeded." key=56f3e529fb9231000a000001a8c686c17e3f49cb5a8b2a20b0c72c32 origin=192.168.10.1 path="/s-api/" 
time="Apr 27 13:50:28" level=debug msg="X-Forwarded-For set, remote IP: 192.168.10.1" 
time="Apr 27 13:50:28" level=debug msg="X-Forwarded-For set, remote IP: 192.168.10.1" 
time="Apr 27 13:50:28" level=debug msg="X-Forwarded-For set, remote IP: 192.168.10.1" 
time="Apr 27 13:50:28" level=debug msg="Returning error header" 
time="Apr 27 13:50:28" level=debug msg="Adding Healthcheck to: 7b393528bfca4549506b7083657d76e8.BlockedRequest" 
time="Apr 27 13:50:28" level=debug msg="Val is: -1" 
time="Apr 27 13:50:28" level=debug msg="[STORE] SET Raw key is: 56f3e529fb9231000a000001a8c686c17e3f49cb5a8b2a20b0c72c32" 
time="Apr 27 13:50:28" level=debug msg="Incrementing raw key: 7b393528bfca4549506b7083657d76e8.BlockedRequest" 
time="Apr 27 13:50:28" level=debug msg="Input key was: apikey-50f2c89f" 
time="Apr 27 13:50:28" level=debug msg="keyName is: 7b393528bfca4549506b7083657d76e8.BlockedRequest" 
time="Apr 27 13:50:28" level=debug msg="[STORE] Setting key: apikey-50f2c89f" 
time="Apr 27 13:50:28" level=debug msg="Input key was: apikey-50f2c89f" 
time="Apr 27 13:50:28" level=debug msg="Now is:2016-04-27 13:50:28.214472378 +0000 UTC" 
time="Apr 27 13:50:28" level=debug msg="Then is: 2016-04-27 13:49:28.214472378 +0000 UTC" 
time="Apr 27 13:50:28" level=debug msg="Adding Healthcheck to: 7b393528bfca4549506b7083657d76e8.Throttle" 
time="Apr 27 13:50:28" level=debug msg="Val is: -1" 
time="Apr 27 13:50:28" level=debug msg="Pushing to raw key list: tyk-system-analytics" 
time="Apr 27 13:50:28" level=debug msg="Input key was: analytics-tyk-system-analytics" 
time="Apr 27 13:50:28" level=debug msg="Appending to fixed key list: analytics-tyk-system-analytics" 
time="Apr 27 13:50:28" level=debug msg="Input key was: analytics-tyk-system-analytics" 
time="Apr 27 13:50:28" level=debug msg="EVENT FIRED" 
time="Apr 27 13:50:28" level=debug msg="Incrementing raw key: 7b393528bfca4549506b7083657d76e8.Throttle" 
time="Apr 27 13:50:28" level=debug msg="keyName is: 7b393528bfca4549506b7083657d76e8.Throttle" 
time="Apr 27 13:50:28" level=debug msg="Now is:2016-04-27 13:50:28.214717733 +0000 UTC" 
time="Apr 27 13:50:28" level=debug msg="Then is: 2016-04-27 13:49:28.214717733 +0000 UTC" 
time="Apr 27 13:50:28" level=debug msg="Returned: 1" 

Is the policy you are using mixed-auth? As in, are the underlying APIs using the same authentication mode? (note: this includes Open APIs)

Were the API, Policy, Portal, Developer and Catalogue entries all created with a user that belongs to the organisation?

No, It’s just one api in the policy and it’s Access Token based.

I used a user which was created by the bootstrap.sh script in /install dir.

I could look in the db and see what comes up for that dashboard user’s org and see if it differs from the main org.

The bootstrap script creates a new org to generate that user, so they likely do not match - although you wouldn’t be able to access APIs across orgs.

This bug is seriously hard to replicate

I only have one org and it’s the one created by that script.
I had a look in mongo and the org id is tied to user, api and policy.

Can you create tokens manually using the dashboard?

Do you mean the key-requests for a developer? I can generate those.

No, I mean:

  1. Go to keys
  2. Add Key
  3. Add a policy and a base API access right
  4. Click save

Sorry, just realised that, yeah I can generate and that seems to get through

I looked up a key tied to a developer and see that rate and per seconds is 0??

EDIT: so yeah, that seems to be it, the keys are generated like that for the developers for some reason, with 0 rate and per second values.

Is the portal catalogue entry new or old? You mentioned this was an upgraded install with an old DB - we did do a “version switch” for catalogue entries way back that can cause weird effects.

Can you create a new policy, and a new catalogue entry, then try to generate a key for a new dev?

Catalog entry is old. Will try that.

Tried it there now and still the same. Will try again a bit later on too, want to be sure I did things correctly.

Still get rate limit exceeded

This is one tough bug! Ok, leave it with me, this may need some digging.

Last request: What version did you upgrade from, what are you running on (Docker & version etc.) and can you share the tyk.conf and tyk_analytics.conf files?

Dashboard was v0.9.7.2, Gateway was v1.9.1.1 before upgrade to v1.0 and v2.0

Running on docker upon ubuntu 14.04

Client:
 Version:      1.11.0
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   4dc5990
 Built:        Wed Apr 13 18:38:59 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.0
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   4dc5990
 Built:        Wed Apr 13 18:34:23 2016
 OS/Arch:      linux/amd64

Gateway conf:

{
    "listen_port": 8080,
    "secret": "%%TYK_GATEWAY_SECRET%%",
    "node_secret": "%%TYK_GATEWAY_SECRET%%",
    "template_path": "/opt/tyk-gateway/templates",
    "tyk_js_path": "/opt/tyk-gateway/js/tyk.js",
    "middleware_path": "/opt/tyk-gateway/middleware",
    "use_db_app_configs": true,
    "db_app_conf_options": {
        "connection_string": "%%TYK_GATEWAY_DASHBOARD_URL%%",
        "node_is_segmented": false,
        "tags": []
    },
    "app_path": "/opt/tyk-gateway/apps/",
    "storage": {
        "type": "redis",
        "host": "%%TYK_GATEWAY_REDIS_HOST%%",
        "port": %%TYK_GATEWAY_REDIS_PORT%%,
        "username": "%%TYK_GATEWAY_REDIS_USERNAME%%",
        "password": "%%TYK_GATEWAY_REDIS_PASSWORD%%",
        "database": 0,
        "optimisation_max_idle": 100
    },
    "enable_analytics": true,
    "analytics_config": {
        "ignored_ips": []
    },
    "health_check": {
        "enable_health_checks": true,
        "health_check_value_timeouts": 60
    },
    "optimisations_use_async_session_write": true,
    "allow_master_keys": false,
    "policies": {
    	"policy_source": "service",
    	"policy_connection_string": "%%TYK_GATEWAY_DASHBOARD_URL%%"
    },
    "hash_keys": true,
    "close_connections": true
}

Dashboard conf:

{
    "listen_port": 3000,
    "tyk_api_config": {
        "Host": "%%TYK_DASHBOARD_TYK_API_HOST%%",
        "Port": "%%TYK_DASHBOARD_TYK_API_PORT%%",
        "Secret": "%%TYK_DASHBOARD_TYK_API_SECRET%%"
    },
    "shared_node_secret": "%%TYK_DASHBOARD_TYK_API_SECRET%%",
    "mongo_url": "%%TYK_DASHBOARD_MONGO_URL%%",
    "page_size": 10,
    "admin_secret": "%%TYK_DASHBOARD_ADMIN_SECRET%%",
    "redis_port": %%TYK_DASHBOARD_REDIS_PORT%%,
    "redis_host": "%%TYK_DASHBOARD_REDIS_HOST%%",
    "redis_password": "%%TYK_DASHBOARD_REDIS_PASSWORD%%",
    "force_api_defaults": false,
    "notify_on_change": true,
    "license_key": "%%TYK_DASHBOARD_LICENSE_KEY%%",
    "hash_keys": true,
    "email_backend": {
        "enable_email_notifications": %%TYK_DASHBOARD_ENABLE_EMAIL%%,
        "code": "mailgun",
        "settings": {
            "Domain": "mg.domain.com",
            "PrivateKey": "%%TYK_DASHBOARD_MAILGUN_KEY_SECRET%%",
            "PublicKey": "%%TYK_DASHBOARD_MAILGUN_KEY_PUBLIC%%"
        },
        "default_from_email": "[email protected]",
        "default_from_name": "Administrator"
    },
    "hide_listen_path": false,
    "use_sentry": false,
    "sentry_code": "",
    "sentry_js_code": "",
    "show_org_id": true,
    "enable_duplicate_slugs": true,
    "host_config" : {
        "override_hostname": "%%TYK_DASHBOARD_OVERRIDE_HOSTNAME%%",
        "disable_org_slug_prefix": true,
        "enable_host_names": false,
        "hostname": "",
        "portal_domains": {},
        "portal_root_path": "/portal"
    },
    "http_server_options": {
        "use_ssl": false,
        "certificates": [
            {}
        ]
    },
    "ui": {
        "login_page": {},
        "nav" : {},
        "uptime": {},
        "portal": {},
        "designer": {}
    },
    "home_dir": "/opt/tyk-dashboard"
}

Let me know if there are certain collections I can look at to troubleshoot this, checking for consistency etc.

1 Like

Will I just bite the bullet and reconfig our setup on tyk 2 instead of trying to run on the existing db?

Hi Donal,

I’ve got nothing new on this one - we did just release v2.1 which has a few policy fixes in it, but since this issue seems to be specifically related to the portal, you might need to ditch and upgrade :-/

The good news is you can export and import your API Definitions quite easily.

But you’ll need to re-do the portal / policies.

Sorry I haven’t got any better news…

M.

1 Like