Per Session Rate Limiting in CE Edition not working

I just want to check how rate limiting works and got this strange behaviour. I’m using the CE edition

Case 1. API level Global Rate limit - It is working as expected

Case 2. Per Key Per API Rate Limiting - In this case Rate limiting does not even work

Here is how my config looks like
I have a goplugin serving from file system which does Authentication and it sets SessionState to context after validation of key

Here is how my go plugin looks like

func getSession(key string) *user.SessionState {
	now := time.Now()
	extractorDeadline := time.Now().Add(time.Second * 5).Unix()
	return &user.SessionState{
		OrgID:               "default",
		DateCreated:         now,
		LastUpdated:         now.String(),
		IdExtractorDeadline: extractorDeadline,
		ApplyPolicyID:       "test-policy",
		ApplyPolicies:       []string{"test-policy"},
		MetaData: map[string]interface{}{
			"token": key,
		},
	}
}

func HttpCustomAuth(w http.ResponseWriter, r *http.Request) {
	key := r.Header.Get(authorizationHeader)
        // some validation
	ctx.SetSession(r, getSession(key), key, true)

}

Here is my test policy json file

{
	"test-policy": {
	  "access_rights": {
		"gop-1": {
		  "allowed_urls": [],
		  "api_id": "gop-1",
		  "api_name": "Tyk Test API (Plain) 2",
		  "versions": [
			  "Default",
			  ""
		  ]
		},
		"pln-1": {
			"allowed_urls": null,
			"api_id": "pln-1",
			"versions": [
			  "Default"
			],
			"limit": {
			  "rate": 3,
			  "per": 60
			}
		  }
	  },
	  "active": true,
	  "name": "test policy",
	  "rate": 5,
	  "per": 60,
	  "quota_max": -1,
	  "quota_renewal_rate": 3600,
	  "state": "active",
	  "org_id": "default",
	  "tags": []
	}
  }

Here is my api definition

{
    "name": "Tyk Test API (Go plugin)",
    "api_id": "gop-1",
    "org_id": "default",
    "allowed_ips": [],
    "enable_ip_whitelisting": false,
    "definition": {
        "location": "header",
        "key": "x-api-version"
    },
    "auth": {
        "auth_header_name": "Authorization"
    },
    "use_keyless": false,
    "use_go_plugin_auth": true,
    "disable_quota": true,
    "version_data": {
        "not_versioned": true,
        "versions": {
            "Default": {
                "name": "Default",
                "expires": "",
                "use_extended_paths": true,
                "extended_paths": {
                    "ignored": [],
                    "white_list": [],
                    "black_list": [],
                    "circuit_breakers": [
                        {
                            "path": "/*",
                            "method": "GET",
                            "threshold_percent": 0.1,
                            "samples": 5,
                            "return_to_service_after": 60,
                            "disable_half_open_state": false
                        }
                    ]
                }
            }
        }
    },
    "proxy": {
        "preserve_host_header": false,
        "listen_path": "/goplugin-tyk-api-test",
        "target_url": "http://httpstat.us",
        "strip_listen_path": true
    },
    "global_rate_limit": {
        "rate": 0,
        "per": 0
    },
    "custom_middleware": {
        "auth_check": {
          "name": "HttpCustomAuth",
          "path": "./middleware/go/httpCustomAuth.so",
          "require_session": false
        },
        "response": [],
        "driver": "goplugin",
        "id_extractor": {
            "extract_from": "header",
            "extract_with": "value",
            "extractor_config": {
              "header_name": "Authorization"
            }
        }
    },
    "enable_batch_request_support": false
}

After the below call

curl -i http://localhost:8080/goplugin-tyk-api-test/200  -H 'accept: application/json' -H'Authorization: test-13'

full key definition look like below
curl -i http://localhost:8080/tyk/keys/test-13 -H 'X-Tyk-Authorization: 352d20ee67be67f6340b4c0605b044b7'

{
  "last_check": 0,
  "allowance": 0,
  "rate": 5,
  "per": 60,
  "throttle_interval": 0,
  "throttle_retry_limit": 0,
  "max_query_depth": 0,
  "date_created": "2021-10-21T21:26:08.490534+05:30",
  "expires": 0,
  "quota_max": -1,
  "quota_renews": 0,
  "quota_remaining": 0,
  "quota_renewal_rate": 3600,
  "access_rights": {
    "gop-1": {
      "api_name": "Tyk Test API (Plain) 2",
      "api_id": "gop-1",
      "versions": [
        "Default",
        ""
      ],
      "allowed_urls": [],
      "restricted_types": null,
      "limit": {
        "rate": 5,
        "per": 60,
        "throttle_interval": 0,
        "throttle_retry_limit": 0,
        "max_query_depth": 0,
        "quota_max": -1,
        "quota_renews": 0,
        "quota_remaining": 0,
        "quota_renewal_rate": 3600
      },
      "field_access_rights": null,
      "allowance_scope": ""
    },
    "pln-1": {
      "api_name": "",
      "api_id": "pln-1",
      "versions": [
        "Default"
      ],
      "allowed_urls": null,
      "restricted_types": null,
      "limit": {
        "rate": 5,
        "per": 60,
        "throttle_interval": 0,
        "throttle_retry_limit": 0,
        "max_query_depth": 0,
        "quota_max": -1,
        "quota_renews": 0,
        "quota_remaining": 0,
        "quota_renewal_rate": 3600
      },
      "field_access_rights": null,
      "allowance_scope": ""
    }
  },
  "org_id": "default",
  "oauth_client_id": "",
  "oauth_keys": null,
  "certificate": "",
  "basic_auth_data": {
    "password": "",
    "hash_type": ""
  },
  "jwt_data": {
    "secret": ""
  },
  "hmac_enabled": false,
  "enable_http_signature_validation": false,
  "hmac_string": "",
  "rsa_certificate_id": "",
  "is_inactive": false,
  "apply_policy_id": "test-policy",
  "apply_policies": [
    "test-policy"
  ],
  "data_expires": 0,
  "monitor": {
    "trigger_limits": null
  },
  "enable_detail_recording": false,
  "enable_detailed_recording": false,
  "meta_data": {
    "token": "test-13"
  },
  "tags": [],
  "alias": "",
  "last_updated": "2021-10-21 21:26:08.490534 +0530 IST m=+876.514373917",
  "id_extractor_deadline": 1634831773,
  "session_lifetime": 0
}

The above configuration does not throw 429 at all. (ideally gateway should return )
I have spent quite a lot of time figuring out what is going wrong but wasn’t able to figure out.

I’m not sure if i’m setting sessiondata to ctx all the time after validation in middleware is causing the issue

Can someone please point me what is wrong here.

Hi @Yeshwanth.ONE, could you let us know the version of gateway you are on?

Hi @Olu, Sorry i missed adding version… its v3.2.1

Hi @Olu Any suggestion here?

Hi @Yeshwanth.ONE, thanks for sharing the version number. We are checking it and will get back to you.

Hi @Yeshwanth.ONE,

Would you post what the gateway logs when you load this plugin please? I suspect the plugin isn’t loading at all.

The log entry we are looking for contains the string ‘plugin was built with a different version

Cheers,
Pete

Hi @Pete I have used same go version to build tyk project and also the plugin. i.e
go version go1.12.17 darwin/amd64

I have few logs in the plugin as well and those are getting logged.

I have fetched value from redis as well below is the value in redis
below is the value against the key in redis get 'apikey-test1'

{\"last_check\":0,\"allowance\":0,\"rate\":0,\"per\":0,\"throttle_interval\":0,\"throttle_retry_limit\":0,\"max_query_depth\":0,\"date_created\":\"2021-10-26T13:42:05.838676+05:30\",\"expires\":0,\"quota_max\":0,\"quota_renews\":0,\"quota_remaining\":0,\"quota_renewal_rate\":0,\"access_rights\":null,\"org_id\":\"default\",\"oauth_client_id\":\"\",\"oauth_keys\":null,\"certificate\":\"\",\"basic_auth_data\":{\"password\":\"\",\"hash_type\":\"\"},\"jwt_data\":{\"secret\":\"\"},\"hmac_enabled\":false,\"enable_http_signature_validation\":false,\"hmac_string\":\"\",\"rsa_certificate_id\":\"\",\"is_inactive\":false,\"apply_policy_id\":\"test-policy\",\"apply_policies\":[\"test-policy\"],\"data_expires\":0,\"monitor\":{\"trigger_limits\":null},\"enable_detail_recording\":false,\"enable_detailed_recording\":false,\"meta_data\":{\"token\":\"test1\"},\"tags\":null,\"alias\":\"\",\"last_updated\":\"2021-10-26 13:42:05.838676 +0530 IST m=+113.498409271\",\"id_extractor_deadline\":1635235930,\"session_lifetime\":0}

surprisingly curl to keys
curl -i http://localhost:8080/tyk/keys/test1 -H 'X-Tyk-Authorization: 352d20ee67be67f6340b4c0605b044b7'

has different output that the one in redis
below is from keys curl

{"last_check":0,"allowance":0,"rate":5,"per":60,"throttle_interval":0,"throttle_retry_limit":0,"max_query_depth":0,"date_created":"2021-10-26T13:42:05.838676+05:30","expires":0,"quota_max":-1,"quota_renews":0,"quota_remaining":0,"quota_renewal_rate":3600,"access_rights":{"gop-1":{"api_name":"Tyk Test API (Plain) 2","api_id":"gop-1","versions":["Default",""],"allowed_urls":[],"restricted_types":null,"limit":{"rate":5,"per":60,"throttle_interval":0,"throttle_retry_limit":0,"max_query_depth":0,"quota_max":-1,"quota_renews":0,"quota_remaining":0,"quota_renewal_rate":3600},"field_access_rights":null,"allowance_scope":""},"pln-1":{"api_name":"","api_id":"pln-1","versions":["Default"],"allowed_urls":null,"restricted_types":null,"limit":{"rate":5,"per":60,"throttle_interval":0,"throttle_retry_limit":0,"max_query_depth":0,"quota_max":-1,"quota_renews":0,"quota_remaining":0,"quota_renewal_rate":3600},"field_access_rights":null,"allowance_scope":""}},"org_id":"default","oauth_client_id":"","oauth_keys":null,"certificate":"","basic_auth_data":{"password":"","hash_type":""},"jwt_data":{"secret":""},"hmac_enabled":false,"enable_http_signature_validation":false,"hmac_string":"","rsa_certificate_id":"","is_inactive":false,"apply_policy_id":"test-policy","apply_policies":["test-policy"],"data_expires":0,"monitor":{"trigger_limits":null},"enable_detail_recording":false,"enable_detailed_recording":false,"meta_data":{"token":"test1"},"tags":[],"alias":"","last_updated":"2021-10-26 13:42:05.838676 +0530 IST m=+113.498409271","id_extractor_deadline":1635235930,"session_lifetime":0}```

Hi @Yeshwanth.ONE ,

Ah you’re compiling from source, that makes all the difference!

I’m surprised that your plugin compiles. The signature for setsession in 3.2.2 ctx/ctx.go is

func SetSession(r *http.Request, s *user.SessionState, scheduleUpdate bool) {

But in the plugin you call it with 4 parameters.

ctx.SetSession(r, getSession(key), key, true)

That’s probably where the issue is but I’m not sure how to dig deeper.

Cheers,
Pete

@Pete i’m on v3.2.1. I have checkout to tag v3.2.1

@Pete
Link to code

Woops! Sorry about that! I’ll continue investigating.

1 Like

HI @Pete Any update on this?

Hi,

Sorry, no update yet. I’ll get back to you as soon as I can.

Cheers,
Pete

Hi @Yeshwanth.ONE ,

Sorry it’s been so long, but I’m afraid I’m struggling to recreate your setup. Could you post your full plugin code file as well as any go.mod you use and the build command for the plugin you are using?

Cheers,
Pete

Hi @Pete I have only one go mod
Below is the complete code

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io/ioutil"
	"net/http"
	"time"

	"github.com/sirupsen/logrus"

	"github.com/TykTechnologies/tyk/ctx"
	"github.com/TykTechnologies/tyk/log"
	"github.com/TykTechnologies/tyk/user"
)

const (
	authorizationHeader = "Authorization"
)

var (
	httpClient      = &http.Client{Timeout: time.Second * 10}
	logger          = log.Get()
	policyToApply   = "test-policy"
	policiesToApply = []string{policyToApply}
)

func init() {
	logger.WithFields(logrus.Fields{
		"prefix": "HttpCustomAuth",
	}).Infof("Intit test http go pligins")
}

func returnResponse(w http.ResponseWriter, errorMessage string, httpStatusCode int) {
	jsonData, err := json.Marshal(errorMessage)
	if err != nil {
		logger.WithFields(logrus.Fields{
			"prefix": "HttpCustomAuth",
		}).Errorf("Couldn't marshal ", errorMessage)
		httpStatusCode = http.StatusInternalServerError
		jsonData = []byte("Gateway midlleware error")
	}
	w.WriteHeader(httpStatusCode)
	w.Write(jsonData)
}

func getSession(key string) *user.SessionState {
	now := time.Now()
	extractorDeadline := time.Now().Add(time.Second * 5).Unix()
	return &user.SessionState{
		OrgID:               "default",
		DateCreated:         now,
		LastUpdated:         now.String(),
		IdExtractorDeadline: extractorDeadline,
		ApplyPolicyID:       policyToApply,
		ApplyPolicies:       policiesToApply,
		MetaData: map[string]interface{}{
			"token": key,
		},
	}
}

func callAuthSerive(key string) ([]byte, int, error) {
	postBody, _ := json.Marshal(map[string]string{
		"name":  "test",
		"email": key,
	})
	req, err := http.NewRequest(http.MethodPost, "https://postman-echo.com/post", bytes.NewBuffer(postBody))
	if err != nil {
		return nil, -1, fmt.Errorf("Got error %s", err.Error())
	}
	req.Header.Set("user-agent", "tyk golang middleware")
	req.Header.Add("Content-Type", "application/json")
	req.Header.Add(authorizationHeader, key)
	response, err := httpClient.Do(req)
	if err != nil {
		return nil, -1, fmt.Errorf("Got error %s", err.Error())
	}
	defer response.Body.Close()
	var statusCode = response.StatusCode
	body, err := ioutil.ReadAll(response.Body)
	return body, statusCode, nil
}

func validateKey(key string) (bool, string, int) {
	body, statusCode, err := callAuthSerive(key)
	if err != nil || statusCode == -1 {
		logger.WithFields(logrus.Fields{
			"prefix": "HttpCustomAuth",
		}).Errorf("An Error while calling auth service", err)
		return false, "Gateway midlleware error while calling auth", 500
	}
	logger.WithFields(logrus.Fields{
		"prefix": "HttpCustomAuth",
	}).Infof(string(body))
	return true, "", statusCode
}

func HttpCustomAuth(w http.ResponseWriter, r *http.Request) {
	key := r.Header.Get(authorizationHeader)
	r.Header.Set("X-CUSTOM-HEADER", "Test Custom Header")
	if key == "test" {
		returnResponse(w, "", http.StatusUnauthorized)
		return
	}

	valid, errorMessage, statusCode := validateKey(key)
	if !valid {
		returnResponse(w, errorMessage, statusCode)
		return
	}
	ctx.SetSession(r, getSession(key), key, true)

}

func main() {}

I have places below code in this middleware/go/test-http/ directory from project root

this is the command for building the plugin

go build -buildmode=plugin -o <PROJECT_ROOT>/middleware/go/httpCustomAuth.so

And i have specified this plugin in the api configuration as mentioned below

Hi @Pete / @Olu Any update here?

Hi @Yeshwanth.ONE,

I am waiting on some feedback. I know there is a current session bug with version 3.2.1 for go lang plugins. And it probably affects any of the 3.2.x versions. I don’t have much knowledge with using plugins, so I am asking a colleague to help check.

In the meantime could you also try on a previous version of gateway and let us know the result. Something like 3.1.2.

Hi,

I’ve had a review of the details you sent us and noticed you’re missing the field setting in your policy to enable per API policy limits, which means the policy limits may not be enforced as expected.

Can I ask that you modify your policy to include a “partitions” section, defining per_api within it. I’ve reproduced your example policy below and added the relevant section at the end of the policy.

{
	"test-policy": {
	  "access_rights": {
		"gop-1": {
		  "allowed_urls": [],
		  "api_id": "gop-1",
		  "api_name": "Tyk Test API (Plain) 2",
		  "versions": [
			  "Default",
			  ""
		  ]
		},
		"pln-1": {
			"allowed_urls": null,
			"api_id": "pln-1",
			"versions": [
			  "Default"
			],
			"limit": {
			  "rate": 3,
			  "per": 60
			}
		  }
	  },
	  "active": true,
	  "name": "test policy",
	  "rate": 5,
	  "per": 60,
	  "quota_max": -1,
	  "quota_renewal_rate": 3600,
	  "state": "active",
	  "org_id": "default",
	  "tags": [],
      "partitions": {
         "per_api": true
      }

	}
  }

Could you re-test your rate limiting afterwards and see if this causes the results you expect?

Thanks!

Best Regards,
Chris