HMAC Signatures

MrImport · January 19, 2016, 9:05pm

Imported Google Group message. Original thread at: Redirecting to Google Groups Import Date: 2016-01-19 21:05:48 +0000.
Sender:Matthieu Nantern.
Date:Monday, 9 February 2015 12:56:42 UTC.

Hi !

I am currently evaluating Tyk and have some issue with HMAC signatures. Maybe you can help me find what’s wrong:

My API is defined like that:

{
“name”: “TEST_API”,
“api_id”: “d29d5f9e-1ad3-43fa-9dc6-604c40ce78d1”,
“org_id”: “test_api”,
“enable_signature_checking”: true,
“use_basic_auth”: false,
“use_keyless”: false,
“use_oauth2”: false,
“auth”: {
“auth_header_name”: “”
},
“version_data”: {
“not_versioned”: true,
“versions”: {
“Default”: {
“name”: “Default”,
“expires”: “3000-01-02 15:04”,
“use_extended_paths”: true,
“extended_paths”: {
“ignored”: ,
“white_list”: ,
“black_list”:
}
}
}
},
“proxy”: {
“listen_path”: “/api/”,
“target_url”: “http://localhost:8080/my_api/”,
“strip_listen_path”: true
},
“enable_batch_request_support”: false
}

Then I add a key:

curl --request POST -d @create.json -H “X-Tyk-Authorization: 352d20ee67be67f6340b4c0605b044b7” http://<my_ip>/tyk/keys/create

with create.json:
{
“allowance”: 999,
“rate”: 1000,
“per”: 60,
“expires”: 0,
“quota_max”: -1,
“quota_renews”: 1406121006,
“quota_remaining”: 0,
“quota_renewal_rate”: 60,
“org_id”: “posc_ndr”,
“hmac_enabled”: true,
“hmac_string”: “test_secret”
}

And the result:
{“key”:“test_api090d6c921dde4b557819c660853166f0”,“status”:“ok”,“action”:“create”}

So far, so good (I think).
And finally I want to call my API with curl, so I made that small bash script:

#!/bin/bash

date=“$(LC_ALL=C date -u +”%a, %d %b %Y %X %Z")"
ENCODED=echo -n "date:${date}" | openssl sha1 -binary -hmac "test_secret"|base64|sed -r 's/=/%3D/g'

echo $date
echo $ENCODED

curl -v --include -H “Date: ${date}” -H “Authorization: Signature keyId="test_api090d6c921dde4b557819c660853166f0",algorithm="hmac-sha1",signature="${ENCODED}"” ‘http://localhost:8081/api/v1/ndr?count=1&idDevice=7b1afb61-b23a-4d42-95e0-5432f9592c6e’

I’m not really sure how to calculate the signature.

And it’s always the same output:

Mon, 09 Feb 2015 12:50:46 UTC
nJLi0CeZugso89UpgE6lF14syQQ%3D

About to connect() to localhost port 8081 (#0)
Trying ::1… connected
Connected to localhost (::1) port 8081 (#0)

GET /api/v1/ndr?count=1&idDevice=7b1afb61-b23a-4d42-95e0-5432f9592c6e HTTP/1.1
User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Host: localhost:8081
Accept: /
Date: Mon, 09 Feb 2015 12:50:46 UTC
Authorization: Signature keyId=“test_api090d6c921dde4b557819c660853166f0”,algorithm=“hmac-sha1”,signature=“nJLi0CeZugso89UpgE6lF14syQQ%3D”

< HTTP/1.1 400 Bad Request
HTTP/1.1 400 Bad Request
< Content-Type: application/json
Content-Type: application/json
< X-Generator: tyk.io
X-Generator: tyk.io
< Date: Mon, 09 Feb 2015 12:50:46 GMT
Date: Mon, 09 Feb 2015 12:50:46 GMT
< Content-Length: 47
Content-Length: 47

<
{
“error”: “Request signature is invalid”

Connection #0 to host localhost left intact
Closing connection #0

And server-side:

INFO[0221] Request Signature: nJLi0CeZugso89UpgE6lF14syQQ=
INFO[0221] Should be: hMV2SFKzb++RS0fjRjhuehKlOjc=
INFO[0221] Request signature is invalid origin=[::1]:46611 path=/api/v1/ndr

I’m using the last version of Tyk.

Any idea ?

Thank you !

MrImport · January 19, 2016, 9:05pm

Imported Google Group message.
Sender:Martin Buhr.
Date:Thursday, 12 February 2015 15:43:16 UTC.

Haha! Excellent!

show quoted text -

MrImport · January 19, 2016, 9:05pm

Imported Google Group message.
Sender:Martin Buhr.
Date:Monday, 9 February 2015 14:41:42 UTC.

Hi,

HMAC is always tricky, so many steps to get exactly right… I just looked over the test script for this feature, creating the key is in this section, it looks like (not sure if your script does this, my command-line-fu is a little rusty, but basically:

Format the date like so: “Mon, 02 Jan 2006 15:04:05 MST”
Add this date as the Date Header
Create the signature string, this is the header name, all lowercase, followed by a colon, followed by the above date url-encoded: “date:Mon%2C+02+Jan+2006+15%3A04%3A05+MST” (I think this is the bit you are missing)
SHA-1 encode this signature string using your secret
Now Base64-encode your signed string
URL-Encode this string (because it’s a block cipher, it pads with “=” signs, this can break the decoder (I don’t think your code does this bit either)
Now add the key to the structured Auth header, it should be formatted like this: “Signature keyId=”",algorithm=“hmac-sha1”,signature="<YOUR URLENCODED, Base64 encoded signature string>"" (Note that those quotation marks are required) (You have this covered)

Good luck

Cheers,
Martin

show quoted text -

MrImport · January 19, 2016, 9:05pm

Imported Google Group message.
Sender:Martin Buhr.
Date:Monday, 9 February 2015 15:16:19 UTC.

You were right I missed url encode of step 3.

But still no luck

This is what I have with secret “test_secret”:
date: Mon, 09 Feb 2015 15:13:19 UTC
encoded date: Mon%2C%2009%20Feb%202015%2015%3A13%3A19%20UTC
signature: WqFjg2+C/rH120jz+2KBY+cLs7Q=
url_encoded_signature: WqFjg2%2BC%2FrH120jz%2B2KBY%2BcLs7Q%3D

Can you tell me where it’s different for you ?

Last version of the bash script (with urlencode)

#!/bin/bash

function urlencode() {
echo -n “$1” | perl -MURI::Escape -ne ‘print uri_escape($_)’

}

date=“$(LC_ALL=C date -u +”%a, %d %b %Y %X %Z")"

encoded_date=urlencode "${date}"
signature=echo -n "date:${encoded_date}" | openssl sha1 -binary -hmac "test_secret"|base64
url_encoded_signature=urlencode "${signature}"

echo “date: $date”
echo “encoded date: $encoded_date”
echo “signature: $signature”
echo “url_encoded_signature: $url_encoded_signature”

curl -v --include -H “Date: ${date}” -H “Authorization: Signature keyId="e27ec93e6a5949d170f1bf26ca2671bd",algorithm="hmac-sha1",signature="${url_encoded_signature}"” ‘http://localhost:8081/api/v1/n
dr?count=1&idDevice=7b1afb61-b23a-4d42-95e0-5432f9592c6e’

And response from the server:

INFO[0845] Request Signature: WqFjg2+C/rH120jz+2KBY+cLs7Q=
INFO[0845] Should be: /eZspME8uCUaaQjrX6vujvVNWx8=
INFO[0845] Request signature is invalid origin=[::1]:46724 path=/api/v1/ndr

Thanks for your help !

show quoted text -

MrImport · January 19, 2016, 9:05pm

Imported Google Group message.
Sender:Matthieu Nantern.
Date:Monday, 9 February 2015 16:25:53 UTC.

Hi,

I just tried replicating the golang encoding that the source code uses and your bash script for generating the signature they provide the same output:

Golang Playground: Go Playground - The Go Programming Language (click run)

bash output:

martin@vyr ~> echo -n “date:Mon%2C%2009%20Feb%202015%2015%3A13%3A19%20UTC” | openssl sha1 -binary -hmac “test_secret” | base64
WqFjg2+C/rH120jz+2KBY+cLs7Q=

So we know your code is correct. Checking the generating code in the middleware shows that we’re using the same mechanisms in the middleware as we are in the golang playground script: https://github.com/lonelycode/tyk/blob/master/middleware_check_HMAC_signature.go#L221

I can see in the above two outputs that the keys are different and the second key doesn;t seem to have an org ID, keys that are created in tyk take the form of OrgID + APIID, although if tyk couldn’t find the key, the request would fail much earlier

A few things to check:

Does the key have the correct secret set? (you can pull the key data out of tyk with a GET to /tyk/keys/{keyId}??api_id={api-id}
Is the date header being set correctlky? Or twice? Tyk pulls the comparison header from the request, if the headers are different the calculation would fail, maybe use the --verbose flag to see what gets sent?

will get to the bottom of it, the tests are passing, so I really am confused as to what’s wrong…

Let me know if any of the above helps

Martin

show quoted text -

MrImport · January 19, 2016, 9:05pm

Imported Google Group message.
Sender:Martin Buhr.
Date:Wednesday, 11 February 2015 09:22:30 UTC.

Ok, it’s strange, I can’t make it work:

My app (with an empty org_id):

{
“name”: “NDR_API”,
“api_id”: “d29d5f9e-1ad3-43fa-9dc6-604c40ce78d1”,
“org_id”: “”,

"use_basic_auth": false,
"use_keyless": false,
"use_oauth2": false,

"enable_signature_checking": true,

"auth": {
    "auth_header_name": ""
},
"version_data": {
    "not_versioned": true,
    "versions": {
        "Default": {
            "name": "Default",
            "expires": "3000-01-02 15:04",
            "use_extended_paths": true,
            "extended_paths": {
                "ignored": [],
                "white_list": [],
                "black_list": []
            }
        }
    }
},
"proxy": {
    "listen_path": "/api/",

    "target_url": "http://localhost:8080/",

    "strip_listen_path": true
},
"enable_batch_request_support": false

}

Then I created a new key:

curl --request POST -d @create.json -H “X-Tyk-Authorization: 352d20ee67be67f6340b4c0605b044b7” http://172.21.5.18:8081/tyk/keys/create

with create.json:
{
“allowance”: 4,
“rate”: 5,
“per”: 60,
“expires”: 0,
“quota_max”: 10,
“quota_renewal_rate”: 86400,
“org_id”: “”,
“hmac_enabled”: true,
“hmac_string”: “test_secret”
}

The response:

{“key”:“69011da4b26543d144c5a9ee4714fb6f”,“status”:“ok”,“action”:“create”}

And then I use my bash script:

./test_getNDR_HMAC.sh
date: Wed, 11 Feb 2015 08:47:06 UTC
encoded date: Wed%2C%2011%20Feb%202015%2008%3A47%3A06%20UTC
signature: SvSlA0Xefzb1+wnHJvXGFRsiuZw=
url_encoded_signature: SvSlA0Xefzb1%2BwnHJvXGFRsiuZw%3D

The result is the same in Golang Playground but tyk is failing:

INFO[0330] Request signature is invalid origin=[::1]:47297 path=/api/v1/ndr
INFO[0590] Request Signature: SvSlA0Xefzb1+wnHJvXGFRsiuZw=
INFO[0590] Should be: LfMMt/uSfXur0Igro5OXMymLPGM=

Key secret:

curl -H “X-Tyk-Authorization: 352d20ee67be67f6340b4c0605b044b7” http://172.21.5.18:8081/tyk/keys/69011da4b26543d144c5a9ee4714fb6f?api_id=d29d5f9e-1ad3-43fa-9dc6-604c40ce78d1
{“last_check”:0,“allowance”:4,“rate”:5,“per”:60,“expires”:0,“quota_max”:10,“quota_renews”:1423730519,“quota_remaining”:0,“quota_renewal_rate”:86400,“access_rights”:null,“org_id”:“”,“oauth_client_id”:“”,“basic_auth_data”:{“password”:“”},“hmac_enabled”:true,“hmac_string”:“MzBjODkwOTE1OTk1NDExYjUwNDNkOTk5N2FhMWQ4ZTY=”,“is_inactive”:false,“meta_data”:null}

Is hmac_string encoded ? Because it’s not “test_secret”.

Date header is not set twice:

About to connect() to localhost port 8081 (#0)
Trying ::1… connected
Connected to localhost (::1) port 8081 (#0)

GET /api/v1/ndr?count=1&idDevice=7b1afb61-b23a-4d42-95e0-5432f9592c6e HTTP/1.1
User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Host: localhost:8081
Accept: *
/*
Date: Wed, 11 Feb 2015 08:47:06 UTC
Authorization: Signature keyId=“69011da4b26543d144c5a9ee4714fb6f”,algorithm=“hmac-sha1”,signature=“SvSlA0Xefzb1%2BwnHJvXGFRsiuZw%3D”

< HTTP/1.1 400 Bad Request

Le lundi 9 février 2015 17:25:53 UTC+1, Martin Buhr a écrit :
Hi,

I just tried replicating the golang encoding that the source code uses and your bash script for generating the signature they provide the same output:

Golang Playground:
…

MrImport · January 19, 2016, 9:05pm

Imported Google Group message.
Sender:Matthieu Nantern.
Date:Wednesday, 11 February 2015 09:30:38 UTC.

Hi Matthieu,

Of course! Sorry, this is completely my fault, because I forgot that the /create method is all clever and generates an HMAC secret for the key it generates (to add a new key with your own settings you need to POST to /tyk/keys). have you tried using that key in your request?

Apologies, it completely slipped my mind :-/

Thanks,
Martin

show quoted text -

MrImport · January 19, 2016, 9:05pm

Imported Google Group message.
Sender:Martin Buhr.
Date:Wednesday, 11 February 2015 10:51:13 UTC.

Good to know ! But why /create doesn’t send back the hmac_string ?

{“key”:“69011da4b26543d144c5a9ee4714fb6f”,“status”:“ok”,“action”:“create”}

Anyway I replaced “test_secret” by the hmac_string from /tyk/keys (MzBjODkwOTE1OTk1NDExYjUwNDNkOTk5N2FhMWQ4ZTY=) but it’s not better:

INFO[0457] Request Signature: Z2+OGW1UlsXgla4zNVuQHcN55uQ=
INFO[0457] Should be: RN3T0msU8v5/g0P/X2tQ8yU3o0M=
INFO[0457] Request signature is invalid origin=[::1]:47344 path=/api/v1/ndr

show quoted text -

MrImport · January 19, 2016, 9:05pm

Imported Google Group message.
Sender:Matthieu Nantern.
Date:Wednesday, 11 February 2015 11:06:05 UTC.

Hmmm, very odd. Could you share your scripts with me? I’ll run some tests on my end…

Thanks,
Martin

show quoted text -

MrImport · January 19, 2016, 9:05pm

Imported Google Group message.
Sender:Martin Buhr.
Date:Wednesday, 11 February 2015 11:55:55 UTC.

Of course:

#!/bin/bash

function urlencode() {
echo -n “$1” | perl -MURI::Escape -ne ‘print uri_escape($_)’
}

date=“$(LC_ALL=C date -u +”%a, %d %b %Y %X %Z")"
encoded_date=urlencode "${date}"

signature=echo -n "date:${encoded_date}" | openssl sha1 -binary -hmac "MzBjODkwOTE1OTk1NDExYjUwNDNkOTk5N2FhMWQ4ZTY="|base64

url_encoded_signature=urlencode "${signature}"

echo “date: $date”
echo “encoded date: $encoded_date”
echo “signature: $signature”
echo “url_encoded_signature: $url_encoded_signature”

curl -v --include -H “Date: ${date}” -H “Authorization: Signature keyId="69011da4b26543d144c5a9ee4714fb6f",algorithm="hmac-sha1",signature="${url_encoded_signature}"” ‘http://localhost:8081/api/v1/ndr?count=1&idDevice=7b1afb61-b23a-4d42-95e0-5432f9592c6e’

show quoted text -

MrImport · January 19, 2016, 9:05pm

Imported Google Group message.
Sender:Matthieu Nantern.
Date:Wednesday, 11 February 2015 16:09:48 UTC.

Ok, I think I have figured it out…

When generating the HMAC signature string, we use the date header, which is:

Wed, 11 Feb 2015 12:35:08 UTC

Our encoder turns this into:
date:Wed%2C+11+Feb+2015+12%3A35%3A08+UTC

Your encoder creates:
date:Wed%2C%2011%20Feb%202015%2012%3A35%3A08%20UTC

I am not sure which is the correct implementation, as they both decode correctly, but, since the signature is generated from this value, they will never match.

The difference is the spaces, they are encoded as ‘+’ in ours and ‘%20’ in yours.

:-/

Will need ot make a note of that in the docs as that could really piss people off…

Cheers,
Martin

show quoted text -

MrImport · January 19, 2016, 9:05pm

Imported Google Group message.
Sender:Martin Buhr.
Date:Thursday, 12 February 2015 15:09:10 UTC.

I use URI::Escape to encode my url and it changes space in %20.

If I replace %20 by + it’s working !

The final script:

#!/bin/bash

function urlencode() {
echo -n “$1” | perl -MURI::Escape -ne ‘print uri_escape($_)’ | sed “s/%20/+/g”

}

date=“$(LC_ALL=C date -u +”%a, %d %b %Y %X %Z")"
encoded_date=urlencode "${date}"
signature=echo -n "date:${encoded_date}" | openssl sha1 -binary -hmac "MzBjODkwOTE1OTk1NDExYjUwNDNkOTk5N2FhMWQ4ZTY="|base64
url_encoded_signature=urlencode "${signature}"

echo “date: $date”
echo “encoded date: $encoded_date”
echo “signature: $signature”
echo “url_encoded_signature: $url_encoded_signature”

curl -H “Date: ${date}” -H “Authorization: Signature keyId="69011da4b26543d144c5a9ee4714fb6f",algorithm="hmac-sha1",signature="${url_encoded_signature}"” ‘http://localhost:8081/api/v1/ndr?count=1&id
Device=7b1afb61-b23a-4d42-95e0-5432f9592c6e’

show quoted text -

…

MrImport · January 19, 2016, 9:05pm

Imported Google Group message.
Sender:Matthieu Nantern.
Date:Thursday, 12 February 2015 15:43:16 UTC.

Haha! Excellent!

show quoted text -

jbarro · July 4, 2016, 10:28am

Hi, we are testing the hmac access control feature on a test api. We have followed this post but we can not make it to work. Launching the script… the result is:

date: Mon, 04 Jul 2016 10:09:24 UTC
encoded date: Mon%2C+04+Jul+2016+10%3A09%3A24+UTC
signature: et9JytS568R7P+Lnl92qJXDf2Lo=
url_encoded_signature: et9JytS568R7P%2BLnl92qJXDf2Lo%3D
{
“error”: “Authorization field missing, malformed or invalid”
}

Checking the logs (on debug mode)… we have this error:

time=“Jun 29 13:16:48” level=debug msg=“keyId="5716713c06a24183b4000001732468f99fd0426356a53de1ac1f5bd3",algorithm="hmac-sha1",signature="MQZevOGKO7l1Tnt%2FlAXzl4lPtRU%3D"”
time=“Jun 29 13:16:48” level=debug msg=“Checking: [keyId "5716713c06a24183b4000001732468f99fd0426356a53de1ac1f5bd3"]”
time=“Jun 29 13:16:48” level=debug msg=“Checking: [algorithm "hmac-sha1"]”
time=“Jun 29 13:16:48” level=debug msg=“Checking: [signature "MQZevOGKO7l1Tnt%2FlAXzl4lPtRU%3D"]”
time=“Jun 29 13:16:48” level=debug msg=“Got date header”
time=“Jun 29 13:16:48” level=debug msg=“Generated sig string: date: Wed, 29 Jun 2016 11:16:48 UTC”
time=“Jun 29 13:16:48” level=debug msg=“[STORE] Getting WAS: 5716713c06a24183b4000001732468f99fd0426356a53de1ac1f5bd3”
time=“Jun 29 13:16:48” level=debug msg=“Input key was: apikey-5716713c06a24183b4000001732468f99fd0426356a53de1ac1f5bd3”
time=“Jun 29 13:16:48” level=debug msg=“[STORE] Getting: apikey-5716713c06a24183b4000001732468f99fd0426356a53de1ac1f5bd3”
time=“Jun 29 13:16:48” level=debug msg=“Input key was: apikey-5716713c06a24183b4000001732468f99fd0426356a53de1ac1f5bd3”
time=“Jun 29 13:16:48” level=error msg=“Signature string does not match!” expected=“6g%2BjeN4EiD%2FjvT4H9Pimd%2FuLnEk%3D” got=“MQZevOGKO7l1Tnt%2FlAXzl4lPtRU%3D”
time=“Jun 29 13:16:48” level=info msg=“Authorization field missing or malformed” origin=“x.x.x.x:x” path=“/api/d/”
time=“Jun 29 13:16:48” level=debug msg=“Returning error header”
time=“Jun 29 13:16:48” level=debug msg=“Pushing to raw key list: tyk-system-analytics”
time=“Jun 29 13:16:48” level=debug msg=“Input key was: analytics-tyk-system-analytics”
time=“Jun 29 13:16:48” level=debug msg=“Appending to fixed key list: analytics-tyk-system-analytics”
time=“Jun 29 13:16:48” level=debug msg=“Input key was: analytics-tyk-system-analytics”
time=“Jun 29 13:16:48” level=debug msg=“Adding Healthcheck to: e3e570bdf18341575ab4ba3e861118bb.BlockedRequest”
time=“Jun 29 13:16:48” level=debug msg=“Val is: -1”
time=“Jun 29 13:16:48” level=debug msg=“Incrementing raw key: e3e570bdf18341575ab4ba3e861118bb.BlockedRequest”
time=“Jun 29 13:16:48” level=debug msg=“keyName is: e3e570bdf18341575ab4ba3e861118bb.BlockedRequest”
time=“Jun 29 13:16:48” level=debug msg=“Now is:2016-06-29 13:16:48.781951308 +0200 CEST”
time=“Jun 29 13:16:48” level=debug msg=“Then is: 2016-06-29 13:15:48.781951308 +0200 CEST”
time=“Jun 29 13:16:48” level=debug msg=“Returned: 0”

Note that the output from the script and the log data are on different time windows…

With this data we know the signature expected and changing it manually on the script… all works fine…

This is the script we are using:

datenow=$(LC_ALL=C date -u +“%a, %d %b %Y %X %Z”)
encodeddate=$(echo -n “$datenow” | perl -MURI::Escape -ne ‘print uri_escape($_)’ | sed “s/%20/+/g”)

signature=$(echo -n “date:${encodeddate}” | openssl sha1 -binary -hmac “ZWQ1N2E1NDkzZTgyNGMyMjRmOGVkOWQ5MmVjM2EzMGI=” | base64)

url_encoded_signature=$(echo -n “${signature}” | perl -MURI::Escape -ne ‘print uri_escape($_)’ | sed “s/%20/+/g”)

echo “date: $datenow”
echo “encoded date: $encodeddate”
echo “signature: $signature”
echo “url_encoded_signature: $url_encoded_signature”
curl -H “Date: ${datenow}” -H “Authorization: Signature keyId="5716713c06a24183b4000001732468f99fd0426356a53de1ac1f5bd3",algorithm="hmac-sha1",signature="${url_encoded_signature}"” ‘http://x.x.x:xxxx/api/d/’

And this is the apikey hmac secret configuration:

Hope you can help us…

Thanks a lot…

Martin · July 4, 2016, 2:41pm

This is a pretty old thread - take a look at the tests:

https://github.com/TykTechnologies/tyk/blob/master/middleware_check_HMAC_test.go#L118

And for a more complex signature string example:

https://github.com/TykTechnologies/tyk/blob/master/middleware_check_HMAC_test.go#L388

jbarro · July 5, 2016, 11:25am

Ok… It´s working know… thanks a lot!

Regards…