In general I the the issue here descibed in this thread encryption - ssh-keygen and openssl gives two different public keys - Stack Overflow
And basic idea that openssl and ssh-kegen use different encodings for keys.
You can convert from one to another ssh - Converting keys between openssl and openssh - Information Security Stack Exchange
To convert from ssh .pub to openssl .pem format, you can run: ssh-keygen -f rsa.pub -e -m pem, and then use resulting key with Tyk.
Hope it helps!