Tyk upstream mTLS not working

Hi Buds,

I faced an issue while calling upstream API having mTLS which is self-signed by the way, similar in tyk I have disabled insecure everything used Tyk OSS as well but as last I stuck at “http: proxy error: remote error: tls: bad certificate” but if I use curl it works, so I move to write a go program and got the same error “remote error: tls: bad certificate” it means there was an issue with go not tyk after google I found a link helped me to understand how to make my go program worked and finally it worked, please have a look into below details:

This is working curl command

curl --key cert.key --cert cert.crt -v https://mtls.example-url.com/api -k

This is go broken program

package main

import (
	"crypto/tls"
	"crypto/x509"
	"fmt"
	"io/ioutil"
	"log"
	"net/http"
)

func main() {
	// Load the client certificate and private key
	cert, err := tls.LoadX509KeyPair("cert.crt", "cert.key")
	if err != nil {
		log.Fatalf("Failed to load client certificate and key: %v", err)
	}

	// Load the CA certificate
	caCert, err := ioutil.ReadFile("cert.crt")
	if err != nil {
		log.Fatalf("Failed to read CA certificate: %v", err)
	}

	// Create a CA certificate pool and add the CA certificate
	caCertPool := x509.NewCertPool()
	if ok := caCertPool.AppendCertsFromPEM(caCert); !ok {
		log.Fatalf("Failed to parse CA certificate")
	}

	// Create a TLS configuration with the client certificate and CA certificate pool
	tlsConfig := &tls.Config{
		Certificates:       []tls.Certificate{cert},
		RootCAs:            caCertPool,
		InsecureSkipVerify: true, // Skip verification for self-signed certs
	}

	// Create an HTTP client with the custom TLS configuration
	client := &http.Client{
		Transport: &http.Transport{
			TLSClientConfig: tlsConfig,
		},
	}

	// Make a request to the API
	resp, err := client.Get("https://mtls.example-url.com/api")
	if err != nil {
		log.Fatalf("Failed to make request: %v", err)
	}
	defer resp.Body.Close()

	// Read and print the response body
	body, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		log.Fatalf("Failed to read response body: %v", err)
	}
	fmt.Println(string(body))
}

Output of this

Failed to make request: Get "https://mtls.example-url.com/api": remote error: tls: bad certificate

This is working go program by adding GetClientCertificate function

package main

import (
	"crypto/tls"
	"crypto/x509"
	"fmt"
	"io/ioutil"
	"log"
	"net/http"
)

func main() {
	// Load the client certificate and private key
	cert, err := tls.LoadX509KeyPair("cert.crt", "cert.key")
	if err != nil {
		log.Fatalf("Failed to load client certificate and key: %v", err)
	}

	// Load the CA certificate
	caCert, err := ioutil.ReadFile("cert.crt")
	if err != nil {
		log.Fatalf("Failed to read CA certificate: %v", err)
	}

	// Create a CA certificate pool and add the CA certificate
	caCertPool := x509.NewCertPool()
	if ok := caCertPool.AppendCertsFromPEM(caCert); !ok {
		log.Fatalf("Failed to parse CA certificate")
	}

	// Create a TLS configuration with the client certificate and CA certificate pool
	tlsConfig := &tls.Config{
		Certificates:       []tls.Certificate{cert},
		RootCAs:            caCertPool,
		InsecureSkipVerify: true, // Skip verification for self-signed certs
		GetClientCertificate: func(info *tls.CertificateRequestInfo) (*tls.Certificate, error) {
			return &cert, nil
		},
	}

	// Create an HTTP client with the custom TLS configuration
	client := &http.Client{
		Transport: &http.Transport{
			TLSClientConfig: tlsConfig,
		},
	}

	// Make a request to the API
	resp, err := client.Get("https://mtls.example-url.com/api")
	if err != nil {
		log.Fatalf("Failed to make request: %v", err)
	}
	defer resp.Body.Close()

	// Read and print the response body
	body, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		log.Fatalf("Failed to read response body: %v", err)
	}
	fmt.Println(string(body))
}

Output was the response from the API :tada:

Now can you please let me know how to make this work same for tyk as well ?
We are middle of nowhere now :sob:

Warm Regards,
Shebby

Hi! Can you set logging to debug level and attach here the log? TYK_LOGLEVEL=debug
You should at least see Found upstream mutual TLS certificate log.

Also show me how exactly you disable SSL verification in Tyk.
You can do it 2 ways for this task:

  • proxy.transport.ssl_insecure_skip_verify (if working with classical API format, and seems like there is no similar option in OAS format yet)
  • By setting it in tyk.conf or via env var: “proxy_ssl_insecure_skip_verify”:true or TYK_GW_PROXYSSLINSECURESKIPVERIFY=true

I guess last option should solve your issue.

Yes I have enabled debug already

2024-09-10 23:13:38 time="Sep 10 20:13:38" level=debug msg="Started proxy"
2024-09-10 23:13:38 time="Sep 10 20:13:38" level=debug msg="Stripping proxy listen path: /test-mtls/"
2024-09-10 23:13:38 time="Sep 10 20:13:38" level=debug msg="Upstream path is: /"
2024-09-10 23:13:38 time="Sep 10 20:13:38" level=debug msg=Started api_id=upstream-mtls api_name=Test-MTLS mw=ReverseProxy org_id=default ts=1725999218961601961
2024-09-10 23:13:38 time="Sep 10 20:13:38" level=debug msg="Upstream request URL: /" api_id=upstream-mtls api_name=Test-MTLS mw=ReverseProxy org_id=default
2024-09-10 23:13:38 time="Sep 10 20:13:38" level=debug msg="Outbound request URL: https://api-gateway-mtls.apps.ocpuat.example.com/identity/api/v1/user/login" api_id=upstream-mtls api_name=Test-MTLS mw=ReverseProxy org_id=default
2024-09-10 23:13:38 time="Sep 10 20:13:38" level=debug msg="Found upstream mutual TLS certificate" api_id=upstream-mtls api_name=Test-MTLS mw=ReverseProxy org_id=default
2024-09-10 23:13:38 time="Sep 10 20:13:38" level=debug msg="Creating new transport" api_id=upstream-mtls api_name=Test-MTLS mw=ReverseProxy org_id=default
2024-09-10 23:13:38 time="Sep 10 20:13:38" level=debug msg="Setting timeout for outbound request to: 30"
2024-09-10 23:13:38 time="Sep 10 20:13:38" level=debug msg="Out request url: https://[api-gateway-mtls.apps.ocpuat.example.com](http://api-gateway-mtls.apps.ocpuat.example.com)/identity/api/v1/user/login" api_id=upstream-mtls api_name=Test-MTLS mw=ReverseProxy org_id=default
2024-09-10 23:13:39 time="Sep 10 20:13:39" level=error msg="http: proxy error: remote error: tls: bad certificate" api_id=upstream-mtls api_name=Test-MTLS mw=ReverseProxy org_id=default prefix=proxy server_name="localhost:8080" user_id=-- user_ip=192.168.65.1 user_name=
2024-09-10 23:13:39 time="Sep 10 20:13:39" level=debug msg=Finished api_id=upstream-mtls api_name=Test-MTLS mw=ReverseProxy ns=98199416 org_id=default
2024-09-10 23:13:39 time="Sep 10 20:13:39" level=debug msg="Upstream request took (ms): 98.325625"

And this is my API def using Tyk Gateway OSS

{
    "name": "Test-MTLS",
    "api_id": "upstream-mtls",
    "org_id": "default",
    "definition": {
        "location": "header",
        "key": "version"
    },
    "use_keyless": true,
    "version_data": {
        "not_versioned": true,
        "versions": {
            "Default": {
                "name": "Default"
            }
        }
    },
    "use_mutual_tls_auth": false,
    "client_certificates": [],
    "upstream_certificates": {
        "*": "defaultb1f810a4d84882844023c7d5a93cb0852de75f46d2aafb6d307bec46a58b29d3"
    },
    "proxy": {
        "preserve_host_header": true,
        "listen_path": "/test-mtls/",
        "target_url": "https://api-gateway-mtls.apps.ocpuat.example.com/identity/api/v1/user/login",
        "disable_strip_slash": true,
        "strip_listen_path": true,
        "enable_load_balancing": false,
        "target_list": [],
        "check_host_against_uptime_tests": false,
        "transport": {
            "ssl_insecure_skip_verify": true,
            "ssl_ciphers": [],
            "ssl_min_version": 0,
            "ssl_max_version": 0,
            "ssl_force_common_name_check": false,
            "proxy_url": ""
        }
    }
}

this is my tyk.conf

{
  "log_level": "debug" ,
  "listen_port": 8080,
  "secret": "352d20ee67be67f6340b4c0605b044b7",
  "template_path": "/opt/tyk-gateway/templates",
  "tyk_js_path": "/opt/tyk-gateway/js/tyk.js",
  "middleware_path": "/opt/tyk-gateway/middleware",
  "use_db_app_configs": false,
  "app_path": "/opt/tyk-gateway/apps/",
  "storage": {
    "type": "redis",
    "host": "tyk-redis",
    "port": 6379,
    "username": "",
    "password": "",
    "database": 0,
    "optimisation_max_idle": 2000,
    "optimisation_max_active": 4000
  },
  "enable_analytics": false,
  "analytics_config": {
    "type": "",
    "ignored_ips": []
  },
  "health_check": {
    "enable_health_checks": false,
    "health_check_value_timeouts": 60
  },
  "enable_non_transactional_rate_limiter": true,
  "enable_sentinel_rate_limiter": false,
  "enable_redis_rolling_limiter": false,
  "allow_master_keys": false,
  "policies": {
    "policy_source": "file",
    "policy_path": "/opt/tyk-gateway/policies"
  },
  "hash_keys": true,
  "close_connections": false,
  "http_server_options": {
    "enable_websockets": true
  },
  "allow_insecure_configs": true,
  "coprocess_options": {
    "enable_coprocess": true,
    "coprocess_grpc_server": ""
  },
  "enable_bundle_downloader": true,
  "bundle_base_url": "",
  "global_session_lifetime": 100,
  "force_global_session_lifetime": false,
  "max_idle_connections_per_host": 500,
  "enable_jsvm": true,
  "proxy_ssl_insecure_skip_verify": true
}

tried with true and false both not worked, can you please let me know if I want to use go post plugin and update TLSConfig how to do that

tlsConfig := &tls.Config{
	Certificates:       []tls.Certificate{cert},
	RootCAs:            caCertPool,
	InsecureSkipVerify: true, // Skip verification for self-signed certs
	GetClientCertificate: func(info *tls.CertificateRequestInfo) (*tls.Certificate, error) {
		return &cert, nil
	},
}

I need to modify tls.Config{} by adding GetClientCertificate func in the plugin then test in tyk

To be frank I’m not sure how GetClientCertificate helps here, you are just hiding the problem.
By adding GetClientCertificate to tls.Config, you essentially ignore any client certificates from the upstream at all, and just replace them, with what specifified in your cert object. Basically you just make it “skip” the mutual tls part, it does not solve the root cause. :thinking:

We found the cause it was because we need to add chain in the certificate pem to upload in the dashboard as upstream trust it.

Actually we have an another certificate in the dashboard which was working and it made us confused that our certificate was not working in Tyk and just working with curl/postman this was very weird for us, so now we also created new certificate and ask upstream to trust that one and it is working fine in tyk as well.

Thanks Leon :hand_with_index_finger_and_thumb_crossed:

Great! Happy to help!

1 Like