Upstream Certificate Manual does not work

we would like to use the upstream mtls function in an api but unfortunately this feature does not work for us. We have created the certifacts exactly according to the following instructions: https://github.com/TykTechnologies/tyk-operator/blob/master/config/samples/httpbin_upstream_cert_manual.yaml. We have set the other settings according to the documentation (Upstream mTLS).

According to the logs, the problem is that Tyk cannot read the certificate. Decryption password incorrect also appears in the logs, although neither the certificate nor the private key are encrypted (were created with the openssl command from the tyk operator doku).

The stored certificate ID is definitely correct. We can find the certificate ID from the logs exactly the same in the dashboard. Don’t understand why it can’t be found in cert_storage.

time=“Jul 13 14:37:45” level=debug msg=“Started proxy”
time=“Jul 13 14:37:45” level=debug msg=“Stripping: /api/upstream-test/”
time=“Jul 13 14:37:45” level=debug msg=“Upstream Path is: "
time=“Jul 13 14:37:45” level=debug msg=Started api_id=b3e390902b534f486f61dbd5974acdff api_name=upstream-int-test mw=ReverseProxy org_id=60880e360ec4af00010da05c ts=1657723065917020791
time=“Jul 13 14:37:45” level=debug msg=“Upstream request URL: " api_id=b3e390902b534f486f61dbd5974acdff api_name=upstream-int-test mw=ReverseProxy org_id=60880e360ec4af00010da05c
time=“Jul 13 14:37:45” level=debug msg=“Outbound request URL: ***.com/api/upstream” api_id=b3e390902b534f486f61dbd5974acdff api_name=upstream-int-test mw=ReverseProxy org_id=60880e360ec4af00010da05c
time=“Jul 13 14:37:45” level=debug msg=“Input key was: cert-raw-60880e360ec4af00010da05cfd765e7bd178e0b3ad507402245718ee9a979b2c79570f4245d2571d49df6fab”
time=“Jul 13 14:37:45” level=debug msg=“Using cache for: cert-raw-60880e360ec4af00010da05cfd765e7bd178e0b3ad507402245718ee9a979b2c79570f4245d2571d49df6fab”
time=“Jul 13 14:37:45” level=debug msg=”–> Found? false”
time=“Jul 13 14:37:45” level=debug msg=“Error trying to get value:x509: decryption password incorrect”
time=“Jul 13 14:37:45” level=debug msg=“GetKey took 2.42377ms”
time=“Jul 13 14:37:45” level=warning msg=“Can’t retrieve certificate: 60880e360ec4af00010da05cfd765e7bd178e0b3ad507402245718ee9a979b2c79570f4245d2571d49df6fabopen 60880e360ec4af00010da05cfd765e7bd178e0b3ad507402245718ee9a979b2c79570f4245d2571d49df6fab: no such file or directory” prefix=“cert_storage”
time=“Jul 13 14:37:46” level=debug msg=Finished api_id=b3e390902b534f486f61dbd5974acdff api_name=debit-online-int-test mw=ReverseProxy ns=499742081 org_id=60880e360ec4af00010da05c
time=“Jul 13 14:37:46” level=debug msg=“Upstream request took (ms): 499.813135”
time=“Jul 13 14:37:46” level=debug msg=“Checking: 60880e360ec4af00010da05c” api_id=b3e390902b534f486f61dbd5974acdff api_name=debit-online-int-test org_id=60880e360ec4af00010da05c
time=“Jul 13 14:37:46” level=debug msg=“Input key was: orgkey.eyJvcmciOiI2MDg4MGUzNjBlYzRhZjAwMDEwZGEwNWMiLCJpZCI6Ijc4OTkyZTVlZWFjOTQ0OGE4NWRlOGNiOTQ4ZjU3MzNmIiwiaCI6Im11cm11cjEyOCJ9”
time=“Jul 13 14:37:46” level=debug msg=“Using cache for: orgkey.eyJvcmciOiI2MDg4MGUzNjBlYzRhZjAwMDEwZGEwNWMiLCJpZCI6Ijc4OTkyZTVlZWFjOTQ0OGE4NWRlOGNiOTQ4ZjU3MzNmIiwiaCI6Im11cm11cjEyOCJ9”
time=“Jul 13 14:37:46” level=debug msg=“–> Found? false”
time=“Jul 13 14:37:46” level=debug msg=“Error trying to get value:Key not found”
time=“Jul 13 14:37:46” level=debug msg=“GetKey took 99.902736ms”

Hi @umxm and welcome to the community.

Are you using Operator v0.9.0 and above? Upstream mTLs was introduced from that point.

If yes, then can you check your gateway security configuration for the value of private_certificate_encoding_secret?

The thing was that certs used for upstream have private keys. Certs with private keys get encrypted
The encryption key is either the setting in private_certificate_encoding_secret or the gateway secret if private_certificate_encoding_secret is not defined

As an addition, can you confirm if your hashing algorithm may have changed recently?