Golang plugins not working with bundle downloader

Dirk · January 4, 2024, 4:11pm

Branch/Environment/Version

Branch/Version: Release v5.2.5
Environment: On-prem (OSS)

Describe the bug
We try to register a Golang plugin, using enable_bundle_downloader, as described here.
Unfortunately, this does not work, as the driver information in the manifest.json is not considered at all, and it is interpreted as a coprocess plugin.

Reproduction steps
Steps to reproduce the behaviour:

Configure Tyk to disable coprocess_options.enable_coprocess. plugin, but enable enable_bundle_downloader (with according URL)
Create, build and publish a simple Golang plugin (e.g. pre, just adding a static header)
Create an API with custom_middleware_bundle using the above bundle

Actual behaviour
The gateway starts up, but complains about us trying to register a CP plugin while this is disabled:

Your API specifies a CP custom middleware, either Tyk wasn't build with CP support or CP is not enabled in your Tyk configuration file!

The plugin is (not surprigingly) not working as a result.

When getting the API definition via the gateway API, the custom_middleware block contains an empty string for the driver key, while in the manifest.json, it is set to goplugin.

Expected behaviour
The goplugin driver entry from the manifest will be respected, and the plugin should be registered & working.

Screenshots/Video

not applicable

Logs (debug mode or log file):

time="Jan 04 15:06:36" level=debug msg="----> Saving Bundle: TestPlugin.zip" prefix=main
time="Jan 04 15:06:36" level=info msg="----> Loading bundle: TestPlugin.zip" prefix=main
time="Jan 04 15:06:36" level=info msg="----> Verifying bundle: TestPlugin.zip" prefix=main
time="Jan 04 15:06:36" level=info msg="----> Bundle verification failed: TestPlugin.zip" prefix=main
time="Jan 04 15:06:36" level=info msg="----> Bundle is valid, adding to spec: TestPlugin.zip" prefix=main
time="Jan 04 15:06:36" level=info msg="Detected 1 APIs" prefix=main
time="Jan 04 15:06:36" level=info msg="Loading API configurations." prefix=main
time="Jan 04 15:06:36" level=info msg="Tracking hostname" api_name="Test backend" domain="(no host)" prefix=main
time="Jan 04 15:06:36" level=info msg="Initialising Tyk REST API Endpoints" prefix=main
time="Jan 04 15:06:36" level=debug msg="Loaded API Endpoints" prefix=main
time="Jan 04 15:06:36" level=info msg="API bind on custom port:0" prefix=main
time="Jan 04 15:06:36" level=debug msg="Initializing API" api_id=1 api_name="Test backend" org_id=default
time="Jan 04 15:06:36" level=debug msg="Loading custom PRE-PROCESSOR middleware: TestPlugin" prefix=main
time="Jan 04 15:06:36" level=debug msg=Init api_id=1 api_name="Test backend" mw=VersionCheck org_id=default
time="Jan 04 15:06:36" level=debug msg="Registering coprocess middleware, hook name: TestPluginhook type: Pre, driver: " api_id=1 api_name="Test backend" org_id=default prefix=coprocess
time="Jan 04 15:06:36" level=error msg="Your API specifies a CP custom middleware, either Tyk wasn't build with CP support or CP is not enabled in your Tyk configuration file!" prefix=coprocess

Configuration (tyk config file):

No config file, since running in Kubernetes using the Helm chart. Relevant environment variables:

      TYK_GW_MIDDLEWAREPATH:  /mnt/tyk-gateway/middleware
      TYK_GW_HTTPSERVEROPTIONS_ENABLEHTTP2: true
      TYK_GW_HTTPSERVEROPTIONS_FLUSHINTERVAL:  1
      TYK_GW_PROXYENABLEHTTP2: true
      TYK_GW_LOGLEVEL: debug
      TYK_GW_ENABLEBUNDLEDOWNLOADER: true
      TYK_GW_BUNDLEBASEURL: ttp://bundle-server.default.svc.cluster.local/bundles/
      TYK_GW_BUNDLEINSECURESKIPVERIFY: true
      TYK_GW_COPROCESSOPTIONS_ENABLECOPROCESS: false

Additional context

Looking at the code base, it seems to me that bundle downloader is only working for coprocess anyway, since the only place in the code where the config property is read is in gateway/coprocess_bundle.go.
But the docs linked above clearly state that this should work for goplugin driver also.

Pete · January 9, 2024, 3:35pm

I’ve tried to reproduce this issue using the environment variables provided and CE edition of the gateway at 5.2.5 but failed.

Could you provide the manifest json you’re using please?

Also, this looks like a typo. It’s missing the h in http

Cheers,
Pete

Dirk · January 10, 2024, 8:48am

Sure, here it is. The missing h was just a copy&paste error, since I copied those from the values file, where they are organized a bit more bloated.

{
  "file_list": [
    "TestPlugin_v5.2.5_linux_amd64.so"
  ],
  "custom_middleware": {
    "driver": "goplugin",
    "pre": [
      {
        "name": "TestPlugin",
        "path": "TestPlugin_v5.2.5_linux_amd64.so"
      }
    ]
  }
}

Pete · January 10, 2024, 11:16am

Thanks for the manifest file. I’ll continue to investigate but I wanted to point out that the following would be a better manifest that would allow you to upgrade the gateway version more seamlessly in future:

{
  "file_list": [
    "TestPlugin_v5.2.5_linux_amd64.so"
  ],
  "custom_middleware": {
    "driver": "goplugin",
    "pre": [
      {
        "name": "TestPlugin",
        "path": "TestPlugin.so"
      }
    ]
  }
}

There’s more information at Golang plugins but as an example, if you were to want to run a mixture of 5.2.5 and 5.2.6 gateways during an upgrade you would recompile the plugin with the 5.2.6 plugin compiler and add it to the bundle with a manifest that looked like this:

{
  "file_list": [
    "TestPlugin_v5.2.5_linux_amd64.so",
    "TestPlugin_v5.2.6_linux_amd64.so"
  ],
  "custom_middleware": {
    "driver": "goplugin",
    "pre": [
      {
        "name": "TestPlugin",
        "path": "TestPlugin.so"
      }
    ]
  }
}

The 5.2.5 and 5.2.6 gateways would then each load the right shared object.

Of course this is all moot if you can’t get plugins working.

Could you confirm that you’re using the official, unchanged docker image for gateway 5.2.5?

Cheers,
Pete

Dirk · January 10, 2024, 12:38pm

Thanks for the hint!

Yes, I can confirm that:

Image:         docker.tyk.io/tyk-gateway/tyk-gateway:v5.2.5
Image ID:      docker.tyk.io/tyk-gateway/tyk-gateway@sha256:de57c5e5a0d026a831ff37583b74c1ea91c9db7e0ecef41fa9ed074f1dc30c48

Pete · January 10, 2024, 4:00pm

Hi Dirk,

I can use your manifest to load a golang plugin correctly so I started experimenting with typos in the manifest.

This error message is very telling:

The driver name should be after the colon ( : ) character but it’s missing. When I leave out the driver I get exactly that message. If I set the driver to be "fred" I get the message Pre, driver: fred" so I’m pretty sure that there’s a manifest in /mnt/tyk-gateway/middleware with a missing driver or the driver key missing all together.

You may need to look through the bundles subdirectory to find it, but it must be there.

Cheers,
Pete

Dirk · January 10, 2024, 4:13pm

I’m pretty sure I did this and found goplugin there, but I’m going to set this up from scratch once more tomorrow. Thanks a lot for the support so far

Pete · January 11, 2024, 10:41am

It might not be a manifest from a bundle that’s doing it either. It might be coded into the API definition itself.

Check the pre section of this one.

Dirk · January 11, 2024, 11:12am

Yes and no. The API definition (in the file we use) does not have a custom_middleware section at all, just the "custom_middleware_bundle": "TestPlugin.zip".

But when querying the API after startup, the custom_middleware section contains the stuff from the manifest, with the exception that the driver field has an empty string as value, as written in the original post above.

Dirk · January 11, 2024, 12:02pm

Sorry so much for wasting your precious time
After setting up from scratch, it now works as designed .
I’ve not yet figured out what exactly is different now, but I hope that Git helps to identify the issue. If yes, I’ll post the diff here.

Pete · January 11, 2024, 12:10pm

Great news, I’m very glad to hear it!

Dirk · January 12, 2024, 2:19pm

Seems that I did not commit the faulty version, since it was not working anyway. I guess it must have been a typo in the driver field of the manifest.json, as you hinted at. I could verify that with an invalid value, exactly the outcome described above occurs.
Maybe validating this when loading the bundle, and failing with a descriptive error message might be a nice enhancement
Once more, many thanks for your support @Pete

timothy_park · April 1, 2024, 10:26pm

@Dirk @Pete I can’t seem to use the docker plugin-compiler to build a custom plugin successfully. I opened up an issue here (Can't compile custom-go-plugin using v5.2.5). Is there alternative documentation that outlines how to run the steps to compile a simple script to use as part of api middleware? Thanks for any help!

Dirk · April 2, 2024, 4:30am

Hey Timothy,
I responded in your own issue, but summarizing my knowledge here also: afaik, using a different version than the gateway is not possible with native Go plugins