500 Internal Server Error during AccessRightsCheck with Golang plugin on v5.3.5

Aniket_Singh · May 7, 2025, 10:10am

I have an on-prem tyk setup running on k8s. I use Golang plugin for session management. The plugin works perfectly for all APIs. However, when I rebuilt the plugin with a few changes and applied it to one of my APIs, the API calls fail with 500 Internal Server Error

Below are snippets from the Tyk Gateway logs during the request:

time="May 06 14:33:58" level=debug msg=Started api_id=09647da4437b483640db0a0872c97cbc api_name="Auth Service" mw=AccessRightsCheck org_id=64e78f85a299fe0001f8a828 origin=192.168.6.117 path=/auth/users/v1/login ts=1746531238136825376
time="May 06 14:33:58" level=debug msg=Finished api_id=09647da4437b483640db0a0872c97cbc api_name="Auth Service" code=200 mw=AccessRightsCheck ns=139256 org_id=64e78f85a299fe0001f8a828 origin=192.168.6.117 path=/auth/users/v1/login
time="May 06 14:33:58" level=debug msg=Started api_id=09647da4437b483640db0a0872c97cbc api_name="Auth Service" mw=AccessRightsCheck org_id=64e78f85a299fe0001f8a828 origin=192.168.6.117 path=/auth/users/v1/login ts=1746531238137104770

==== Internal Server Error====
time="May 06 14:33:58" level=debug msg=Finished api_id=09647da4437b483640db0a0872c97cbc api_name="Auth Service" code=500 error="Internal Server Error" mw=AccessRightsCheck ns=341182 org_id=64e78f85a299fe0001f8a828 origin=192.168.6.117 path=/auth/users/v1/login

I am unable to understand if this is related to some policy as there is a “Started api_id=” log but error in the “Finished api_id=” log, OR it’s a plugin issue? Is there a way to view more verbose logs?

Attaching the plugin manifest.json I am using for building the plugin:

{
  "file_list": ["plugin.so"],
  "custom_middleware": {
    "pre": [
      {
        "disabled": false,
        "name": "PreHeaderInjectors38",
        "path": "./plugin.so",
        "require_session": false,
        "raw_body_only": false
      }
    ],
    "post": [],
    "post_key_auth": [
      {
        "disabled": false,
        "name": "LocationHeaderInjector38",
        "path": "./plugin.so",
        "require_session": false,
        "raw_body_only": false
      },
      {
        "disabled": false,
        "name": "AuthHeaderInjector38",
        "path": "./plugin.so",
        "require_session": false,
        "raw_body_only": false
      }
    ],
    "auth_check": {
      "disabled": false,
      "name": "",
      "path": "",
      "require_session": false,
      "raw_body_only": false
    },
    "response": [
      {
        "disabled": false,
        "name": "StripResponseHeaders38",
        "path": "plugin.so",
        "require_session": false,
        "raw_body_only": false
      }
    ],
    "driver": "goplugin",
    "id_extractor": {
      "disabled": false,
      "extract_from": "",
      "extract_with": "",
      "extractor_config": null
    }
  },
  "checksum": "",
  "signature": ""
}

Can someone help with what might be wrong here?

Isaac · May 8, 2025, 11:21pm

Hi Aniket,

Thanks for reaching out. Based on the logs and the behavior you’re observing, the 500 Internal Server Error is most likely due to an issue within the updated Golang plugin, rather than a policy or API configuration.

The error occurs during the AccessRightsCheck middleware phase, indicating the plugin is being executed but likely fails due to a runtime issue. We recommend reviewing the recent changes in your plugin closely or reverting to the previous version to confirm the plugin is the cause.

Let us know if this helps.

Regards

Aniket_Singh · May 12, 2025, 7:12am

I am getting “Internal Server Error” for APIs for which the plugin is ignored. Check the screenshot.

I have two bundles (bundle40.zip & bundle.zip) built from the same code, just different manifest.json files. bundle40.zip is working fine and bundle.zip is giving “500 Internal Server Error”. Reverting back to bundle40.zip works fine but it is the exact same code as bundle.zip with just function name change.

Here are the logs of this bundle getting loaded fine:

time="May 12 09:47:58" level=info msg="----> Fetching Bundle: bundle.zip" prefix=main
time="May 12 09:47:58" level=info msg="Attempting to download plugin bundle: https://objectstorage.me-jeddah-1.oraclecloud.com/p/1QaXnAPyiKERPFtMoom7ZbJXjejbxQ82uOmowPBpeC3XxUqgUKLYcxhO1bia8fsI/n/axbp9ow0rohq/b/dev-tyk-plugins/o/bundle.zip"
time="May 12 09:47:58" level=debug msg="----> Saving Bundle: bundle.zip" prefix=main
time="May 12 09:47:58" level=info msg="----> Loading bundle: bundle.zip" prefix=main
time="May 12 09:47:58" level=info msg="----> Verifying bundle: bundle.zip" prefix=main
time="May 12 09:47:58" level=info msg="----> Bundle is valid, adding to spec: bundle.zip" prefix=main
time="May 12 09:47:58" level=debug msg="URLSpec: /users/v1/initiate-login => /users/v1/initiate-login type=1" error="<nil>"
time="May 12 09:47:58" level=debug msg="URLSpec: /users/v1/refresh-token => /users/v1/refresh-token type=1" error="<nil>"
time="May 12 09:47:58" level=debug msg="URLSpec: /users/v1/login => /users/v1/login type=1" error="<nil>"

Is there a way to see more logs during API call to identify where in the plugin it’s going wrong?

Steps to build the plugin (Using v5.3.5):

docker run --rm -v `pwd`:/plugin-source tykio/tyk-plugin-compiler:v5.3.5 plugin.so
/opt/tyk-gateway/tyk bundle build

Have attached manifest.json files of both plugins:
bundle40.zip (works)

{
  "file_list": [
    "plugin_v5.3.5_linux_amd64.so"
  ],
  "custom_middleware": {
    "pre": [
      {
        "disabled": false,
        "name": "PreHeaderInjectors39",
        "path": "plugin.so",
        "require_session": false,
        "raw_body_only": false
      }
    ],
    "post": [],
    "post_key_auth": [
      {
        "disabled": false,
        "name": "LocationHeaderInjector39",
        "path": "plugin.so",
        "require_session": false,
        "raw_body_only": false
      },
      {
        "disabled": false,
        "name": "AuthHeaderInjector39",
        "path": "plugin.so",
        "require_session": false,
        "raw_body_only": false
      }
    ],
    "auth_check": {
      "disabled": false,
      "name": "",
      "path": "",
      "require_session": false,
      "raw_body_only": false
    },
    "response": [
      {
        "disabled": false,
        "name": "StripResponseHeaders39",
        "path": "plugin.so",
        "require_session": false,
        "raw_body_only": false
      }
    ],
    "driver": "goplugin",
    "id_extractor": {
      "disabled": false,
      "extract_from": "",
      "extract_with": "",
      "extractor_config": null
    }
  },
  "checksum": "1a7195afd21cf1f80dfa0388826aa2ce",
  "signature": ""
}

bundle.zip (fails)

{
  "file_list": [
    "plugin_v5.3.5_linux_amd64.so"
  ],
  "custom_middleware": {
    "pre": [
      {
        "disabled": false,
        "name": "PreHeaderInjectors38",
        "path": "./plugin_v5.3.5_linux_amd64.so",
        "require_session": false,
        "raw_body_only": false
      }
    ],
    "post": [],
    "post_key_auth": [
      {
        "disabled": false,
        "name": "LocationHeaderInjector38",
        "path": "./plugin_v5.3.5_linux_amd64.so",
        "require_session": false,
        "raw_body_only": false
      },
      {
        "disabled": false,
        "name": "AuthHeaderInjector38",
        "path": "./plugin_v5.3.5_linux_amd64.so",
        "require_session": false,
        "raw_body_only": false
      }
    ],
    "auth_check": {
      "disabled": false,
      "name": "",
      "path": "",
      "require_session": false,
      "raw_body_only": false
    },
    "response": [
      {
        "disabled": false,
        "name": "StripResponseHeaders38",
        "path": "./plugin_v5.3.5_linux_amd64.so",
        "require_session": false,
        "raw_body_only": false
      }
    ],
    "driver": "goplugin",
    "id_extractor": {
      "disabled": false,
      "extract_from": "",
      "extract_with": "",
      "extractor_config": null
    }
  },
  "checksum": "a2e5b907b826ae1436a0179f1934ee0a",
  "signature": ""
}

Aniket_Singh · May 14, 2025, 6:58am

@Isaac can you please help here?

Pete · May 14, 2025, 8:48am

Hi,

"path": "./plugin_v5.3.5_linux_amd64.so",

This is not the correct path for the plugin. The correct path is the one used in the previous manifest.

"path": "plugin.so",

When given plugin.so the gateway will work out the file name to load because it knows the version it is, the platform its running on and the architecture being used.

This means that when you upgrade to 5.3.6 all you need to do is use this file_list and the gateway will load the right plugin file

 "file_list": [
   "plugin_v5.3.5_linux_amd64.so",
   "plugin_v5.3.6_linux_amd64.so"
 ],

Please use this method and let us know how you get on.

Cheers,
Pete

Aniket_Singh · May 16, 2025, 8:56am

Hi Pete,
I tried this, and it doesn’t work. I am testing this new plugin version on one of my APIs while other APIs are still on the previous plugin bundle. But both the plugins have the same name (plugin.so).

Could it be an issue as in it’s not able to find which plugin to apply because both have the same name? My understanding is that since bundle is different for each plugin, gateway will be able to segregate this.

@Pete @Isaac

Pete · May 16, 2025, 9:07am

Hi,

It could be that the plugins have the same plugin ID.

You’re get a “Plugin already loaded” error if that’s the case.

This can be fixed by passing in a plugin ID when compiling the plugin. I use the epoch seconds for this.

For example:

docker run --rm -v `pwd`:/plugin-source tykio/tyk-plugin-compiler:v5.3.5 plugin.so $(date +%s)

This makes sure that the plugin ID is different for each plugin.

Cheers,
Pete

Aniket_Singh · May 16, 2025, 11:12am

Thanks Pete! It worked with the timestamp.