Go plugin using grpc compilation

Support
,
1

We’re trying to compile a Golang plugin which uses GRPC. Compiling the plugin on MacOS works fine, but when using the recommended Docker image tyk-plugin-compiler the build fails.

Command used for compilation (Tyk version v4.0.3)
docker run --rm -v $(pwd):/plugin-source tykio/tyk-plugin-compiler:v4.0.3 my-post-plugin.so

go.mod file:

module my_plugin

go 1.15

replace github.com/jensneuse/graphql-go-tools => github.com/TykTechnologies/graphql-go-tools v1.6.2-0.20220426094453-0cc35471c1ca

require (
	github.com/TykTechnologies/tyk v1.9.2-0.20220614105651-6c76e802a298
	github.com/authzed/authzed-go v0.6.0
	github.com/authzed/grpcutil v0.0.0-20220104222419-f813f77722e5
)

We’re getting the following error during compilation:

+ GO111MODULE=off
+ go build -buildmode=plugin -o my-post-plugin.so
../github.com/authzed/authzed-go/proto/authzed/api/v1/permission_service.pb.go:11:2: cannot find package "google.golang.org/genproto/googleapis/api/annotations" in any of:
	/usr/local/go/src/google.golang.org/genproto/googleapis/api/annotations (from $GOROOT)
	/go/src/google.golang.org/genproto/googleapis/api/annotations (from $GOPATH)
../github.com/grpc-ecosystem/grpc-gateway/v2/runtime/handler.go:11:2: cannot find package "google.golang.org/genproto/googleapis/api/httpbody" in any of:
	/usr/local/go/src/google.golang.org/genproto/googleapis/api/httpbody (from $GOROOT)
	/go/src/google.golang.org/genproto/googleapis/api/httpbody (from $GOPATH)
../github.com/grpc-ecosystem/grpc-gateway/v2/runtime/fieldmask.go:9:2: cannot find package "google.golang.org/genproto/protobuf/field_mask" in any of:
	/usr/local/go/src/google.golang.org/genproto/protobuf/field_mask (from $GOROOT)
	/go/src/google.golang.org/genproto/protobuf/field_mask (from $GOPATH)
../github.com/authzed/grpcutil/middleware.go:12:2: cannot find package "google.golang.org/grpc/health" in any of:
	/usr/local/go/src/google.golang.org/grpc/health (from $GOROOT)
	/go/src/google.golang.org/grpc/health (from $GOPATH)
../github.com/authzed/grpcutil/middleware.go:13:2: cannot find package "google.golang.org/grpc/health/grpc_health_v1" in any of:
	/usr/local/go/src/google.golang.org/grpc/health/grpc_health_v1 (from $GOROOT)
	/go/src/google.golang.org/grpc/health/grpc_health_v1 (from $GOPATH)
../github.com/authzed/grpcutil/reflection.go:5:2: cannot find package "google.golang.org/grpc/reflection" in any of:
	/usr/local/go/src/google.golang.org/grpc/reflection (from $GOROOT)
	/go/src/google.golang.org/grpc/reflection (from $GOPATH)
../github.com/authzed/grpcutil/reflection.go:6:2: cannot find package "google.golang.org/grpc/reflection/grpc_reflection_v1alpha" in any of:
	/usr/local/go/src/google.golang.org/grpc/reflection/grpc_reflection_v1alpha (from $GOROOT)
	/go/src/google.golang.org/grpc/reflection/grpc_reflection_v1alpha (from $GOPATH)
../github.com/grpc-ecosystem/grpc-gateway/v2/runtime/convert.go:9:2: cannot find package "google.golang.org/protobuf/encoding/protojson" in any of:
	/usr/local/go/src/google.golang.org/protobuf/encoding/protojson (from $GOROOT)
	/go/src/google.golang.org/protobuf/encoding/protojson (from $GOPATH)
../github.com/envoyproxy/protoc-gen-validate/validate/validate.pb.go:12:2: cannot find package "google.golang.org/protobuf/types/descriptorpb" in any of:
	/usr/local/go/src/google.golang.org/protobuf/types/descriptorpb (from $GOROOT)
	/go/src/google.golang.org/protobuf/types/descriptorpb (from $GOPATH)
../github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2/options/openapiv2.pb.go:12:2: cannot find package "google.golang.org/protobuf/types/known/structpb" in any of:
	/usr/local/go/src/google.golang.org/protobuf/types/known/structpb (from $GOROOT)
	/go/src/google.golang.org/protobuf/types/known/structpb (from $GOPATH)
../github.com/grpc-ecosystem/grpc-gateway/v2/runtime/convert.go:12:2: cannot find package "google.golang.org/protobuf/types/known/wrapperspb" in any of:
	/usr/local/go/src/google.golang.org/protobuf/types/known/wrapperspb (from $GOROOT)
	/go/src/google.golang.org/protobuf/types/known/wrapperspb (from $GOPATH)
../github.com/authzed/grpcutil/x509util.go:6:2: cannot find package "io/fs" in any of:
	/usr/local/go/src/io/fs (from $GOROOT)
	/go/src/io/fs (from $GOPATH)

Is there some limitation on the dependencies that a plugin is allowed to use?

Kind regards, Barry

1 Like
2

I’ve added the code to a Github repo, including an Actions workflow to show the problem:

When running go build the compilation is fine, however when using the tyk-plugin-compiler it fails.

3

Hi @Barry_Lagerweij,

Not too sure about the steps you used to create this plugin, but I’m sure you followed this: Golang plugins.

However, the plugin-compiler is for compiling native go plugins and not gRPC-based go plugins. In fact, gRPC go-plugins don’t need to be compiled as such.
Also, the method signature for native go plugins is different from that of gRPC-based go plugins, so you’ll need to do some rewriting.

See a few sample gRPC-based go plugins:

And here’s a guide on writing gRPC plugins: How to write gRPC Plugins

This is helpful?

1 Like
4

Perhaps I should have rephrased the title a bit better: We can’t use the grpc feature of Tyk, our Golang plugin uses a go module to connect to an external service. This go-module that we’re using was generated from a protobuf spec.

The errors that we see during compilation seem to imply that the plugin compiler cannot find specific packages:

../github.com/authzed/authzed-go/proto/authzed/api/v1/permission_service.pb.go:11:2: cannot find package "google.golang.org/genproto/googleapis/api/annotations" in any of:
	/usr/local/go/src/google.golang.org/genproto/googleapis/api/annotations (from $GOROOT)
	/go/src/google.golang.org/genproto/googleapis/api/annotations (from $GOPATH)
Error: ../github.com/grpc-ecosystem/grpc-gateway/v2/runtime/handler.go:11:2: cannot find package "google.golang.org/genproto/googleapis/api/httpbody" in any of:
	/usr/local/go/src/google.golang.org/genproto/googleapis/api/httpbody (from $GOROOT)
	/go/src/google.golang.org/genproto/googleapis/api/httpbody (from $GOPATH)
Error: ../github.com/grpc-ecosystem/grpc-gateway/v2/runtime/fieldmask.go:9:2: cannot find package "google.golang.org/genproto/protobuf/field_mask" in any of:
	/usr/local/go/src/google.golang.org/genproto/protobuf/field_mask (from $GOROOT)
	/go/src/google.golang.org/genproto/protobuf/field_mask (from $GOPATH)

Hence the reference to grpc in the topic title. Hope that clears it up.

5

After some digging I think I’ve found the reason: The go module that we’re linking was compiled with go 1.18, while the Tyk plugin uses go version 1.15.

I will see if we can workaround this issue by generating the GRPC client in our project (instead of linking the pre-compiled module)

6

UPDATE:
After spending a few hours on go module versions, it turns out that the tyk-plugin-compiler build.sh script is flawed: after go mod vendor the script performs some magic in an attempt to harmonize the versions for the plugin dependencies: if the same module exists in Tyk, we should use the Tyk version instead of the version the plugin uses.

However, since some modules share a common base path, the script removes too many files.

I’ve come up with the following script to improve the build (Tyk version 4.0.3):

# for any dependency also present in Tyk, change the module version to Tyk's version
go list -m -f '{{ if not .Main }}{{ .Path }} {{ .Version }}{{ end }}' all > /tmp/plugin-deps.txt
(cd $TYK_GW_PATH && go list -m -mod=mod -f '{{ if not .Main }}{{ .Path }} {{ .Version }}{{ end }}' all > /tmp/gw-deps.txt)
awk 'NR==FNR{seen[$1]=$2; next} seen[$1] && seen[$1] != $2' /tmp/plugin-deps.txt /tmp/gw-deps.txt | while read PKG VER; do
  go mod edit -replace=$PKG=$PKG@$VER
done

go build -buildmode=plugin -o my-plugin.so

The above script uses go list to list all dependencies used by Tyk and the plugin. Then it uses awk to find any module which is shared by the plugin and Tyk, but with a different version. For each discrepancy, we use go mod edit -replace to fix the version to Tyk’s.

After doing this, I realized that the Go 1.18 dependency has some issues with the old versions, so I have use protoc during my build to generate fresh Go stubs for v1.15. At least you’ll see any compatibility issues during compilation. I hope the above script can be used by the Tyk plugin compiler ?