Test fails with no matches for kind \"Route\" although routev1 is added to schema

[avrahams]

Bug Report

What did you do?

I am following the code skeleton for writing test for controllers

Since my controller also uses openshift routes I added the route to the schema

//controllers/suite_test.go
import (
       ...
	routev1 "github.com/openshift/api/route/v1"
	"k8s.io/client-go/kubernetes/scheme"
	...
)

func TestAPIs(t *testing.T) {
	RegisterFailHandler(Fail)
	RunSpecsWithDefaultAndCustomReporters(t,
		"Controller Suite",
		[]Reporter{printer.NewlineReporter{}})
}

var _ = BeforeSuite(func(done Done) {
	logf.SetLogger(zap.LoggerTo(GinkgoWriter, true))

	By("bootstrapping test environment")
	testEnv = &envtest.Environment{
		CRDDirectoryPaths: []string{filepath.Join("..", "config", "crd", "bases")},
	}

	var err error
	cfg, err = testEnv.Start()
	Expect(err).ToNot(HaveOccurred())
	Expect(cfg).ToNot(BeNil())

	err = elmv1.AddToScheme(scheme.Scheme)
	Expect(err).NotTo(HaveOccurred())
	//ADD ROUTE-V1
	err = routev1.AddToScheme(scheme.Scheme)
	Expect(err).NotTo(HaveOccurred())	


	k8sClient, err = client.New(cfg, client.Options{Scheme: scheme.Scheme})
	Expect(err).ToNot(HaveOccurred())

	k8sManager, err := ctrl.NewManager(cfg, ctrl.Options{
		Scheme: scheme.Scheme,
	})
	Expect(err).ToNot(HaveOccurred())
        
        //try to add route v1 again - does not help 
	err = routev1.AddToScheme(k8sManager.GetScheme())
	Expect(err).NotTo(HaveOccurred())

	err = (&ELMReconciler{
		Client: k8sManager.GetClient(),
		Log:    ctrl.Log.WithName("controllers").WithName("ELM"),
	}).SetupWithManager(k8sManager)
	Expect(err).ToNot(HaveOccurred())

	go func() {
                //this call will fail with no matches for kind Route
		err = k8sManager.Start(ctrl.SetupSignalHandler())
		Expect(err).ToNot(HaveOccurred())
	}()

	k8sClient = k8sManager.GetClient()
	Expect(k8sClient).ToNot(BeNil())

	close(done)
}, 60)

What did you expect to see?

I expected the manager to start in the test suite without errors as it does in runtime

What did you see instead? Under which circumstances?

After manager.start is called the test fails with no matches for kind "Route" error

2021-01-26T13:55:05.853+0200    ERROR   controller-runtime.source       if kind is a CRD, it should be installed before calling Start   {"kind": "Route.route.openshift.io", "error": "no matches for kind \"Route\" in version \"route.openshift.io/v1\""}
github.com/go-logr/zapr.(*zapLogger).Error
        /home/avrahams/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start
        /home/avrahams/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/source/source.go:117
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
        /home/avrahams/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:143
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
        /home/avrahams/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:184
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1
        /home/avrahams/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/manager/internal.go:661
panic: 
Your test failed.
Ginkgo panics to prevent subsequent assertions from running.
Normally Ginkgo rescues this panic so you shouldn't see it.

Environment

Operator type:

/language go

Kubernetes cluster type:
OpenShift

$ operator-sdk version
operator-sdk version: "v1.2.0", commit: "215fc50b2d4acc7d92b36828f42d7d1ae212015c", kubernetes version: "v1.18.8", go version: "go1.15.3", GOOS: "linux", GOARCH: "amd64"

$ go version (if language is Go)
go version go1.15.6 linux/amd64
$ kubectl version

Possible Solution

Additional context

[camilamacedo86]

HI @varshaprasad96 @rashmigottipati,

It is related to our chat. some references:

• [Discussion] V3 breaking changes kubernetes-sigs/kubebuilder#1772 (comment)
• blog post with the legacy layout https://developers.redhat.com/blog/2020/02/04/how-to-use-third-party-apis-in-operator-sdk-projects/
• sdk docs: https://sdk.operatorframework.io/docs/building-operators/golang/advanced-topics/#adding-3rd-party-resources-to-your-operator

It is something that we could maybe doc in upstream/kb and link here. Upstream is also missing a docs related.

• How to scaffold external/third-party apis? How config the test for external/third-party apis?

[avrahams]

@camilamacedo86 I understand that the code I expected to work as in kubernetes-sigs/kubebuilder#1772 (comment) is not working.
Is there a workaround that I can use until it's fixed ?

[camilamacedo86]

Hi @avrahams,

It might be missing some detail. To solve this issue, IHMO is required to do a POC and a few checks. Then, properly doc that as well.

By solving this one we might be able to close kubernetes-sigs/controller-runtime#1191 as well.

[estroz]

@varshaprasad96 will try to reproduce this. While kubernetes-sigs/controller-runtime#1191 might be the root cause, we are not sure.

/assign @varshaprasad96
/triage needs-information

[camilamacedo86]

Shows that for ENV TEST we need also update the CRDDirectoryPath to pass the right info. In this way, to solve this issue we just need to confirm that by doing this test with for example mechached sample in the testdata and doc it.

[mpreu]

While adding the CRDs seems to work, in case of OpenShift API resources it is a bit cumbersome. Since the api is not published via tags I have to use pseudo-versions. Keeping the CRD paths in this format in sync with the go.mod is doable, but maybe streamlining that experience is possible? Especially since operator-sdk targets OCP, I kinda expect to not have to do these steps manually.

Storing the CRDs as fixtures in the project repository would also be possible, but that still would require to manually put them there each time I update the go.mod for the OCP api.

Another question:
While a lot of CRDs for OCP APIs can be found in the corresponding api repository, I am not able to find CRDs for stuff like Routes, DeploymentConfigs, Projects (project.openshift.io/v1), ... and it seems they cannot be generated from the api repository either. Am I missing something here? Right now I do not find a way to add these resources to envtest.

[camilamacedo86]

Hi @mpreu,

External types are not officially supported by the tool yet. I mean the tool is not prepared for dealing with this scenario and users need to change the files manually. However, see that we have an issue to starts to support in upstream. Then, it will also be applied in SDK. See: kubernetes-sigs/kubebuilder#1999

And then, I also understand that we need to check here how is the best approach to set up the CRD paths for external-types for Envtest in order to solve this scenario. I mean, we need to confirm if the solution described by you still the best/recommended way to do it.

[ahalimx86]

I am facing the same issue with envtest even though I am not adding any external CRDs. Getting no matches for kind...
Seems like the CRDs are not being installed correctly.
Any workarounds?

[estroz]

While adding the CRDs seems to work, in case of OpenShift API resources it is a bit cumbersome. Since the api is not published via tags I have to use pseudo-versions. Keeping the CRD paths in this format in sync with the go.mod is doable, but maybe streamlining that experience is possible? Especially since operator-sdk targets OCP, I kinda expect to not have to do these steps manually.

@mpreu I don't see how this is a generally solvable problem unfortunately, since CRDs can be hosted anywhere for any project your operator depends on (doesn't have to be a k8s distro's APIs either, they could be another operator's). operator-sdk is meant to be general, i.e. doesn't specifically target OCP but does support it, but can't know where all distro's/operator's CRDs are hosted and appropriately download/version them. The best operator-sdk could do is wrap curl https://my.domain/crd.yaml, which isn't that useful.

While a lot of CRDs for OCP APIs can be found in the corresponding api repository, I am not able to find CRDs for stuff like Routes, DeploymentConfigs, Projects (project.openshift.io/v1), ... and it seems they cannot be generated from the api repository either

I would bring this issue up in the openshift/api repo.

Any workarounds?

@ahalim-intel you'll have to download the CRD manifests and reference them during envtest setup.

---

I've confirmed that this issue is a duplicate of kubernetes-sigs/controller-runtime#1191, so all conversation should be moved to that issue. I'm going to close this with an FAQ update.

/remove-triage needs-information
/triage duplicate

[varshaprasad96]

Have submitted a PR upstream (kubernetes-sigs/controller-runtime#1393) to document this.

[ahalimx86]

Hi All,
As I mentioned. my problem was slightly different then the OP's one(no external CRD) but with same error.
What I was doing differently is that I added multi-group API in my project. The kubebuilder documentation doesn't mention anywhere that suite_test.go should be updated accordingly as adding multi-group api adds additional directory to the apis/ and controller/ tree.

So basically with generated suite_test.go we get:

	testEnv = &envtest.Environment{
		CRDDirectoryPaths: []string{filepath.Join("..", "config", "crd", "bases")},
	}

Since adding multi-group api puts everything one directory down we needed to add additional ".." to the filepath.

So, adding that solved my particular problem.

	testEnv = &envtest.Environment{
		CRDDirectoryPaths: []string{filepath.Join("..", "..", "config", "crd", "bases")},
	}

Took to me a good while to figure this simple error. I think this should added in somewhere like here?

I hope someone facing similar issue find this helpful.

[camilamacedo86]

Hi @ahalim-intel,

The kubebuilder documentation doesn't mention anywhere that suite_test.go should be updated accordingly as adding multi-group api adds additional directory to the apis/ and controller/ tree.

When you enable multi-group then the new scaffolds will be generating with the path changed as you described. However, it should be described in the doc linked by you too. WDYT about help us and push a PR to update it? Here is the place: https://github.com/kubernetes-sigs/kubebuilder/blob/master/docs/book/src/migration/multi-group.md

[ahalimx86]

Hi @camilamacedo86,

WDYT about help us and push a PR to update it? Here is the place: https://github.com/kubernetes-sigs/kubebuilder/blob/master/docs/book/src/migration/multi-group.md

I've submitted a PR as suggested. PTAL kubernetes-sigs/kubebuilder#2054

[estroz]

Update: for OCP and in general externally-defined APIs that use extra-CRD validation, like webhooks and custom apiserver imperative validation, there is no good solution for this problem with envtest; positive test outcomes can be misleading if CRDs alone are used because some other validation could fail server-side in prod. Therefore OCP CRDs alone cannot solve this issue.

These facts should be documented in the envtest library, and an FAQ should be added for like-kind errors with this explanation.

/remove-triage duplicate
/kind documentation
/priority important-longterm

[varshaprasad96]

This is similar to the upstream issue (kubernetes-sigs/controller-runtime#1191). This needs to be handled upstream since envtest library needs more documentation.

[avrahams]

@camilamacedo86,
Please see the replay I got in the upstream issue - seems that this issue with openshift resources does not have a clean solution even when the upstream issue will be fixed
kubernetes-sigs/controller-runtime#1191 (comment)

[dengshaojiang]

Hi All, As I mentioned. my problem was slightly different then the OP's one(no external CRD) but with same error. What I was doing differently is that I added multi-group API in my project. The kubebuilder documentation doesn't mention anywhere that suite_test.go should be updated accordingly as adding multi-group api adds additional directory to the apis/ and controller/ tree.

So basically with generated suite_test.go we get:

	testEnv = &envtest.Environment{
		CRDDirectoryPaths: []string{filepath.Join("..", "config", "crd", "bases")},
	}

Since adding multi-group api puts everything one directory down we needed to add additional ".." to the filepath.

So, adding that solved my particular problem.

	testEnv = &envtest.Environment{
		CRDDirectoryPaths: []string{filepath.Join("..", "..", "config", "crd", "bases")},
	}

Took to me a good while to figure this simple error. I think this should added in somewhere like here?

I hope someone facing similar issue find this helpful.

Thanks a lot!
It perfectly solves the problem of third-party CRDs in envtest.