So, this post is about covering these two topics:

• Creating dependencies between resources in a composition
• Debugging complex or nested compositions when they aren't working

Pre-requisites

• AWS Account
• AWS CLI
• Crossplane k8s cluster with AWS Provider & K8s Provider

Playground

Here's the repository we will be using to go through the above topics.

Structure of the repository:

.
├── claims          # xplane claims for nested compositions
├── compositions    # xplane compositions defining a host-cluster i.e. vpc-subnets, eks, services
└── README.md

We have the following composition:

├── eks        # composition defining the eks cluster and nodegroup
├── service    # composition defining k8s manifest via k8s provider
└── xcluster   # composition defining dependency between eks & service

Creating dependencies between resources in a composition

While using crossplane native providers to provision infra resources (e.g. k8s cluster), there can be a need to bootstrap some application flows. For example: store the k8s cluster kubeconfig to secrets manager OR install a helm application on the cluster once the cluster is ready. In most cases for real-world application scenarios you likely would end up in a scenario as such.

In our example we will:

1. Install the VPC composition and the claim

kubectl apply -f compositions/vpc-subnets/
kubectl apply -f claims/vpc-subnets-claim.yaml

2. Check the claim status and verify on AWS console

NAME                                        READY   SYNCED   AGE
vpcsubnet.network.glide.io/xplane-glide-vpc-subnets   True    True     45d

We will need the subnet IDs in the upcoming section.

3. Apply the other compositions

Once we have the claim in ready state, we will apply the other compositions that we intend to create dependency with. In our example we create EKS cluster & Service composition and create dependency with an XCluster composition:

kubectl apply -f compositions/eks
kubectl apply -f compositions/services
kubectl apply -f compositions/xcluster

4. Update the subnet IDs in the claim definition

subnetIds:   # Update your private subnet ids
  - "subnet-xx"
  - "subnet-xx"
  - "subnet-xx"

5. Apply the claim

kubectl apply -f claims/xcluster-claim.yaml

This will now create both the EKS & service composition via this intermediary xcluster composition and claim.

The Trick

The trick is to create a dependency between two composites by combining them in a third composite and use patching with policy.fromFieldPath: Required to block creation of the second composite until some information from the first composite is available. Generally, use a status field that is not available until the composite is Ready and use that to trigger creation of the second composite.

Debugging complex or nested compositions when they aren't working

While writing complex compositions with multiple managed objects and dependencies between other compositions, we are bound to face situations when our claims are not ready.

Example:

kubectl apply -f claims/xcluster-claim.yaml

After applying the dependency claim we notice that the claim itself isn't ready.

In such cases, we can start debugging via the claim and the nested objects in the following way:

kubectl describe cls.supercluster.glide.io/managed-cluster

Events can often help us get into the direction needed to further debug the issue:

Events:
  Type     Reason                     Age   Message
  ----     ------                     ----  -------
  Warning  ConfigureCompositeResource 19m   cannot apply composite resource...
  Normal   BindCompositeResource      4m9s  Composite resource is not yet ready
  Normal   ConfigureCompositeResource 3m9s  Successfully applied composite resource

Get the kind and name of the composition

kubectl get cls.supercluster.glide.io/managed-cluster \
  -o=jsonpath='{.spec.resourceRef}{" "}{.spec.resourceRefs}' | jq

Output:

{
  "apiVersion": "supercluster.glide.io/v1alpha1",
  "kind": "XCls",
  "name": "managed-cluster-srhb7"
}

Describe the nested kind

kubectl describe XCls managed-cluster-srhb7

Events:
  Type     Reason            Age   Message
  ----     ------            ----  -------
  Warning  ComposeResources  24m   cannot compose resources: cannot render...
  Normal   ComposeResources  24m   Composed resource "add-cluster-to-loft" is not yet ready
  Normal   ComposeResources  23m   Successfully composed resources
  Normal   ComposeResources  23m   Composed resource "eks-cluster" is not yet ready

Drill down further

kubectl get XCls managed-cluster-srhb7 \
  -o=jsonpath='{.spec.resourceRef}{" "}{.spec.resourceRefs}' | jq

Output:

[
  {
    "apiVersion": "cluster.glide.io/v1alpha1",
    "kind": "XEks",
    "name": "managed-cluster-srhb7"
  },
  {
    "apiVersion": "supercluster.glide.io/v1alpha1",
    "kind": "XHost",
    "name": "managed-cluster-srhb7-4fv5g"
  }
]

We can now see this composition is defined of two other compositions, which we can further drill down in a similar fashion.

Example with XEks kind

kubectl describe XEks managed-cluster-srhb7

Events:
  Type    Reason            Age   Message
  ----    ------            ----  -------
  Normal  ComposeResources  28m   Composed resource "eks-addons" is not yet ready
  Normal  ComposeResources  28m   Successfully composed resources
  Normal  ComposeResources  28m   Composed resource "eks-cluster" is not yet ready
  Normal  ComposeResources  28m   Composed resource "eks-cluster-key" is not yet ready

These events help to understand issues with our composition. Similar approach can be taken for even more complex or nested compositions.

Conclusion

Hope this information helps you on your journey with platform engineering and Crossplane!