Datadog logs missing after Karpenter cluster update

If your Datadog logs stopped working after a Karpenter cluster update, this is likely due to the Datadog agent not being scheduled on all node pools created by Karpenter.

Why This Happens

When Karpenter is enabled, Qovery creates two distinct node pools:

• Default pool: Where most workloads run (managed by Karpenter)

• Stable pool: For critical system components (static node group)

The Datadog agent DaemonSet likely has node selectors or taints/tolerations that prevent it from running on the stable nodes where some of your services have been scheduled. Services with only one pod typically run on the stable node pool, which explains why logs from these services disappear after the Karpenter update.

Solution: Deploy Datadog Agent on All Node Pools

You need to update your Datadog agent configuration to ensure it runs on both node pools. Update your Datadog agent DaemonSet tolerations in your values.yml file.

Option 1: Tolerate All Taints (Recommended)

tolerations:
  - operator: Exists  # This will tolerate all taints

Option 2: Specific Node Pool Tolerations

If you prefer to be more selective, use these specific tolerations:

tolerations:
  - key: "qovery.com/node-pool"
    operator: "Equal"
    value: "stable"
    effect: "NoSchedule"
  - key: "qovery.com/node-pool"
    operator: "Equal"
    value: "default"
    effect: "NoSchedule"

After updating your Datadog Helm chart with these tolerations, the agent should be able to collect logs from services running on both the default and stable node pools.

For more information about node pool placement and optimization settings, refer to the Qovery documentation.