# Upgrade from Embedded Cluster v2 This page describes how to upgrade from Embedded Cluster v2 to Embedded Cluster v3. It includes information about how to update your application release to support Embedded Cluster v3. It also describes how to upgrade existing installations from Embedded Cluster v2 to v3. ## Benefits of Embedded Cluster v3 Embedded Cluster v3 delivers the following improvements over v2. For the release changes required to adopt them, see [Comparison to Embedded Cluster v2](#comparison-to-embedded-cluster-v2). **One guided experience from the first step.** In v2, cluster infrastructure was set up on the command line before a separate UI handled the application. In v3, a single binary drives cluster configuration, preflight checks, node setup, and application deployment in one wizard. See [Embedded Cluster overview](embedded-overview). **A smaller footprint for customers to secure.** The cluster runs your application and Embedded Cluster's own infrastructure components, with no Replicated management component holding cluster-scoped privileges. The KOTS operator's cluster-scoped RBAC grants and cross-namespace access are not part of a v3 install, which takes them out of a customer's security review. The install and upgrade wizard runs only while an operation is in progress, so no management console runs continuously in the cluster unless a customer opts into the [persistent admin console](embedded-persistent-console). See [Removal of KOTS](#removal-of-kots). **Installs no longer reach out to chart repositories.** v3 resolves every Helm extension from a `.tgz` archive bundled in the release instead of pulling it from an HTTP repository or OCI registry at install time. Installs no longer depend on the customer's network reaching those hosts. See [Helm extension charts must be bundled in the release](#extension-charts). **Private image pulls work for pods you don't control.** In v2, the Replicated pull secret had to be injected into every Pod definition that referenced a proxied image, which meant pods created at runtime by an operator, a Job, or a third-party subchart would fail to pull unless you threaded the secret through. v3 configures the cluster to authenticate to the Replicated proxy registry for all pods, so this is handled regardless of what creates the pod. See [Automatic authentication to the Replicated proxy registry](#automatic-authentication-to-the-replicated-proxy-registry). **Support for customer-controlled registries.** The cluster can pull every image, your application and the infrastructure components alike, from a single OCI registry that the customer operates. See [Install with a customer-managed registry](installing-embedded-byo-registry). **The full lifecycle can be automated.** Install, upgrade, and `node join` each run from the CLI or the [external API](embedded-cluster-external-api) with no console interaction. See [Install using the CLI (Headless)](installing-embedded#install-using-the-cli-headless) and [Upgrade using the CLI (headless)](updating-embedded#upgrade-using-the-cli-headless). ## Changes for end users You can adapt these lists for your own customer communications. ### During the upgrade from v2 These apply once, when an existing v2 installation moves to v3. **Customers won't see the v3 release in their Admin Console.** Releases that use Embedded Cluster v3 don't appear as available updates in the Embedded Cluster v2 KOTS Admin Console. This is by design, to prevent accidental upgrades. To get the v3 release, customers use the [Enterprise Portal](/vendor/enterprise-portal-about) or download the binary directly. Plan to communicate this when you promote a v3-enabled release. **The upgrade is one-way and asks for confirmation.** The v3 binary detects the existing v2 installation and prompts the user to confirm before it proceeds. There's no path back to v2. **Configuration carries forward.** Values the customer already set, such as the application domain, are preserved, so nothing has to be re-entered. **Customers more than one Kubernetes minor version behind need an intermediate release first.** Kubernetes supports upgrading only one minor version at a time, and Embedded Cluster blocks an upgrade that would skip one. See [Increase the Kubernetes minor version](embedded-config#increment-k8s) for how to lead customers through the versions in order. **Multi-node clusters upgrade the remaining nodes manually, this one time.** Streamed upgrades require every remote node to already be running Embedded Cluster 3.2 or later, and nodes still on v2 do not qualify. The wizard detects this and displays manual instructions for those nodes. The user produces an upgrade bundle on the primary controller, copies it to each remaining node, and runs `node upgrade` there. Once every node is on v3, later upgrades stream automatically with no copying. See [create-upgrade-bundle](embedded-cluster-create-upgrade-bundle) and [node upgrade](embedded-cluster-node-upgrade). **Port 30080/TCP must be open on the node.** The Embedded Cluster UI requires it, and it must also be reachable by nodes joining the cluster. Customers upgrading an existing v2 node may need to create a firewall opening for it. If the port is occupied, they can select a different port during installation with `--installer-port`. See [Port requirements](installing-embedded-requirements#port-requirements). **Multi-node clusters also need port 30081/TCP.** The Embedded Cluster daemon API listens on this port and is what lets other nodes join and receive upgrades. Single-node clusters do not need it opened. If the port is occupied, customers can select a different port at install with `--api-port`. See [External API](embedded-cluster-external-api). ### On any v3 installation These are the ongoing differences after the upgrade. **A guided wizard handles install and upgrade.** Install and upgrade operations launch a wizard that walks the user through configuration, preflight checks, node setup, and deployment, and closes when the operation finishes. It replaces the KOTS Admin Console that handled these steps in v2. **An always-on console is available when you want one.** The [persistent admin console](embedded-persistent-console) runs as an opt-in system service, giving end users a browser dashboard without command-line access. From it they can see available updates, view release notes, trigger an upgrade, upload air gap releases, and generate support bundles. It isn't installed by default, and the initial install and console setup require command-line access. **How customers find out about updates depends on whether the console is installed.** With the persistent admin console, available updates appear in the console. Without it, customers get updates from the [Enterprise Portal](/vendor/enterprise-portal-about) or by running the `upgrade` command directly. See [Perform updates in embedded clusters](updating-embedded). **Multi-node clusters upgrade themselves one node at a time.** The user runs the upgrade on the primary controller, and its daemon streams the upgrade to each remaining node over the network. Controllers upgrade first, one at a time to preserve etcd quorum, then workers. Nobody copies files between machines or runs commands on the other nodes. If a node can't be reached, or is running a version that predates streamed upgrades, the wizard reports it and shows manual instructions for that node. See [Multi-node clusters](updating-embedded#multi-node-clusters). **Cluster settings are collected in the wizard.** The v3 wizard includes a *Set Up* page where end users provide cluster-level settings such as proxy configuration. In v2 these were passed as CLI flags at install time. :::note Config value defaults behave differently in v3. All config values, including those that use spec defaults, are resolved and persisted at install time, and previously persisted values take precedence over new defaults on upgrade. Changing a `default` field in a later release won't reach existing installations. See [Config value defaults are persisted at install time](#config-value-defaults-are-persisted-at-install-time). ::: ## Comparison to Embedded Cluster v2 This section describes the key differences between Embedded Cluster v2 and v3. ### Removal of KOTS Embedded Cluster v3 removes Replicated KOTS from the architecture. This reduces the number of dependencies that are running in the cluster, which improves reliability. The KOTS CLI does not work with Embedded Cluster v3, and there is no KOTS Admin Console. Embedded Cluster v3 replaces it with two interfaces: - A **guided install and upgrade UI** that walks end users through configuration, preflight checks, node setup, and application deployment. This UI launches automatically during install and upgrade operations. See [Set up the wizard's Configure screen](/embedded-cluster/v3/embedded-using#configure-screen). - An optional **[persistent admin console](/embedded-cluster/v3/embedded-persistent-console)** that provides an always-on web dashboard for browser-driven upgrades, air gap release uploads, and support bundle generation without SSH access. Embedded Cluster v3 still requires the Replicated HelmChart v2 custom resource to process and deploy Helm charts. Embedded Cluster v3 also still uses Replicated custom resources like the Replicated Application and Replicated Config resources to define aspects of the installation experience. ### Replicated SDK required for instance status reporting Because Embedded Cluster v3 removes KOTS, you must include the [Replicated SDK](/vendor/replicated-sdk-overview) in your application for application status informers to report instance status to the Vendor Portal. Continue to configure status informers in the Replicated Application custom resource. Embedded Cluster v3 passes them to the SDK. For more information, see [Enable and understand application status](/vendor/insights-app-status). ### Application preflight specs must use v1beta3 {#preflight-specs-must-use-v1beta3} Application preflight checks must use API version `troubleshoot.sh/v1beta3`. Embedded Cluster v3 does not support `v1beta2` application preflight specs. In v3, you package preflight specs outside your Helm charts as a release-level YAML file. Embedded Cluster renders the spec through the Helm template engine at install time, giving your preflight spec access to the full Helm rendering context, including chart values, defaults, and helper functions. Because Embedded Cluster renders the spec through Helm, the spec uses Helm template syntax (not `repl{{ }}` Replicated template syntax). v1beta3 supports this model because it treats the spec as Helm template YAML rather than a static Kubernetes resource. :::note **You do not need to migrate host preflight, support bundle, or Redactor specs.** The `v1beta3` requirement applies only to the Preflight custom resource. HostPreflight, SupportBundle, and Redactor specs continue to work with `v1beta2` in Embedded Cluster v3. There is no `v1beta3` version of the HostPreflight custom resource. For more information about host preflight checks, see [Embedded Cluster host preflight checks](embedded-overview#about-host-preflight-checks). For more information about where redactors live in v3, see [Add custom redactors](embedded-using#redactors). ::: **Key differences from v1beta2 for application preflight specs:** - The spec lives outside your Helm chart, at the release level. You do not need to wrap it in a Secret or include it in your chart's `templates/` directory. - The spec uses Helm template syntax (`{{ .Values.something }}`, `{{ include "helper" . }}`) instead of `repl{{ }}` Replicated template syntax. - The spec is not valid YAML before rendering. Standard YAML linters will not work on it directly. - For releases with multiple Helm charts, you can use `{{ if eq .Chart.Name "my-chart" }}` conditionals to gate specific collectors or analyzers to the chart they belong to. Embedded Cluster renders the spec one time per chart in the release, each time in that chart's context. - The connectivity collectors (`http`, `postgres`, `mysql`, `mssql`, `redis`, and `clickhouse`) run wherever the preflight process runs, which is the host in Embedded Cluster v3, rather than inside the cluster as they did under KOTS. To run these checks from within the cluster, wrap them in a `runPod` collector. See [Run connectivity checks inside the cluster](embedded-using#in-cluster-collectors). - For air gap installations, images referenced by the preflight spec (such as the Troubleshoot image used to run checks inside the cluster) are not automatically included in the air gap bundle and must be added to the Application `additionalImages` field. See [Include preflight images in air gap bundles](embedded-using#preflight-air-gap-images). For details about the rendering pipeline and multi-chart examples, see [Add preflight checks](embedded-using#preflights). For more information about the v1beta3 spec format, see [v1beta3 overview](https://troubleshoot.sh/docs/preflight/v1beta3-overview) in the Troubleshoot documentation. ### Package your application with Helm To use Embedded Cluster v3, package your application as one or more Helm charts. Embedded Cluster v3 does not support Kustomize or Kubernetes manifests. ### HelmChart v2 required Embedded Cluster v3 supports installing Helm charts with a corresponding HelmChart v2 custom resource (API version `v1beta2`). It does not support HelmChart v1. Extension charts defined in the Embedded Cluster Config do not need a separate HelmChart CR — the extension schema in the Config serves the same purpose. ### Helm extension charts must be bundled in the release {#extension-charts} In Embedded Cluster v2, Helm extensions defined under `extensions.helm.charts` were pulled at install time from an HTTP Helm repository or an OCI registry, based on the `chartname` and `version` fields (and any repositories listed under `extensions.helm.repositories`). Embedded Cluster v3 does not pull extension charts at install time from any source, whether an HTTP Helm repository or an OCI registry. Instead, it resolves each extension chart from a `.tgz` chart archive that is bundled in the release, matching by chart name and version. The v3 `extensions.helmCharts` schema has no repository or registry field. As a result, upgrading a repo-sourced or OCI-sourced v2 extension is not only a change to the Config YAML shape. You must also package the chart as a `.tgz` archive and include it in the release. For the steps, see [Update your release to Embedded Cluster v3](#update-your-release-to-embedded-cluster-v3) below. ### Config value defaults are persisted at install time In Embedded Cluster v2 (KOTS), only config values explicitly set by the end customer were persisted. Default values from the Config spec were re-evaluated on each release, so changing a `default` field in a new release would take effect for any field the customer hadn't explicitly set. In Embedded Cluster v3, all config values, including those using spec defaults, are resolved and persisted at install time. On upgrade, previously persisted values take precedence over new defaults in the release. This means changing a `default` field in a new release will **not** take effect on existing installations. If you have config fields that need to be vendor-controlled and updateable across releases, use the `value` field on readonly or disabled items instead of `default`. The `value` field is re-evaluated from the Config spec on each release. ### Automatic authentication to the Replicated proxy registry If your application uses the Replicated proxy registry, Embedded Cluster v3 configures the cluster to automatically authenticate to the proxy registry for all pods. This means that it's no longer necessary to manually inject a Replicated pull secret using the ImagePullSecretName template function. ### Changes to Application custom resource fields {#application-cr-changes} Because Embedded Cluster v3 removes the KOTS Admin Console, several fields in the [Application custom resource](/reference/custom-resource-application) no longer have a meaningful equivalent in v3. These fields can be safely removed from your Application manifest during the upgrade. If left in place, they are silently ignored. The following Application spec fields are not used in Embedded Cluster v3: | Field | Purpose in Embedded Cluster v2 (KOTS) | |---|---| | `allowRollback` | Enabled the rollback button in the Admin Console | | `graphs` | Displayed Prometheus metric graphs in the Admin Console | | `minKotsVersion` / `targetKotsVersion` | Gated installations to specific KOTS versions | | `requireMinimalRBACPrivileges` / `supportMinimalRBACPrivileges` | Controlled KOTS operator RBAC mode | | `proxyPublicImages` | Routed public image pulls through the Replicated proxy registry via KOTS | | `consoleFeatureFlags` | Enabled feature flags in the Admin Console UI | | `additionalNamespaces` | Granted KOTS cross-namespace RBAC access | The following Application spec fields **are still used** in Embedded Cluster v3: | Field | How it is used in Embedded Cluster v3 | |---|---| | `title` | Displayed in the Embedded Cluster install and upgrade wizard header. See [Customize the wizard branding](embedded-using#branding). | | `icon` | Displayed next to the title in the wizard. For air gap installations, use a Base64 encoded image since remote URLs are not accessible. See [Customize the wizard branding](embedded-using#branding). | | `releaseNotes` | Used as the default release notes for the channel release when no notes are provided at promote time. See [Include release notes](/vendor/releases-creating-cli#release-notes). | | `additionalImages` | Images listed here are included in air gap bundles, even if not directly referenced in Helm chart templates. | | `excludedImages` | Images listed here are excluded from air gap bundles. | | `ports` | Configures port-forward shortcuts in the Embedded Cluster install and upgrade wizard. | | `statusInformers` | Injected into the Replicated SDK Helm values for instance reporting in the Vendor Portal. Requires the [Replicated SDK](/vendor/replicated-sdk-overview). | ## Update your release to Embedded Cluster v3 To update a release from Embedded Cluster v2 to v3: 1. Remove any standalone Kubernetes manifests or `kustomization.yaml` files from your release. Embedded Cluster v3 only deploys resources defined in Helm charts. If you need to deploy any resources before Embedded Cluster deploys your application, you can use Helm chart [`extensions`](embedded-config#extensions) in the Embedded Cluster Config. 1. In your application Helm chart `Chart.yaml` file, add the SDK as a dependency. With Embedded Cluster v3, you must include the SDK to get instance insights from status informers. If your application uses multiple charts, declare the SDK as a dependency of the chart that customers install first. Do not declare the SDK in more than one chart. ```yaml # Chart.yaml dependencies: - name: replicated repository: oci://proxy.replicated.com/library version: 1.19.6 ``` For the latest version information for the Replicated SDK, see the [replicated-sdk repository](https://github.com/replicatedhq/replicated-sdk/releases) in GitHub. 1. If you use the Replicated proxy registry, update your release to remove the ImagePullSecretName template function from your HelmChart `values` key. Embedded Cluster v3 configures the cluster to automatically authenticate to the proxy registry for all pods, so you don't need to manually inject a pull secret. See the _Embedded Cluster v3_ steps in [Configure your application to use the proxy registry](/vendor/private-images-kots#configure-v3). 1. If you support air gap installations, update your image references to use the Embedded Cluster ReplicatedImageName and ReplicatedImageRegistry template functions. This ensures that image references resolve correctly in both online and air gap installations. See [Add support for air gap installations](embedded-using#local-image-registry). :::note In Embedded Cluster v3, these template functions replace the [LocalImageName](/reference/template-functions-config-context#localimagename), [LocalRegistryHost](/reference/template-functions-config-context#localregistryhost), and [LocalRegistryNamespace](/reference/template-functions-config-context#localregistrynamespace) template functions. ::: 1. Update your application preflight specs to API version `troubleshoot.sh/v1beta3`: - Change `apiVersion` from `troubleshoot.sh/v1beta2` to `troubleshoot.sh/v1beta3`. - Move the spec file out of your Helm chart's `templates/` directory. In v3, the preflight spec is a release-level file, not part of any chart. - If you wrapped your spec in a Kubernetes Secret (the v1beta2 workaround for chart packaging), remove the Secret wrapper. The spec should be a plain `troubleshoot.sh/v1beta3 Preflight` resource. - Replace any `repl{{ }}` Replicated template syntax with Helm template syntax. For example, use `{{ .Values.image.tag }}` instead of `repl{{ ConfigOption "image_tag" }}`. - You do not need to update the API version of support bundle or Redactor specs. They continue to work with `v1beta2` in Embedded Cluster v3. If your release needs to support both KOTS and Embedded Cluster v3 installations during the transition period, include two preflight specs: a `v1beta2` spec for KOTS and a `v1beta3` spec for Embedded Cluster v3. KOTS ignores the `v1beta3` spec, and Embedded Cluster v3 ignores the `v1beta2` spec. Once you no longer need to support KOTS installations, you can remove the `v1beta2` spec. For more information about the v1beta3 migration, see [Migrate from v1beta2 to v1beta3](https://troubleshoot.sh/docs/preflight/v1beta3-migration) in the Troubleshoot documentation. 1. Ensure that your release has a corresponding HelmChart v2 custom resource for each of your application Helm charts. Extension charts defined in the Embedded Cluster Config do not need a separate HelmChart CR. See [HelmChart v2](/reference/custom-resource-helmchart-v2). 1. In your Embedded Cluster Config, update `version` to the latest version of Embedded Cluster v3. You can also optionally increment the Kubernetes version by one minor version. Replace `EC_VERSION` with the version to install, including the Kubernetes suffix. For example, `3.12.0-beta.1+k8s-1.36`. Each entry in the [Embedded Cluster v3 release notes](/release-notes/rn-embedded-cluster-v3) lists the full version strings available for that release. **Example:** ```yaml apiVersion: embeddedcluster.replicated.com/v1beta1 kind: Config spec: version: EC_VERSION ``` 1. Update any existing Helm extensions to the v3 `extensions.helmCharts` format. See [`extensions`](embedded-config#extensions). Because Embedded Cluster v3 resolves extension charts from bundled `.tgz` archives rather than pulling them from a repository or registry (see [Helm extension charts must be bundled in the release](#extension-charts)), updating the Config YAML is not sufficient on its own. For each Helm extension you defined in v2: 1. Pull the chart to a local `.tgz` archive at the version you want to install. For a chart hosted in an HTTP Helm repository (for example, a v2 `chartname` such as `ingress-nginx/ingress-nginx`): ```bash helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm repo update helm pull ingress-nginx/ingress-nginx --version 4.11.3 ``` For a chart hosted in an OCI registry (for example, a v2 `chartname` such as `oci://quay.io/jetstack/charts/cert-manager`): ```bash helm pull oci://quay.io/jetstack/charts/cert-manager --version 1.19.2 ``` 1. Add the resulting `.tgz` archive to your release. 1. Reference the chart in the Embedded Cluster Config under `extensions.helmCharts`, using a `chart.name` and `chart.chartVersion` that match the packaged archive's `Chart.yaml`. **Example Config:** ```yaml apiVersion: embeddedcluster.replicated.com/v1beta1 kind: Config spec: extensions: helmCharts: - chart: name: ingress-nginx chartVersion: "4.11.3" releaseName: ingress-nginx namespace: ingress-nginx values: controller: service: type: NodePort nodePorts: http: "80" https: "443" image: digest: "" digestChroot: "" ``` 1. Promote the release to a development channel that you use for testing. 1. In the VM where you plan to test the installation, create a firewall opening for port 30080/TCP. Embedded Cluster v3 requires that port 30080 is open and available. 1. Test the installation using a development customer. ## Upgrade an installation from Embedded Cluster v2 to Embedded Cluster v3 Embedded Cluster supports upgrading existing installations from v2 to v3 without having to reinstall the application. The upgrade uses the same `upgrade` command as any v3-to-v3 upgrade. The v3 binary automatically detects the v2 installation and handles the upgrade, prompting the user to confirm. Running `install` on a node with an existing v2 installation is blocked — use `upgrade` instead. :::important If your v3 release uses a newer Kubernetes minor version than the customer's v2 installation, the upgrade also upgrades Kubernetes. Kubernetes supports upgrading only one minor version at a time, so Embedded Cluster blocks the upgrade if the customer's Kubernetes version is more than one minor version behind the version in your v3 release. To avoid this, ensure that customers step through the intermediate Kubernetes minor versions before they upgrade to v3. Replicated recommends that you mark each release that increases the Kubernetes minor version as required so that customers cannot skip it. For more information, see [Increase the Kubernetes minor version](embedded-config#increment-k8s). To find out which of your customers are affected, check the Kubernetes version reported by each active instance on the **Instances** tab for the customer in the Vendor Portal. For more information, see [Cluster status](/vendor/instance-insights-details#cluster) in _Instance Details_. Air gap instances report this data less frequently than online instances, so plan for air gap customers to be further behind than their last reported version shows. ::: :::note Releases that use Embedded Cluster v3 will not appear as available updates in the Embedded Cluster v2 KOTS Admin Console. This is by design to prevent accidental upgrades. Your customers will need to get the v3 release from the [Enterprise Portal](/vendor/enterprise-portal-about) or by downloading the binary directly. Plan to communicate this to your customer base when you promote a v3-enabled release. ::: To upgrade an existing installation to Embedded Cluster v3: 1. Get the customer's Embedded Cluster install instructions from the Vendor Portal or from the Enterprise Portal. 1. For application version, select the release that enables Embedded Cluster v3. 1. SSH into the VM where the Embedded Cluster v2 installation is running. 1. On the VM, run the commands to download and extract the installation assets for the target release. The installation assets include the Embedded Cluster binary, the license file, and the release assets. 1. Run the following command to upgrade using the Embedded Cluster v3 upgrade wizard: ```bash sudo ./APP_SLUG upgrade --license license.yaml ``` Where `APP_SLUG` in the unique application slug. 1. When prompted, type `yes` to confirm that you want to upgrade to v3. ```bash Detected EC v2 installation. This will upgrade the cluster from v2 to v3. This is a one-way upgrade and cannot be undone. Do you want to proceed with the upgrade? (yes/NO): yes ``` 1. When the upgrade command completes, go to the URL provided to access the upgrade wizard. ```bash Installation started. Connect to the web interface to continue the installation. Open the following URL in your browser: https://kotsadm.default.svc.cluster.local:30080 Note: You may see a browser warning for the self-signed certificate. Click "Advanced" > "Proceed" to continue. Press Ctrl+C when the installation is complete to stop the web interface. ``` 1. Log in to the upgrade wizard using the existing password for the Admin Console. ![upgrade wizard log in screen](/images/embedded-cluster-v3-slackernews-upgrade-wizard.png) [View a larger version of this image](/images/embedded-cluster-v3-slackernews-upgrade-wizard.png) 1. Follow the steps in the wizard to upgrade any other nodes in the cluster, configure the application, and then deploy the application. ![upgrade wizard app upgrade screen](/images/embedded-cluster-v3-slackernews-upgrade-progress.png) [View a larger version of this image](/images/embedded-cluster-v3-slackernews-upgrade-progress.png) 1. Press Ctrl+C when the upgrade is complete to close the wizard.