# Setting Up CDE to CWO Git Migration

Cloudera Migration Assistant (CMA) can migrate Airflow DAGs from Cloudera Data Engineering (CDE) virtual clusters into version-controlled CWO Git repositories. The migration produces a Pull Request (GitHub), Merge Request (GitLab), or branch (plain Git), which you review and merge — the source CDE environment is never modified.

Before migration engineers can use this feature, you must configure the CMA Agent with credentials for both the CDP Control Plane (CDE source) and the target CWO Git repository. CMA then automatically registers the required clusters.

## How It Works

When the CMA Agent starts with the required configuration, it automatically registers two additional managed clusters in CMA Master:

- **CDE cluster** (`<clusterName>-cde`) — represents the Cloudera Data Engineering environment. The agent uses CDP API credentials to discover CDE services, virtual clusters, and Airflow DAGs.
- **Git cluster** (`<clusterName>-git`) — represents the target CWO Git repository. The agent uses the repository URL and access token to scan the repository and create Pull Requests or Merge Requests.

Both clusters appear in the CMA Master UI and can be used as source and target in a migration plan.

## Prerequisites

Before you begin:

- CMA is deployed via the parcel deployment model. For more information, see [Deploy the CMA server](cma-server-setup.md).
- The CMA Agent service is installed and running on the cluster (Public Cloud or CDP Private Cloud Base).
- The CMA Agent has network access to the CDP Control Plane API endpoint.
- The CMA Agent has network access to the CDE.
- The CMA Agent has outbound HTTPS access to the Git provider (GitHub, GitLab, or self-hosted).
- If the Git provider uses a custom or self-signed TLS certificate, you have access to the CMA Agent truststore file and its password.
- You have CDP access key credentials (access key ID and private key) for the Control Plane.
- You have a personal access token (PAT) for the target CWO Git repository with permissions to read the repository and create Pull Requests or Merge Requests.
- The target CWO Git repository exists and the base branch (for example, `main`) is present.

## Step 1 — Enable the Feature Flag

The CDE to CWO Git migration feature is controlled by the `cma.cde-git-migration-enabled` feature flag. It is **disabled by default**. Perform this step only if the flag was previously disabled.

1. Open the CMA Master UI and navigate to **Settings**.
2. Click the **Feature Flags** tab.
3. Locate the flag **cma.cde-git-migration-enabled** and set it to **true**.
4. Click **Save**.

## Step 2 — Configure the CMA Agent Safety Valve

CDE and Git credentials are passed to the CMA Agent through the Cloudera Manager safety valve `CMA_AGENT_SERVER_role_env_safety_valve`. These environment variables are read at startup; the agent uses them to register the CDE and Git clusters.

In Cloudera Manager on the cluster where the CMA Agent is deployed:

1. Navigate to the **CMA_AGENT** service.
2. Open the **Configuration** tab.
3. Search for **`CMA_AGENT_SERVER_role_env_safety_valve`**.
4. Add the environment variables listed in the sections below.

### CDP Control Plane Settings

These settings allow the CDE Agent to authenticate with the CDP Control Plane and the CDE Knox gateway.

| Variable | Description | Example |
|---|---|---|
| `CMA_AGENT_CLUSTER_CLOUD_CONTROL_DATACENTER` | CDP Control Plane API endpoint URL. | `https://api.us-west-1.cdp.cloudera.com` |
| `CMA_AGENT_CLUSTER_CLOUD_CONTROL_ACCESS_KEY_ID` | CDP access key ID for Control Plane authentication. Obtained from the CDP Management Console under **User Management > Access Keys**. | `a1b2c3d4-...` |
| `CMA_AGENT_CLUSTER_CLOUD_CONTROL_PRIVATE_KEY` | CDP private key corresponding to the access key ID. | `<YOUR_CDP_PRIVATE_KEY>` |
| `CMA_AGENT_CLUSTER_CLOUD_CONTROL_USERNAME` | CDE username. Must be an **LDAP user**. Used by the CDE Agent to authenticate against the CDE Knox gateway when fetching DAG metadata. | `admin` |
| `CMA_AGENT_CLUSTER_CLOUD_CONTROL_PASSWORD` | CDE password corresponding to the username above. | `<CDE_PASSWORD>` |

> **Note:** `CMA_AGENT_CLUSTER_CLOUD_CONTROL_USERNAME` and `CMA_AGENT_CLUSTER_CLOUD_CONTROL_PASSWORD` are CDE credentials, not CDP credentials. They are used only for Knox authentication against CDE virtual cluster APIs.

### CWO Git Repository Settings

These settings allow the Git Agent to scan the target repository and create Pull Requests or Merge Requests.

| Variable | Description | Example |
|---|---|---|
| `CMA_AGENT_CLUSTER_GIT_REPOURL` | HTTPS URL of the target Git repository. SSH URLs are not supported. | `https://github.com/your-org/your-repo.git` |
| `CMA_AGENT_CLUSTER_GIT_TOKEN` | Personal access token for the Git repository. For GitHub, the token requires the `repo` scope. For GitLab, it requires `api` or `read_repository` + `write_repository`. | `<YOUR_GIT_PAT>` |
| `CMA_AGENT_CLUSTER_GIT_BRANCH` | Base branch name. CMA creates a source branch named `cma/<planName>` from this branch and opens a Pull Request or Merge Request back to it. Defaults to `main` if not set. | `main` |

## Step 3 — (Optional) Import a Custom SSL Certificate

If your Git provider uses a custom or self-signed TLS certificate (for example, a self-hosted GitHub Enterprise or GitLab instance with an internal certificate authority), the CMA Agent must trust that certificate before it can establish HTTPS connections to the provider.

Perform this step on the node where the CMA Agent is installed, before saving and restarting the agent in Step 4.

1. Log in to the node where the CMA Agent parcel is installed.

2. Locate the certificate import script:

   ```
   /opt/cloudera/parcels/cma/bin/download-ssl-cert.sh
   ```

3. Run the script to download the server certificate and import it into the CMA Agent truststore:

   ```bash
   /opt/cloudera/parcels/cma/bin/download-ssl-cert.sh \
     -s <host:port> \
     -t <truststore-path> \
     -p <truststore-password> \
     -f <truststore-format> \
     -a <alias>
   ```

   The following table describes the script parameters:

   | Parameter | Description | Example                                     |
   |---|---|---------------------------------------------|
   | `-s <host:port>` | Hostname and port of the Git provider to download the certificate from. | `github.infra.cloudera.com:443`             |
   | `-t <truststore-path>` | Absolute path to the CMA Agent truststore file where the certificate will be imported. | `/opt/cloudera/security/cma-truststore.jks` |
   | `-p <truststore-password>` | Password for the truststore file. | `trustStorePasswd`                        |
   | `-f <truststore-format>` | Truststore format. Use `JKS` for a Java KeyStore. | `JKS`                                       |
   | `-a <alias>` | Alias under which the certificate is stored in the truststore. Choose a short, descriptive name. | `github`                                    |

   **Example:**

   ```bash
   /opt/cloudera/parcels/cma/bin/download-ssl-cert.sh \
     -s github.infra.cloudera.com:443 \
     -t /opt/cloudera/security/cma-truststore.jks \
     -p trustStorePasswd \
     -f JKS \
     -a github
   ```

4. Verify that the certificate was imported successfully by checking the script output for a confirmation message.

> **Note:** Run this step once for each Git provider host that uses a certificate not already trusted by the default Java truststore. You do not need to repeat this step after subsequent CMA Agent restarts, unless the server certificate is replaced.

## Step 4 — Save and Restart the CMA Agent

1. Click **Save** to save the safety valve configuration.
2. Restart the CMA Agent service.

## Step 5 — Verify Cluster Registration

After the CMA Agent restarts, it registers both clusters with CMA Master automatically. No manual steps are required in the CMA UI.

To verify:

1. Open the CMA Master UI and navigate to **Clusters**.
2. Confirm that two new clusters appear:

   | Cluster Name | Platform Label | Purpose |
   |---|---|---|
   | `<clusterName>-cde` | Cloudera Data Services | CDE source — used for scanning DAGs |
   | `<clusterName>-git` | Git | Git target — used for scanning the repo and creating PRs/MRs |

3. If the clusters do not appear within a few minutes, check the CMA Agent log for registration errors. See [Troubleshooting](#troubleshooting).

## CWO Git Provider Auto-Detection

CMA automatically selects the correct Git provider based on the repository URL. You do not need to configure the provider type manually in most cases.

| Detection Rule | Provider | Change Type |
|---|---|---|
| URL contains `github.com` | GitHub | Pull Request |
| URL contains `gitlab.com` | GitLab | Merge Request |
| HTTP probe responds to `/api/v4/version` | GitLab (self-hosted) | Merge Request |
| HTTP probe responds to `/api/v3/meta` | GitHub Enterprise | Pull Request |
| None of the above | Plain Git | Branch push |

To override auto-detection, set the `git.providerType` cluster configuration property to one of `GITHUB`, `GITLAB`, or `JGIT`.

> **Note:** Plain Git (the fallback provider) only supports HTTPS. SSH-based Git URLs are not supported.

## Network Requirements

If a firewall is in place between CMA components and external services, the following ports must be open:

| Source | Destination | Port | Protocol | Purpose |
|---|---|---|---|---|
| CMA Agent (CDE) | CDP Control Plane API | 443 | HTTPS | Discover CDE services and virtual clusters |
| CMA Agent (CDE) | CDE Knox gateway | 443 | HTTPS | Fetch DAG metadata and download DAG files |
| CMA Agent (Git) | GitHub / GitLab / self-hosted Git | 443 | HTTPS | Scan repository, create PRs/MRs |
| CMA Agent (CDE and Git) | CMA Master gRPC port | 8091 | gRPC | Agent-to-Master communication |

## Known Limitations

- **No SSH support**: For plain Git HTTPS repository URLs are supported only.
- **No incremental migration**: Each migration execution creates a full PR/MR containing all mapped DAGs.
- **No automated rollback**: Once a PR/MR is created, it must be manually closed or reverted in the Git provider.
- **Single branch per plan**: All DAGs in a migration plan target one source branch (`cma/<planName>`).

## Troubleshooting

| Symptom | Likely Cause | Resolution |
|---|---|---|
| CDE or Git clusters do not appear in the CMA UI | CMA Agent did not restart, or safety valve variables are missing or misspelled. | Restart the CMA Agent service and check the Agent log for `Cluster registration failed` entries. |
| CDE scan fails with authentication error | `CMA_AGENT_CLUSTER_CLOUD_CONTROL_USERNAME` / `CMA_AGENT_CLUSTER_CLOUD_CONTROL_PASSWORD` are incorrect, or the CDE Knox gateway is unreachable. | Verify that the Cloudera Manager credentials are correct and that the Agent can reach the CDE Knox gateway on port 443. |
| CDE scan fails — no CDE services found | `CMA_AGENT_CLUSTER_CLOUD_CONTROL_DATACENTER` points to the wrong region, or the access key credentials are invalid. | Verify the Control Plane URL and confirm the access key is valid in the CDP Management Console. |
| Git scan fails with SSL/TLS handshake error or certificate validation failure | The Git provider uses a custom or self-signed certificate that is not trusted by the CMA Agent. | Import the server certificate into the CMA Agent truststore using `download-ssl-cert.sh`. See [Step 3 — (Optional) Import a Custom SSL Certificate](#step-3--optional-import-a-custom-ssl-certificate). |
| Git scan fails with authentication error | `CMA_AGENT_CLUSTER_GIT_TOKEN` is missing, expired, or lacks the required permissions. | Regenerate the PAT in the Git provider and update `CMA_AGENT_CLUSTER_GIT_TOKEN` in the safety valve. |
| Git scan fails — branch not found | `CMA_AGENT_CLUSTER_GIT_BRANCH` refers to a branch that does not exist. | Create the branch in the Git repository, or update the variable to point to an existing branch. |
| CDE cluster registered but no VCs appear after scan | A specific virtual cluster could not be reached (Knox auth failure or empty `vcApiUrl`). | Check the CDE scan command log in the CMA UI for `SKIPPED` entries with the affected VC name. |

## Related Documentation

- [CMA Deployment Architecture Overview](cma-deployment-architecture.md)
- [Deploy the CMA server](cma-server-setup.md)
- [Registering Clusters](../migration/cma-cluster-registration.md)
- [Migrating Airflow DAGs from CDE to a CWO Git Repository](../migration/cma-cde-to-airflow-v3-migration.md)
