# Migrating Airflow DAGs from CDE to a CWO Git Repository

This guide walks you through migrating Airflow DAGs and their resource files from a Cloudera Data Engineering (CDE) virtual cluster into a CWO Git repository using Cloudera Migration Assistant (CMA).

## What You Will Achieve

By the end of this guide, CMA will have created a Pull Request (GitHub or GitHub Enterprise), Merge Request (GitLab), or branch push (plain Git) on the target repository. The PR/MR contains:

- Your selected Airflow DAG files, organized under `/dags/<dagId>/`.
- Any associated resource files (Python environments, supporting scripts), organized under `/include/<service>/<vc>/<pool>/`.

Your source CDE environment is not modified at any point during this process.

## Before You Begin

Before you start, confirm the following with your CMA administrator:

- Both the CDE cluster (listed as **Cloudera Data Services** platform) and the Git cluster (listed as **Git** platform) are visible on the **Clusters** page in the CMA UI.
- The CDE cluster has been configured with valid CDP Control Plane credentials.
- The Git cluster has been configured with a valid repository URL, access token, and base branch.

For setup instructions, see [Setting Up CDE to CWO Git Migration](../administrator/cma-cde-git-migration-setup.md).

---

## Step 1 — Scan the CDE Cluster

You need to scan the CDE cluster to discover all available Airflow DAGs before you can migrate them.

1. On the **Clusters** page, locate and click the cluster with the **Cloudera Data Services** platform label (its name ends with `-cde`).
2. Click **Start Scanning** to open the Scan Settings.
3. Select **Scan CDE for Airflow**.
4. Click **Scan selected**. You are redirected to the scanning progress view.
5. Wait for the scan to complete. The status indicator turns green when it finishes.

### Reading the Scan Results

After the scan completes, click the **Discovery** tab on the cluster page. Results are displayed as a three-level hierarchy:

**CDE Service → Virtual Cluster → Airflow DAGs**

Multiple CDE services and multiple virtual clusters are fully supported. Each level can be expanded and collapsed independently.

The following columns are shown for each Airflow DAG:

| Column | Description |
|---|---|
| **DAG Name** | The job name as configured in CDE. |
| **DAG ID** | The `dag_id` value from the DAG Python file (`dag_id` parameter). |
| **Cron** | The cron schedule expression, if the DAG is scheduled. |
| **Active** | Whether the DAG schedule is enabled. |
| **Operators** | Airflow operator classes detected in the DAG source (for example, `BashOperator`, `PythonOperator`). |
| **Triggers** | DAGs that this DAG triggers (outbound dependencies). |
| **Triggered By** | DAGs that trigger this DAG (inbound dependencies). |
| **Resource Pools** | CDE resource pool names that contain files mounted by this DAG. |

**Switching views:**

- Use the **Data** button to see DAGs as files (file-centric view, useful for the mapping step).
- Use the **Metadata** button to see DAGs as logical entities (metadata-centric view, useful for the mapping step).
- Use the tree/list toggle to switch between the hierarchical tree view and a flat list of all DAGs.

> **Note:** If a virtual cluster shows no DAGs, the CDE Agent may have been unable to authenticate to that virtual cluster during the scan. Check the scan command log for `SKIPPED` entries.

---

## Step 2 — Scan the Git Repository

You need to scan the Git repository so CMA can display its current file tree and detect any existing DAG files.

1. On the **Clusters** page, locate and click the cluster with the **Git** platform label (its name ends with `-git`).
2. Click **Start Scanning**.
3. Select **Git Repository Scan**.
4. Click **Scan selected** and wait for the scan to complete.

The **Discovery** tab shows the repository file tree with **Name** and **Size** columns. An empty repository is supported — CMA creates all necessary directories when the migration runs.

---

## Step 3 — Create a Migration Plan

A migration plan connects the CDE source cluster to the Git target cluster and holds all your mapping and configuration choices.

1. Navigate to the **Plans** page.
2. Click **Create Plan**.
3. Fill in the plan details:
   - **Name** — used as the Git branch name prefix (`cma/<planName>`). Choose a name that is valid as a Git branch name (no spaces or special characters).
   - **Source cluster** — select the cluster with the **Cloudera Data Services** platform (the `-cde` cluster).
   - **Target cluster** — select the cluster with the **Git** platform (the `-git` cluster).
   - **Transition type** — select **CDE DAGs to Git Repository**.
4. Click **Create**. The plan opens on its detail page.

---

## Step 4 — Label DAGs for Migration (Collections Tab)

Labels control which DAGs are included in the migration. Only labeled DAGs appear on the Data and Metadata tabs for mapping. You can create multiple collections to organize DAGs for phased migration.

1. Inside the migration plan, click the **Collections** tab.
2. Browse the CDE service and virtual cluster hierarchy.
3. Select the DAGs you want to migrate.
4. Click **Add to collection** to assign the **Default** label, or click **+** next to **Collections** to create a new collection with a custom name and color.
5. Repeat until all DAGs you want to migrate are labeled.

> **Tip:** You can return to this tab at any time to add or remove DAGs from collections. Changes take effect immediately on the Data and Metadata tabs.

---

## Step 5 — Map DAGs to the Git Repository (Metadata Tab)

The **Metadata** tab is the primary place for creating DAG mappings. It shows a split view of CDE source DAGs (left) and target Git DAGs (right), and lets you map them with a single action.

1. Inside the plan, click the **Metadata** tab.
2. Map DAGs using either method:
   - **One by one** — click the **Map** button on an individual DAG row to map it immediately.
   - **In bulk** — select one or more unmapped DAGs, then click **Copy to Git** in the toolbar.
3. CMA automatically creates mappings with the following default target paths:
   - DAG files: `/dags/<dagId>/<dagId>.py`
   - Resource files: `/include/<service>/<vc>/<pool>/<file>`
4. The mapped DAGs move to **mapped** status. Corresponding target DAG entries appear in the right panel.

**To remove a mapping:**

1. In the left panel, click a mapped DAG to focus it.
2. Click **Unmap** in the toolbar. The mapping and all associated target entries are removed (shared resource files are preserved).

> **Warning:** If two different virtual clusters contain DAGs with the same `dag_id`, CMA shows an error badge on the affected DAGs and blocks execution creation. You must resolve all duplicate DAG ID conflicts in Step 7 before you can run the migration.

---

## Step 6 — Verify File Mappings (Data Tab)

The **Data** tab shows a file-level split view of the CDE source file tree (left) and the Git repository file tree (right). Use it to review the target file paths that were created during the mapping in Step 5.

1. Click the **Data** tab.
2. Confirm that each mapped DAG and its resource files appear in the right panel at the expected paths:
   - DAG files: `/dags/<dagId>/<dagId>.py`
   - Resource files: `/include/<service>/<vc>/<pool>/<file>`

No mapping actions are needed here. If the paths look incorrect, return to the **Metadata** tab to unmap and remap the affected DAG.

---

## Step 7 — Review and Edit DAG Content (Workloads Tab)

Use this tab to review the DAG Python source code that will be pushed to Git and to resolve any duplicate DAG ID conflicts.

> **Required** if CMA shows an error badge for duplicate DAG IDs. Execution creation is blocked until all conflicts are resolved.

1. Click the **Workloads** tab. For each mapped DAG, the tab shows a Monaco diff editor:
   - **Left side (read-only)** — original DAG content as scanned from CDE.
   - **Right side (editable)** — content that will be pushed to Git. Initially identical to the source.
2. Click any DAG in the list on the left to open it in the editor.
3. Edit the right side as needed. Common reasons to edit:
   - Rename a `dag_id` to resolve a duplicate conflict.
   - Update connection IDs or variable references for the target environment.
   - Remove or update imports for operators not available in the target Airflow version.
4. Click **Save** after editing. CMA saves the edited content.

> **Important:** If you change the `dag_id` value in the editor, CMA automatically renames the target file path to match (`/dags/<newDagId>/<newDagId>.py`). Once all duplicates are resolved, the error badge clears and execution creation becomes available.

---

## Step 8 — Create and Run the Execution

When your DAGs are mapped and reviewed, create an execution to push them to Git.

1. Click the **Execution** tab inside the plan.
2. Click **Create Execution**.
3. The execution runs three steps automatically. Each step must complete successfully before the next one starts.

### Execution Steps

**Step 1 — Upload CDE Resource Files**

The CDE Agent reads the resource files (Python environments, supporting scripts) from the local disk where they were saved during the scan, then uploads them to CMA Master's staging area.

> **Note:** This step is skipped if no DAGs with resource files are mapped.

**Step 2 — Prepare CDE Resources**

CMA Master writes all DAG content — including any edits you made in Step 7 — to the staging area alongside the resource files. Edited content takes priority over the original scanned content.

**Step 3 — Create Git Change**

The Git Agent downloads all staged files from CMA Master, then calls the Git provider API to:
- Create a new branch named `cma/<planName>` from the base branch.
- Commit all DAG and resource files to that branch.
- Open a Pull Request (GitHub), Merge Request (GitLab), or push to a branch (plain Git fallback).

---

## Step 9 — View the Result

After Step 3 completes:

1. Click the completed **Create Git Change** step to expand its details.
2. Click the result link. The link opens the Pull Request, Merge Request, or branch in your Git provider.
3. Review the changes in your Git provider and merge the PR/MR when ready.

> **After merging:** The Airflow DAG files are now in the target Git repository on the base branch. Your source CDE environment is unchanged.

---

## Known Issues

### AMCM-4227 CDE execution includes all plan mappings instead of only selected collection

When you create an execution and select a single collection, the resulting Git PR/MR contains DAG files from **all collections** in the plan, not just the one you selected.

**Workaround:** Create a separate migration plan for each collection you want to migrate independently.

---

## Troubleshooting

| Symptom | Likely Cause | Resolution |
|---|---|---|
| No DAGs appear on the Discovery tab after CDE scan | The scan did not complete, or no DAGs are labeled. | Check the scan command log. If labeled DAGs are missing, verify that the label filter is not active. |
| A virtual cluster shows no DAGs | Knox authentication failed for that VC during the scan. | Check the scan log for `SKIPPED` entries. Verify the Cloudera Manager credentials with your CMA administrator. |
| Duplicate DAG ID warning on the Data or Workloads tab | Two VCs have DAGs with the same `dag_id`. | Rename one `dag_id` in the Workloads tab editor and save. |
| Execution Step 1 fails | Resource files from the CDE scan are no longer available on the Agent's disk. | Re-run the CDE scan to re-download resource files, then create a new execution. |
| Execution Step 3 fails with Git authentication error | The access token is expired or lacks permissions. | Ask your CMA administrator to update `CMA_AGENT_CLUSTER_GIT_TOKEN` in the safety valve and restart the CMA Agent. |
| Execution Step 3 fails — base branch not found | The base branch configured in `CMA_AGENT_CLUSTER_GIT_BRANCH` was deleted or renamed. | Ask your CMA administrator to update the branch configuration and restart the CMA Agent. |
| The PR/MR link does not appear after execution | The CMA Master log may contain details of a Git provider error. | Check the CMA Master log and verify that the Git repository is accessible from the Git Agent. |

## Related Documentation

- [Setting Up CDE to CWO Git Migration](../administrator/cma-cde-git-migration-setup.md)
- [Registering Clusters](cma-cluster-registration.md)
- [Scanning Clusters](cma-scanning.md)
- [Creating Collections](cma-collections.md)
