# Troubleshooting

## CMA Agent Issues

### Agent port conflict (port 9090 already in use)

**Symptom:** Agent fails to start. Log shows `Web server failed to start. Port 9090 was already in use.`

**Cause:** Port 9090 commonly conflicts with **HBase ThriftServer** on the same host.

**Fix:** Change the agent HTTP port in Cloudera Manager:
1. Go to the CMA Agent service configuration
2. Set `cma_agent_http_port` to an unused port (e.g., `9190`)
3. Set `cma_agent_https_port` to `9193`
4. Restart the CMA Agent service

### Agent not connecting to Master

**Symptom:** Agent health shows `OUT_OF_SERVICE`. Agent log shows connection errors or `Initializing connection to master` without a subsequent `Sending agent instance` message.

**Possible causes and fixes:**

| Cause | Fix |
|-------|-----|
| `cma_master_url` points to the UI/gateway port instead of the gRPC port | The agent needs the **gRPC port** (default 8091), not the gateway port (8090). Set `cma_master_url` to `http://<master-host>:8091`. |
| Firewall blocking gRPC port | Ensure outbound connectivity from the agent host to `master-host:8091`. Test with: `timeout 5 bash -c '</dev/tcp/<master-host>/8091' && echo OPEN || echo CLOSED` |
| Master using TLS but agent URL uses `http://` | Change `cma_master_url` to `https://` |
| Master gRPC port was changed | Check the actual gRPC port on the master host: `grep grpc /var/run/cloudera-scm-agent/process/*-cma_master-CMA_SERVER/conf/cma-server.properties` |

### CMA_AGENT service type not available in Cloudera Manager

**Cause:** The CMA CSD is not installed or CM was not restarted after CSD installation.

**Fix:**
1. Copy the CSD JAR to `/opt/cloudera/csd/` on the CM host
2. Set ownership: `chown cloudera-scm:cloudera-scm /opt/cloudera/csd/CMA-*.jar`
3. Set permissions: `chmod 640 /opt/cloudera/csd/CMA-*.jar`
4. Restart Cloudera Manager Server: `systemctl restart cloudera-scm-server`
5. Wait for CM to come back up (~90-120 seconds)

### Agent starts but cluster does not appear in CMA Master UI

**Possible causes:**
- Agent is still connecting (allow ~30 seconds after agent start for registration)
- Check agent log for `Sending agent instance` message
- Verify the CM credentials configured for the agent are correct (agent uses them to query Cloudera Manager for cluster metadata)

## Stuck Commands

### Scan or migration commands stuck in IN_PROGRESS state

**Symptom:** A scan or migration command remains in `IN_PROGRESS` state indefinitely. Cancelling via the UI or API returns success but the state does not change.

**Cause:** There is a known issue where the cancel operation does not always persist the state change to the database.

**Fix (H2 database, parcel deployment):**

```shell
ssh root@<master-host> '
H2_JAR=$(ls /opt/cloudera/parcels/cma/lib/h2-*.jar)
DB_URL="jdbc:h2:/var/lib/cma/data/db/cma-server;IFEXISTS=TRUE;AUTO_SERVER=TRUE"
java -cp $H2_JAR org.h2.tools.Shell -url "$DB_URL" -user admin -password s3cret \
  -sql "UPDATE CLUSTER_COMMAND_EXECUTION SET STATE = '"'"'FAILED'"'"' WHERE STATE = '"'"'IN_PROGRESS'"'"';"
'
```

To update only specific commands, use `WHERE ID IN (9, 10, 11)` instead.

**Fix (local/Docker deployment):**

The H2 database is located at `~/.cma/cma-server/data/db/cma-server`. Use the H2 JAR from the CMA installation directory.

## CMA Upgrade Issues

### Liquibase checksum failure after upgrade

**Symptom:** After a CMA parcel upgrade, one or more CMA roles show `healthSummary: BAD`. The application log contains:
```
liquibase.exception.ValidationFailedException: Validation Failed:
     N changesets check sum
```

**Cause:** The new version of CMA modified database schema files. The existing database stores checksums from the previous build that no longer match.

**Fix Option 1 — Clean database wipe (dev/test environments):**

> **Warning:** This deletes all CMA data including migration plans, execution history, cluster registrations, and scan results.

1. Stop all CMA services (agents first, then master)
2. On the master host: `sudo -u cma rm -f /var/lib/cma/data/db/cma-server.* /var/lib/cma/data/db/cma-auth.*`
3. On each agent host: `rm -f /var/lib/cma-agent/data/db/cma-agent.*`
4. Start CMA Master first, then start all CMA Agent services

> **Important:** If you wipe the master database, you **must** also wipe all agent databases. Agents store references to cluster and credential records that no longer exist in the fresh master database.

**Fix Option 2 — Clear specific checksums (preserves data):**

1. Stop the failing role
2. SSH to the host and clear the stale checksums:

```shell
java -cp /opt/cloudera/parcels/cma/lib/h2-*.jar org.h2.tools.Shell \
  -url 'jdbc:h2:/var/lib/cma/data/db/cma-server;IFEXISTS=TRUE;AUTO_SERVER=TRUE' \
  -user admin -password s3cret \
  -sql "UPDATE DATABASECHANGELOG SET MD5SUM = NULL WHERE ID IN ('<changeset-id-1>', '<changeset-id-2>');"
```

3. Start the role. Liquibase will recalculate the checksums on startup.

### Restart order for multi-cluster upgrades

When upgrading CMA across multiple clusters, restart services in this order:
1. Restart CMA Agent on **all** clusters first
2. Restart CMA Master **last** (so agents are ready to reconnect when the Master's gRPC server comes back)

### CMA Master slow to start

**Symptom:** CMA Master takes a long time to start or appears unresponsive after starting.

**Cause:** Liquibase database migration runs on first startup after an upgrade. This can take 60-90 seconds.

**Fix:** Wait for the startup to complete. Check `/var/log/cma/cma-server.log` for progress.

## Database and Data Cleanup

### H2 database locations and credentials

| Role | Database Path | Default Credentials |
|------|---------------|---------------------|
| CMA Master Server | `/var/lib/cma/data/db/cma-server` | `admin` / `s3cret` |
| CMA Auth Server | `/var/lib/cma/data/db/cma-auth` | `root` / `root` |
| CMA Agent Server | `/var/lib/cma-agent/data/db/cma-agent` | `admin` / `s3cret` |

Production deployments should override credentials via the `CMA_DBA_USER` and `CMA_DBA_PASS` environment variables.

### Cleaning up migration data

To delete migration plans, transitions, and RM replication policies, follow this order:
1. Delete RM replication policies on the **target** cluster via the CM API (not the RM API)
2. Clean the H2 database on the master host (disable referential integrity for FK chains)
3. **Restart CMA Master** after cleanup — the master repopulates essential plugin data from YAML config files on startup

### Cleaning up RM replication policies

Use the **CM API** to list and delete replication policies:

```shell
# List policies
curl -s -u admin:admin \
  "http://<target-cm-host>:7180/api/v49/clusters/Cluster%201/services/HDFS-1/replications"

# Delete a policy
curl -s -X DELETE -u admin:admin \
  "http://<target-cm-host>:7180/api/v49/clusters/Cluster%201/services/HDFS-1/replications/<id>"
```

> **Note:** Do not use the RM API (`/dmx/api/policies`) for deletion — it fails with 404 due to pipe characters in policy IDs.

## Authentication Issues

### Default credentials

| Service | Credentials | Notes |
|---------|-------------|-------|
| CMA Web UI | `admin` / `admin` | Login via gateway on port 8090 |
| CMA Auth Server | `admin` / `admin` | Direct access on port 9000 |
| CMA Agent API | `admin` / `admin` | HTTP Basic Auth |
| H2 Database (server) | `admin` / `s3cret` | Override via `CMA_DBA_USER` / `CMA_DBA_PASS` |

### JWT token issuer mismatch when calling CMA API remotely

**Symptom:** API calls return `WWW-Authenticate: Bearer error="invalid_token", error_description="The iss claim is not valid"`.

**Cause:** The Auth Server sets `iss: http://localhost:<port>` in the JWT. When calling from a different host, the Master rejects the token because the issuer does not match.

**Workaround:** Run API calls on the master host itself via SSH so `localhost` matches.

## Vault Troubleshooting

### Vault token file accidentally deleted or empty

After the first successful initialization, the startup script creates a backup of the token file in the user's home folder with the name **.cma-server-vault**. Replace the token file with the backup and restart Vault server.

**Local mode:**
```shell
cp ~/.cma-server_vault ~/.cma/cma-server/conf/.vault
```

**Parcel mode:**
```shell
cp ~/.cma-server_vault /var/lib/cma/conf/.vault
```

## Log Locations

### Parcel deployment (Cloudera Manager)

| Service | Log Path |
|---------|----------|
| CMA Master Server | `/var/log/cma/cma-server.log` |
| CMA Agent Server | `/var/log/cma-agent/cma-agent-server.log` |
| CMA Auth Server | `/var/log/cma/cma-auth-server.log` |
| CMA Gateway Server | `/var/log/cma/cma-gateway-server.log` |

### Local / Docker deployment

| Service | Log Path |
|---------|----------|
| Master | `~/.cma/cma-server/logs/cma-server.log` |
| Auth Server | `~/.cma/cma-server/logs/cma-auth-server.log` |
| Gateway Server | `~/.cma/cma-server/logs/cma-gateway-server.log` |

## Key File Paths (Parcel Deployment)

| Path | Description |
|------|-------------|
| `/opt/cloudera/parcels/cma/` | CMA parcel install directory |
| `/opt/cloudera/csd/` | CSD JAR location |
| `/var/lib/cma/data/db/` | CMA Master databases (H2) |
| `/var/lib/cma-agent/data/db/` | CMA Agent database (H2) |
| `/var/lib/cma/data/backups/` | Automatic DB backups from upgrades |
| `/var/lib/cma/conf/` | CMA Master configuration |
| `/var/log/cma/` | CMA Master logs |
| `/var/log/cma-agent/` | CMA Agent logs |

