ApiTransformCopylistingReq Data Model

Represents the arguments required for transforming copylisting files.

  • pathMappings: Mapping rules from source to target paths.
  • statusType: The type of the status objects in the input sequence file. Based on the values of com.cloudera.enterprise.distcp.util.StatusTypes.
  • fromSequenceFile: Absolute path (including protocol and HDFS namenode address/Ozone service ID) to the source sequence file. This file must exist before calling the endpoint Example: hdfs://namenode:8020/source/seqfile or ofs://ozone-service/vol/bkt/file.
  • fromRenameDeleteSequenceFile: Absolute path (including protocol and HDFS namenode address/Ozone service ID) to the source rename/delete sequence file. This file must exist before calling the endpoint
  • toSequenceFile: Absolute path (including protocol and HDFS namenode address/Ozone service ID) to the destination sequence file. This file must not exist before calling the endpoint
  • toRenameDeleteSequenceFile: Absolute path (including protocol and HDFS namenode address/Ozone service ID) to the destination rename/delete sequence file. This file must not exist before calling the endpoint
  • replicationDestinationPath: The destination path of the replication. This should be the same path that was used for generating the copylisting files. It is mandatory to be defined when there is a renameDeleteSequence file in the payload.
  • additionalConfigs: A map of additional configuration options for the transformation. These will be merged with the configuration gathered for the command automatically. null values are not allowed.
  • unsetConfigs: A set of configuration options to be unset. null values are not allowed.
  • additionalEnvironmentVariables: A map of additional environment variables to be used in the one-off process. null values are not allowed.
  • delegationTokens: A serialized and base64 encoded hadoop org.apache.hadoop.security.Credentials object. Can be null for unsecured clusters.

Properties
name data type description
fanningSpec ApiFanningTransformCopylistingSpec Transformation for many-to-many fanning. This is an optional field. When specified the specs for one-to-one mapping are optional.
pathMappings map of string Mapping rules from source to target paths.
  • The mapping must be strict, meaning that the paths should be absolute paths without protocol and HDFS namenode address/Ozone service ID, without dot and dot-dots.

    Example: /source/path to /target/path

  • The mapping must be invertible, i.e. it must be collision-free in both direction.

    Example: A mapping of /source to /target1 and /source/path to /target2 is not collision-free on the source side.

    Example: A mapping of /source1 to /target and /source2 to /target/path is not collision-free on the target side.

null values are not allowed.
statusType string The type of the status objects in the input sequence file.

Based on the values of com.cloudera.enterprise.distcp.util.StatusTypes.

fromSequenceFile string Absolute path (including protocol and HDFS namenode address/Ozone service ID) to the source sequence file.

This file must exist before calling the endpoint Example: hdfs://namenode:8020/source/seqfile or ofs://ozone-service/vol/bkt/file.

fromRenameDeleteSequenceFile string Absolute path (including protocol and HDFS namenode address/Ozone service ID) to the source rename/delete sequence file.

This file must exist before calling the endpoint

toSequenceFile string Absolute path (including protocol and HDFS namenode address/Ozone service ID) to the destination sequence file.

This file must not exist before calling the endpoint

toRenameDeleteSequenceFile string Absolute path (including protocol and HDFS namenode address/Ozone service ID) to the destination rename/delete sequence file.

This file must not exist before calling the endpoint

replicationDestinationPath string The destination path of the replication.

This should be the same path that was used for generating the copylisting files. It is mandatory to be defined when there is a renameDeleteSequence file in the payload.

additionalConfigs map of string A map of additional configuration options for the transformation. These will be merged with the configuration gathered for the command automatically. null values are not allowed.
unsetConfigs array of string A set of configuration options to be unset. null values are not allowed.
additionalEnvironmentVariables map of string A map of additional environment variables to be used in the one-off process

This is the place to add environment variables related to e.g. JVM options.

null values are not allowed; this interface must be changed should that be necessary.

delegationTokens string Base64-encoded serialized Hadoop Credentials (delegation tokens) to use for authenticating against the file system service.

When present, the delegation tokens are loaded into the process's UserGroupInformation before accessing the file system, enabling operation against a Kerberized cluster without requiring a Kerberos ticket on the execution host. Impersonation, if required, must be encoded in the delegation tokens themselves (obtained via the obtainDelegationTokens endpoint).

When null, the tool runs without pre-obtained tokens. For non-secured (non-kerberized) services either null or an empty Credentials object is acceptable.

One can obtain delegation tokens via native hadoop service APIs, hadoop cli (e.g. ozone sh token get) or the /obtainDelegationTokens CM API endpoint.

Example

{
  "fanningSpec" : {
    "fromSequenceFiles" : {
      "property1" : {
        "path" : "...",
        "statusType" : "..."
      },
      "property2" : {
        "path" : "...",
        "statusType" : "..."
      }
    },
    "fromRenameDeleteSequenceFiles" : {
      "property1" : {
        "path" : "...",
        "replicationDestinationPath" : "..."
      },
      "property2" : {
        "path" : "...",
        "replicationDestinationPath" : "..."
      }
    },
    "toSequenceFiles" : {
      "property1" : {
        "path" : "...",
        "statusType" : "..."
      },
      "property2" : {
        "path" : "...",
        "statusType" : "..."
      }
    },
    "toRenameDeleteSequenceFiles" : {
      "property1" : {
        "path" : "...",
        "replicationDestinationPath" : "..."
      },
      "property2" : {
        "path" : "...",
        "replicationDestinationPath" : "..."
      }
    },
    "mappings" : [ {
      "between" : { },
      "pathMappings" : {
        "property1" : "...",
        "property2" : "..."
      }
    }, {
      "between" : { },
      "pathMappings" : {
        "property1" : "...",
        "property2" : "..."
      }
    } ]
  },
  "pathMappings" : {
    "property1" : "...",
    "property2" : "..."
  },
  "statusType" : "...",
  "fromSequenceFile" : "...",
  "fromRenameDeleteSequenceFile" : "...",
  "toSequenceFile" : "...",
  "toRenameDeleteSequenceFile" : "...",
  "replicationDestinationPath" : "...",
  "additionalConfigs" : {
    "property1" : "...",
    "property2" : "..."
  },
  "unsetConfigs" : [ "...", "..." ],
  "additionalEnvironmentVariables" : {
    "property1" : "...",
    "property2" : "..."
  },
  "delegationTokens" : "..."
}