ApiFormatCopylistingReq Data Model

Represents the arguments required for formatting copylisting files.

  • inputPath: Absolute path (including protocol and HDFS namenode address/Ozone service ID) to the input file. This file must exist before calling the endpoint. Example: hdfs://namenode:8020/source/seqfile or ofs://ozone-service/vol/bkt/file.
  • outputPath: Absolute path (including protocol and HDFS namenode address/Ozone service ID) to the output file. This file must not exist before calling the endpoint.
  • statusType: The type of the status objects in the input file. Based on the values of com.cloudera.enterprise.distcp.util.StatusTypes.
  • inputFormat: The format of the input file. Valid values: "BINARY".
  • outputFormat: The format of the output file. Valid values: "JSONL".
  • additionalConfigs: A map of additional configuration options for the formatting. These will be merged with the configuration gathered for the command automatically. null values are not allowed.
  • unsetConfigs: A set of configuration options to be unset. null values are not allowed.
  • additionalEnvironmentVariables: A map of additional environment variables to be used in the one-off process. null values are not allowed.

Properties
name data type description
inputPath string Absolute path (including protocol and HDFS namenode address/Ozone service ID) to the input file.

This file must exist before calling the endpoint. Example: hdfs://namenode:8020/source/seqfile or ofs://ozone-service/vol/bkt/file.

outputPath string Absolute path (including protocol and HDFS namenode address/Ozone service ID) to the output file.

This file must not exist before calling the endpoint.

statusType string The type of the status objects in the input file.

Based on the values of com.cloudera.enterprise.distcp.util.StatusTypes.

inputFormat string The format of the input file.

Valid values: "BINARY"

outputFormat string The format of the output file.

Valid values: "JSONL"

additionalConfigs map of string A map of additional configuration options for the formatting. These will be merged with the configuration gathered for the command automatically. null values are not allowed.
unsetConfigs array of string A set of configuration options to be unset. null values are not allowed.
additionalEnvironmentVariables map of string A map of additional environment variables to be used in the one-off process

This is the place to add environment variables related to e.g. kinit that acquires the TGT on the execution-side (usually the destination) or JVM options.

null values are not allowed; this interface must be changed should that be necessary.

delegationTokens string Base64-encoded serialized Hadoop Credentials (delegation tokens) to use for authenticating against the file system service.

When present, the delegation tokens are loaded into the process's UserGroupInformation before accessing the file system, enabling operation against a Kerberized cluster without requiring a Kerberos ticket on the execution host. Impersonation, if required, must be encoded in the delegation tokens themselves (obtained via the obtainDelegationTokens endpoint).

When null, the tool runs without pre-obtained tokens. For non-secured (non-kerberized) services either null or an empty Credentials object is acceptable.

One can obtain delegation tokens via native hadoop service APIs, hadoop cli (e.g. ozone sh token get) or the /obtainDelegationTokens CM API endpoint.

Example

{
  "inputPath" : "...",
  "outputPath" : "...",
  "statusType" : "...",
  "inputFormat" : "...",
  "outputFormat" : "...",
  "additionalConfigs" : {
    "property1" : "...",
    "property2" : "..."
  },
  "unsetConfigs" : [ "...", "..." ],
  "additionalEnvironmentVariables" : {
    "property1" : "...",
    "property2" : "..."
  },
  "delegationTokens" : "..."
}