ApiHdfsReplicationArguments

Properties

Name Type Description Notes
source_service ApiServiceRef [optional]
source_path str The path to replicate. [optional]
destination_path str The destination to replicate to. [optional]
mapreduce_service_name str The mapreduce service to use for the replication job. [optional]
scheduler_pool_name str Name of the scheduler pool to use when submitting the MapReduce job. Currently supports the capacity and fair schedulers. The option is ignored if a different scheduler is configured. [optional]
user_name str The user which will execute the MapReduce job. Required if running with Kerberos enabled. [optional]
source_user str The user which will perform operations on source cluster. Required if running with Kerberos enabled. [optional]
num_maps int The number of mappers to use for the mapreduce replication job. [optional]
dry_run bool Whether to perform a dry run. Defaults to false. [optional]
bandwidth_per_map int The maximum bandwidth (in MB) per mapper in the mapreduce replication job. [optional]
abort_on_error bool Whether to abort on a replication failure. Defaults to false. [optional]
remove_missing_files bool Whether to delete destination files that are missing in source. Defaults to false. [optional]
preserve_replication_count bool Whether to preserve the HDFS replication count. Defaults to false. [optional]
preserve_block_size bool Whether to preserve the HDFS block size. Defaults to false. [optional]
preserve_permissions bool Whether to preserve the HDFS owner, group and permissions. Defaults to false. Starting from V10, it also preserves ACLs. Defaults to null (no preserve). ACLs is preserved if both clusters enable ACL support, and replication ignores any ACL related failures. [optional]
log_path str The HDFS path where the replication log files should be written to. [optional]
skip_checksum_checks bool Whether to skip checksum based file validation during replication. Defaults to false. [optional]
skip_listing_checksum_checks bool Whether to skip checksum based file comparison during replication. Defaults to false. [optional]
skip_trash bool Whether to permanently delete destination files that are missing in source. Defaults to null. [optional]
replication_strategy ReplicationStrategy [optional]
preserve_x_attrs bool Whether to preserve XAttrs, default to false This is introduced in V10. To preserve XAttrs, both CDH versions should be >= 5.2. Replication fails if either cluster does not support XAttrs. [optional]
exclusion_filters list[str] Specify regular expression strings to match full paths of files and directories matching source paths and exclude them from the replication. Optional. Available since V11. [optional]
raise_snapshot_diff_failures bool Flag indicating if failures during snapshotDiff should be ignored or not. When it is set to false then, replication will fallback to full copy listing in case of any error in snapshot diff handling and it will ignore snapshot delete/rename failures at the end of a replication. The flag is by default set to false in distcp tool which means it will ignore snapshot diff failures and mark replication as success for snapshot delete/rename failures. In UI, the flag is set to true by default when source CM Version is greater than 5.14. [optional]
delete_latest_source_snapshot_on_job_failure bool A flag configuring distcp behaviour for the case when the distcp mapreduce job fails. This failure is sometimes due to some issues with the snapshot. By default, in case of mapreduce job failure the latest source side snapshot is deleted. Set this flag to false to make this replication not delete the last successfully replicated old snapshot of the source dataset in case of job failure. The default value of this flag is true. Not setting this flag is equivalent to the value true. [optional]
num_fetch_threads int The number of threads to use for fetching the file statuses from HDFS during source file listing. The value 0 means don't use parallel fetching. The null value means to use distcp defaults. [optional]
destination_cloud_account str The cloud account name which is used in direct hive cloud replication, if specified. [optional]

[Back to Model list] [Back to API list] [Back to README]