-- Logs begin at Sat 2022-03-12 20:01:16 UTC, end at Sat 2022-03-12 22:10:13 UTC. -- Mar 12 20:38:56.293819 ncdv-master-1 systemd[1]: Started Devstack devstack@c-bak.service. Mar 12 20:38:59.815441 ncdv-master-1 cinder-backup[129000]: INFO cinder.cmd.backup [-] Backup running with 1 processes. Mar 12 20:38:59.816414 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_concurrency.lockutils [-] Acquired lock "singleton_lock" {{(pid=129000) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:294}} Mar 12 20:38:59.816681 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_concurrency.lockutils [-] Releasing lock "singleton_lock" {{(pid=129000) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:312}} Mar 12 20:38:59.985034 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_db.sqlalchemy.engines [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129000) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 12 20:39:00.227211 ncdv-master-1 cinder-backup[129000]: INFO cinder.rpc [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] Automatically selected cinder-backup objects version 1.39 as minimum service version. Mar 12 20:39:00.231872 ncdv-master-1 cinder-backup[129000]: INFO cinder.rpc [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] Automatically selected cinder-backup RPC version 2.3 as minimum service version. Mar 12 20:39:00.236572 ncdv-master-1 cinder-backup[129000]: INFO cinder.rpc [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] Automatically selected cinder-volume objects version 1.39 as minimum service version. Mar 12 20:39:00.240421 ncdv-master-1 cinder-backup[129000]: INFO cinder.rpc [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] Automatically selected cinder-volume RPC version 3.18 as minimum service version. Mar 12 20:39:00.245403 ncdv-master-1 cinder-backup[129000]: INFO cinder.rpc [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] Automatically selected cinder-scheduler objects version 1.39 as minimum service version. Mar 12 20:39:00.259810 ncdv-master-1 cinder-backup[129000]: INFO cinder.rpc [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] Automatically selected cinder-scheduler RPC version 3.12 as minimum service version. Mar 12 20:39:00.325284 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_db.sqlalchemy.engines [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129000) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 12 20:39:00.354316 ncdv-master-1 cinder-backup[129000]: INFO oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] Starting 1 workers Mar 12 20:39:00.361351 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] Started child 129443 {{(pid=129000) _start_child /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:575}} Mar 12 20:39:00.362655 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] Full set of CONF: {{(pid=129000) wait /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:649}} Mar 12 20:39:00.363104 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] ******************************************************************************** {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Mar 12 20:39:00.363481 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] Configuration options gathered from: {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Mar 12 20:39:00.363980 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] command line args: ['--config-file', '/etc/cinder/cinder.conf'] {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Mar 12 20:39:00.364784 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] config files: ['/etc/cinder/cinder.conf'] {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Mar 12 20:39:00.365154 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] ================================================================================ {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Mar 12 20:39:00.366793 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] allow_availability_zone_fallback = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.367457 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] allow_compression_on_image_upload = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.368277 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] allowed_direct_url_schemes = [] {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.368920 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] api_rate_limit = True {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.369514 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] auth_strategy = keystone {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.370067 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_availability_zone = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.370570 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backup_api_class = cinder.backup.api.API {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.371069 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backup_ceph_chunk_size = 134217728 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.371559 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backup_ceph_conf = /etc/ceph/ceph.conf {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.372008 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backup_ceph_image_journals = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.372468 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backup_ceph_pool = backups {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.372920 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backup_ceph_stripe_count = 0 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.373385 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backup_ceph_stripe_unit = 0 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.373902 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backup_ceph_user = cinder-bak {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.374371 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backup_driver = cinder.backup.drivers.ceph.CephBackupDriver {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.374856 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backup_driver_init_check_interval = 60 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.375297 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backup_driver_stats_polling_interval = 60 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.375722 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backup_manager = cinder.backup.manager.BackupManager {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.376687 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backup_max_operations = 15 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.377300 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backup_metadata_version = 2 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.378064 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backup_name_template = backup-%s {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.378938 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backup_native_threads_pool_size = 60 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.379408 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backup_object_number_per_notification = 10 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.379835 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backup_service_inithost_offload = True {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.380269 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backup_timer_interval = 120 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.380550 ncdv-master-1 cinder-backup[129443]: INFO cinder.service [-] Starting cinder-backup node (version 19.1.0) Mar 12 20:39:00.381103 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backup_use_same_host = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.381103 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backup_use_temp_snapshot = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.381588 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backup_workers = 1 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.383165 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] chap_password = **** {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.383669 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] chap_username = {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.384137 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.384640 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cinder_internal_tenant_project_id = e45fccdf84cf4b5abb66a1257e6cb2cd {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.385101 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cinder_internal_tenant_user_id = 4a34e93fbd234e3bb82ff3ad9963ff66 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.385543 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] compression_format = gzip {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.386111 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] config_dir = [] {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.387112 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] config_file = ['/etc/cinder/cinder.conf'] {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.388118 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] config_source = [] {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.388657 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] consistencygroup_api_class = cinder.consistencygroup.api.API {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.389295 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] control_exchange = openstack {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.389807 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] debug = True {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.390279 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] default_availability_zone = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.390691 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] default_group_type = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.391392 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'swiftclient=WARN', 'botocore=WARN'] {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.393364 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] default_volume_type = ceph {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.394021 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] driver_client_cert = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.394649 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] driver_client_cert_key = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.395310 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] driver_data_namespace = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.395916 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] driver_ssl_cert_path = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.397244 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] driver_ssl_cert_verify = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.397728 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] driver_use_ssl = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.398415 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] enable_new_services = True {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.398965 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] enable_unsupported_driver = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.400146 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] enabled_backends = ['ceph'] {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.400536 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] enforce_multipath_for_image_xfer = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.400905 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] filter_function = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.401248 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] glance_api_insecure = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.401623 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] glance_api_servers = ['http://10.222.0.30/image'] {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.402017 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] glance_api_ssl_compression = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.403720 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] glance_ca_certificates_file = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.403720 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] glance_catalog_info = image:glance:publicURL {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.403720 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] glance_certfile = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.403972 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] glance_core_properties = ['checksum', 'container_format', 'disk_format', 'image_name', 'image_id', 'min_disk', 'min_ram', 'name', 'size'] {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.409820 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] glance_keyfile = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.409820 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] glance_num_retries = 3 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.409820 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] glance_request_timeout = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.409820 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] goodness_function = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.409820 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] graceful_shutdown_timeout = 5 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.409820 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] group_api_class = cinder.group.api.API {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.409820 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] host = ncdv-master-1 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.410448 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] iet_conf = /etc/iet/ietd.conf {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.410448 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] image_compress_on_upload = True {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.410448 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] image_conversion_address_space_limit = 1 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.410448 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] image_conversion_cpu_limit = 60 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.410448 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] image_conversion_dir = /opt/stack/data/cinder/conversion {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.410448 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] image_upload_use_cinder_backend = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.410820 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] image_upload_use_internal_tenant = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.410820 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] image_volume_cache_enabled = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.411212 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] image_volume_cache_max_count = 0 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.411425 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] image_volume_cache_max_size_gb = 0 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.411811 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] instance_format = [instance: %(uuid)s] {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.412162 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] instance_uuid_format = [instance: %(uuid)s] {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.412540 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] iscsi_iotype = fileio {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.412919 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] iscsi_secondary_ip_addresses = [] {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.413281 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] iscsi_target_flags = {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.413657 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] iscsi_write_cache = on {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.414033 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] iser_helper = tgtadm {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.414765 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] iser_ip_address = 10.222.0.30 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.415221 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] iser_port = 3260 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.415603 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.415960 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] log_config_append = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.416343 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.416695 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] log_dir = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.417496 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] log_file = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.417808 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] log_options = True {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.418333 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] log_rotate_interval = 1 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.419010 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] log_rotate_interval_type = days {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.419448 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] log_rotation_type = none {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.419748 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.420199 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.420574 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.421826 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.421826 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] logging_user_identity_format = %(user)s %(tenant)s %(domain)s %(user_domain)s %(project_domain)s {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.421826 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] max_age = 0 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.422141 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] max_logfile_count = 30 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.422402 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] max_logfile_size_mb = 200 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.422785 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] max_over_subscription_ratio = 20.0 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.423145 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] message_reap_interval = 86400 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.423525 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] message_ttl = 2592000 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.423872 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] monkey_patch = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.424225 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] monkey_patch_modules = [] {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.424523 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] my_ip = 10.222.0.30 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.424866 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] no_snapshot_gb_quota = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.425228 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] num_iser_scan_tries = 3 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.425610 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] num_shell_tries = 3 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.426275 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] num_volume_device_scan_tries = 3 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.428321 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] nvmet_ns_id = 10 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.429138 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] nvmet_port_id = 1 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.429558 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] osapi_max_limit = 1000 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.429964 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] osapi_volume_ext_list = [] {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.431148 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] osapi_volume_extension = ['cinder.api.contrib.standard_extensions'] {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.431583 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] osapi_volume_listen = 0.0.0.0 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.431956 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] osapi_volume_listen_port = 8776 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.432310 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] osapi_volume_use_ssl = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.432670 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] osapi_volume_workers = 2 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.433024 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] per_volume_size_limit = -1 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.433363 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] periodic_fuzzy_delay = 60 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.433729 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] periodic_interval = 60 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.434152 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] project_id_regex = [0-9a-f\-]+ {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.434542 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] public_endpoint = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.434889 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] publish_errors = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.435289 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] quota_backup_gigabytes = 1000 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.435655 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] quota_backups = 10 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.436022 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] quota_consistencygroups = 10 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.436396 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] quota_driver = cinder.quota.DbQuotaDriver {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.436765 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] quota_gigabytes = 1000 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.438524 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] quota_groups = 10 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.439336 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] quota_snapshots = 10 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.439726 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] quota_volumes = 10 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.440084 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] rate_limit_burst = 0 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.440639 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] rate_limit_except_level = CRITICAL {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.440994 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] rate_limit_interval = 0 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.441350 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] replication_device = **** {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.441715 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] report_discard_supported = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.442105 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] report_interval = 10 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.442581 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] reservation_clean_interval = 86400 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.442877 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] reservation_expire = 86400 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.443231 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] reserved_percentage = 0 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.443609 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] resource_query_filters_file = /etc/cinder/resource_filters.json {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.443969 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] restore_discard_excess_bytes = True {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.444325 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] rootwrap_config = /etc/cinder/rootwrap.conf {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.444733 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] rpc_response_timeout = 120 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.445095 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] run_external_periodic_tasks = True {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.445523 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] scheduler_manager = cinder.scheduler.manager.SchedulerManager {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.446439 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] scst_target_driver = iscsi {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.446794 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] scst_target_iqn_name = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.447147 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] service_down_time = 60 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.447478 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.lockutils [None req-6a961ee4-3438-47ea-a76d-c7a9db533359 None None] Lock "cleanup_incomplete_backups_129000" acquired by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: waited 0.006s {{(pid=129443) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:386}} Mar 12 20:39:00.447598 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] snapshot_name_template = snapshot-%s {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.447922 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] split_loggers = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.448424 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-6a961ee4-3438-47ea-a76d-c7a9db533359 None None] Cleaning up incomplete backup operations. Mar 12 20:39:00.448690 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] state_path = /opt/stack/data/cinder {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.449492 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] storage_availability_zone = nova {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.449910 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] storage_protocol = iscsi {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.450274 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] syslog_log_facility = LOG_USER {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.450839 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] target_helper = lioadm {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.451295 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] target_ip_address = 10.222.0.30 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.451695 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] target_port = 3260 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.452176 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] target_prefix = iqn.2010-10.org.openstack: {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.452657 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] target_protocol = iscsi {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.453027 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] trace_flags = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.453384 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] transfer_api_class = cinder.transfer.api.API {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.453740 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] transport_url = **** {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.454126 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] until_refresh = 0 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.454483 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] use_chap_auth = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.454830 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] use_default_quota_class = True {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.455169 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] use_eventlog = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.455518 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] use_journal = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.455859 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] use_json = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.456197 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] use_multipath_for_image_xfer = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.456536 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] use_stderr = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.456872 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] use_syslog = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.457261 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] verify_glance_signatures = enabled {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.457639 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] volume_api_class = cinder.volume.api.API {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.458005 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] volume_backend_name = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.458870 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] volume_clear = zero {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.459897 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] volume_clear_ionice = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.460251 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] volume_clear_size = 0 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.460616 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.461132 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] volume_copy_bps_limit = 0 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.461489 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] volume_dd_blocksize = 1M {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.462444 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] volume_manager = cinder.volume.manager.VolumeManager {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.462809 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] volume_name_template = volume-%s {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.463230 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] volume_usage_audit_period = month {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.463725 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] volumes_dir = /opt/stack/data/cinder/volumes {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.464076 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] watch_log_file = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 12 20:39:00.464571 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.464972 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.465366 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_policy.enforce_new_defaults = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.465715 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_policy.enforce_scope = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.466134 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_policy.policy_default_rule = default {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.466745 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_policy.policy_dirs = ['policy.d'] {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.467626 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_policy.policy_file = policy.yaml {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.469025 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.470232 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.470856 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.471387 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.471750 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.472162 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_concurrency.disable_process_locking = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.472544 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_concurrency.lock_path = /opt/stack/data/cinder {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.472973 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.admin_password = **** {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.473531 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.admin_tenant_name = admin {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.474551 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.admin_token = **** {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.476894 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.admin_user = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.476894 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.auth_admin_prefix = {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.476894 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.477276 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.auth_port = 35357 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.477477 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.auth_protocol = https {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.479828 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.auth_section = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.479828 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.auth_type = password {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.480004 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.auth_uri = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.485792 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.auth_version = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.485792 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.cache = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.485792 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.cafile = /opt/stack/data/ca-bundle.pem {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.485792 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.certfile = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.485792 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.delay_auth_decision = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.485792 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.enforce_token_bind = permissive {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.486249 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.http_connect_timeout = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.486249 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.http_request_max_retries = 3 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.486249 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.identity_uri = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.486249 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.include_service_catalog = True {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.486249 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.insecure = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.486249 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.interface = public {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.486595 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.keyfile = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.486934 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.487305 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.487830 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.488225 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.488576 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.490662 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.memcache_secret_key = **** {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.490662 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.memcache_security_strategy = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.490662 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.490999 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.memcached_servers = ['localhost:11211'] {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.491283 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.region_name = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.491808 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.service_token_roles = ['service'] {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.497790 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.service_token_roles_required = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.497790 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.service_type = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.497790 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.token_cache_time = 300 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.497790 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] keystone_authtoken.www_authenticate_uri = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.497790 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.backend = dogpile.cache.null {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.497790 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.backend_argument = **** {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.498221 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.config_prefix = cache.oslo {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.498221 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.dead_timeout = 60.0 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.498221 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.debug_cache_backend = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.498221 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.enable_retry_client = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.498221 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.enable_socket_keepalive = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.498221 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.enabled = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.498221 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.expiration_time = 600 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.498645 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.hashclient_retry_attempts = 2 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.498645 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.hashclient_retry_delay = 1.0 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.498645 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.memcache_dead_retry = 300 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.498645 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.memcache_pool_connection_get_timeout = 10 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.498880 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.memcache_pool_flush_on_reconnect = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.499524 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.memcache_pool_maxsize = 10 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.500554 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.memcache_pool_unused_timeout = 60 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.501050 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.memcache_servers = ['localhost:11211'] {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.501472 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.memcache_socket_timeout = 1.0 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.501999 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.proxies = [] {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.502396 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.retry_attempts = 2 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.502947 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.retry_delay = 0.0 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.503787 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.socket_keepalive_count = 1 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.504169 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.socket_keepalive_idle = 1 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.504643 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.socket_keepalive_interval = 1 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.504991 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.tls_allowed_ciphers = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.505677 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.tls_cafile = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.506128 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.tls_certfile = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.506693 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.tls_enabled = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.507049 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cache.tls_keyfile = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.507787 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] coordination.backend_url = etcd3+http://10.222.0.30:2379 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.507959 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] database.backend = sqlalchemy {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.509375 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] database.connection = **** {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.513796 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] database.connection_debug = 0 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.513796 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] database.connection_parameters = {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.513796 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] database.connection_recycle_time = 3600 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.513796 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] database.connection_trace = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.513796 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] database.db_inc_retry_interval = True {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.513796 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] database.db_max_retries = 20 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.513796 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] database.db_max_retry_interval = 10 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.514845 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] database.db_retry_interval = 1 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.514845 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] database.max_overflow = 50 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.514845 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] database.max_pool_size = 5 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.514845 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] database.max_retries = 10 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.514845 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] database.mysql_enable_ndb = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.515523 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] database.mysql_sql_mode = TRADITIONAL {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.515523 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] database.pool_timeout = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.515860 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] database.retry_interval = 10 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.516242 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] database.slave_connection = **** {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.516604 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] database.sqlite_synchronous = True {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.516973 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] database.use_db_reconnect = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.517364 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.517721 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.518116 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_metrics.metrics_process_name = {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.518476 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.518834 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.519382 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] profiler.connection_string = messaging:// {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.520058 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] profiler.enabled = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.520846 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] profiler.es_doc_type = notification {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.521202 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] profiler.es_scroll_size = 10000 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.525812 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] profiler.es_scroll_time = 2m {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.525812 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] profiler.filter_error_trace = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.525812 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] profiler.hmac_keys = SECRET_KEY {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.525812 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] profiler.sentinel_service_name = mymaster {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.525812 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] profiler.socket_timeout = 0.1 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.525812 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] profiler.trace_sqlalchemy = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.525812 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] privsep_osbrick.capabilities = [21] {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.526980 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] privsep_osbrick.group = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.526980 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] privsep_osbrick.helper_command = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.526980 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] privsep_osbrick.logger_name = oslo_privsep.daemon {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.526980 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] privsep_osbrick.thread_pool_size = 4 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.526980 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] privsep_osbrick.user = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.526980 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cinder_sys_admin.capabilities = [0, 1, 2, 3, 12, 21] {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.526980 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cinder_sys_admin.group = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.527890 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cinder_sys_admin.helper_command = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.527890 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cinder_sys_admin.logger_name = oslo_privsep.daemon {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.527890 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cinder_sys_admin.thread_pool_size = 4 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.528324 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] cinder_sys_admin.user = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.528464 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] service_user.auth_section = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.528713 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] service_user.auth_type = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.529061 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] service_user.cafile = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.529549 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] service_user.certfile = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.530544 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] service_user.collect_timing = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.531321 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] service_user.insecure = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.531675 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] service_user.keyfile = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.532036 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] service_user.send_service_user_token = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.532555 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] service_user.split_loggers = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.532901 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] service_user.timeout = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.533336 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.backend_availability_zone = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.533714 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.chap_password = **** {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.534115 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.chap_username = {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.534479 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.537796 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.deferred_deletion_delay = 0 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.537796 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.deferred_deletion_purge_interval = 60 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.537796 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.driver_client_cert = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.537796 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.driver_client_cert_key = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.537796 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.driver_data_namespace = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.537796 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.driver_ssl_cert_path = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.538771 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.driver_ssl_cert_verify = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.538771 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.driver_use_ssl = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.538771 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.enable_deferred_deletion = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.538771 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.enable_unsupported_driver = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.538771 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.enforce_multipath_for_image_xfer = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.538771 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.filter_function = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.539564 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.goodness_function = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.539564 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.iet_conf = /etc/iet/ietd.conf {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.541284 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.image_upload_use_cinder_backend = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.541541 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.image_upload_use_internal_tenant = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.541932 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.image_volume_cache_enabled = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.542313 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.image_volume_cache_max_count = 0 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.542853 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.image_volume_cache_max_size_gb = 0 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.543229 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.iscsi_iotype = fileio {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.543889 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.iscsi_secondary_ip_addresses = [] {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.544260 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.iscsi_target_flags = {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.544625 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.iscsi_write_cache = on {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.544994 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.iser_helper = tgtadm {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.545534 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.iser_ip_address = 10.222.0.30 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.545932 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.iser_port = 3260 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.546300 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.546851 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.max_over_subscription_ratio = 20.0 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.547267 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.num_iser_scan_tries = 3 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.547622 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.num_shell_tries = 3 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.548020 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.num_volume_device_scan_tries = 3 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.548373 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.nvmet_ns_id = 10 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.548726 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.nvmet_port_id = 1 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.549093 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.rados_connect_timeout = -1 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.549462 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.rados_connection_interval = 5 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.549990 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.rados_connection_retries = 3 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.550941 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.rbd_ceph_conf = {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.551475 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.rbd_cluster_name = ceph {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.551829 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.rbd_exclusive_cinder_pool = True {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.552186 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.rbd_flatten_volume_from_snapshot = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.552893 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.rbd_max_clone_depth = 5 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.553258 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.rbd_pool = rbd {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.553605 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.rbd_secret_uuid = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.553982 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.rbd_store_chunk_size = 4 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.554333 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.rbd_user = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.554680 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.replication_connect_timeout = 5 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.555969 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.replication_device = **** {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.555969 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.report_discard_supported = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.556300 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.report_dynamic_total_capacity = True {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.561474 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.reserved_percentage = 0 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.561474 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.scst_target_driver = iscsi {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.561474 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.scst_target_iqn_name = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.561474 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.storage_protocol = iscsi {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.561474 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.target_helper = tgtadm {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.561474 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.target_ip_address = 10.222.0.30 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.562472 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.target_port = 3260 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.562472 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.target_prefix = iqn.2010-10.org.openstack: {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.562472 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.target_protocol = iscsi {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.562472 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.trace_flags = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.562472 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.unique_fqdn_network = True {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.562472 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.use_chap_auth = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.563284 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.use_multipath_for_image_xfer = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.563284 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.volume_backend_name = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.563284 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.volume_clear = zero {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.563284 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.volume_clear_ionice = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.563820 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.volume_clear_size = 0 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.563820 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.564086 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.volume_copy_bps_limit = 0 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.564365 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.volume_dd_blocksize = 1M {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.564818 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] backend_defaults.volumes_dir = /opt/stack/data/cinder/volumes {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.565195 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_reports.file_event_handler = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.565573 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_reports.file_event_handler_interval = 1 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.565978 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_reports.log_dir = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.566406 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.566783 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.567162 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.567531 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.567914 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.568259 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.568644 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_rabbit.heartbeat_in_pthread = True {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.568999 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_rabbit.heartbeat_rate = 2 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.569348 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.569694 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.570199 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.571158 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.571690 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.572114 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.572414 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.572780 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.573260 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.573616 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.574005 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.574357 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.574723 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.575076 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.575588 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_rabbit.ssl = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.576085 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_rabbit.ssl_ca_file = {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.576456 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_rabbit.ssl_cert_file = {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.576819 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_rabbit.ssl_key_file = {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.577181 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_rabbit.ssl_version = {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.577603 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_notifications.driver = [] {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.582335 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_notifications.retry = -1 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.583005 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.583579 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] oslo_messaging_notifications.transport_url = **** {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.584243 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] key_manager.backend = cinder.keymgr.conf_key_mgr.ConfKeyManager {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.585173 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] barbican.auth_endpoint = http://localhost/identity/v3 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.585734 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] barbican.barbican_api_version = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.586879 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] barbican.barbican_endpoint = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.587489 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] barbican.barbican_endpoint_type = public {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.588040 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] barbican.barbican_region_name = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.588820 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] barbican.cafile = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.589366 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] barbican.certfile = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.589965 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] barbican.collect_timing = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.590645 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] barbican.insecure = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.591577 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] barbican.keyfile = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.592269 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] barbican.number_of_retries = 60 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.593017 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] barbican.retry_delay = 1 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.593566 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] barbican.send_service_user_token = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.594270 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] barbican.split_loggers = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.594821 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] barbican.timeout = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.595366 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] barbican.verify_ssl = True {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.595936 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] barbican.verify_ssl_path = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.596487 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] barbican_service_user.auth_section = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.597031 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] barbican_service_user.auth_type = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.597570 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] barbican_service_user.cafile = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.598336 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] barbican_service_user.certfile = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.598941 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] barbican_service_user.collect_timing = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.599493 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] barbican_service_user.insecure = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.600027 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] barbican_service_user.keyfile = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.600668 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] barbican_service_user.split_loggers = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.601564 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] barbican_service_user.timeout = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.602975 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] vault.approle_role_id = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.603545 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] vault.approle_secret_id = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.604096 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] vault.cafile = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.604637 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] vault.certfile = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.605197 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] vault.collect_timing = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.605741 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] vault.insecure = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.606330 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] vault.keyfile = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.606886 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] vault.kv_mountpoint = secret {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.607427 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] vault.kv_version = 2 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.607993 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] vault.namespace = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.608535 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] vault.root_token_id = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.609077 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] vault.split_loggers = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.609625 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] vault.ssl_ca_crt_file = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.610208 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] vault.timeout = None {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.610885 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] vault.use_ssl = False {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.612512 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] vault.vault_url = http://127.0.0.1:8200 {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 12 20:39:00.613204 ncdv-master-1 cinder-backup[129000]: DEBUG oslo_service.service [None req-51c7da63-1c9f-4fab-b272-6d4ea4aa8cf3 None None] ******************************************************************************** {{(pid=129000) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Mar 12 20:39:00.646716 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_db.sqlalchemy.engines [None req-6a961ee4-3438-47ea-a76d-c7a9db533359 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129443) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 12 20:39:00.671662 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.lockutils [None req-6a961ee4-3438-47ea-a76d-c7a9db533359 None None] Lock "cleanup_incomplete_backups_129000" "released" by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: held 0.227s {{(pid=129443) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:400}} Mar 12 20:39:00.677020 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_db.sqlalchemy.engines [None req-6a961ee4-3438-47ea-a76d-c7a9db533359 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129443) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 12 20:39:00.683091 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-6a961ee4-3438-47ea-a76d-c7a9db533359 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 20:39:00.707425 ncdv-master-1 cinder-backup[129443]: INFO cinder.keymgr.migration [None req-56700c4e-a4ba-4a25-9924-7eae51881fb0 None None] Not migrating encryption keys because the ConfKeyManager is still in use. Mar 12 20:39:00.716075 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.service [None req-6a961ee4-3438-47ea-a76d-c7a9db533359 None None] Creating RPC server for service cinder-backup {{(pid=129443) start /opt/stack/cinder/cinder/service.py:231}} Mar 12 20:39:00.724791 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.service [None req-6a961ee4-3438-47ea-a76d-c7a9db533359 None None] Pinning object versions for RPC server serializer to 1.39 {{(pid=129443) start /opt/stack/cinder/cinder/service.py:237}} Mar 12 20:40:34.762723 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 20:40:34.763437 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 20:41:34.767416 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 20:41:34.768102 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 20:42:34.763271 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 20:42:34.763949 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 20:43:34.758730 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 20:43:34.759391 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 20:44:34.763149 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 20:44:34.763830 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 20:45:34.767500 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 20:45:34.768093 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 20:46:34.772096 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 20:46:34.772686 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 20:47:34.777042 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 20:47:34.777685 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 20:48:34.786098 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 20:48:34.788668 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 20:49:34.790322 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 20:49:34.791854 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 20:50:34.788972 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 20:50:34.790222 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 20:51:34.785593 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 20:51:34.786244 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 20:52:34.788173 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 20:52:34.788924 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 20:53:34.794374 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 20:53:34.795083 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 20:54:34.799780 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 20:54:34.800431 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 20:55:34.805294 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 20:55:34.806004 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 20:56:34.809006 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 20:56:34.809571 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 20:57:34.804438 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 20:57:34.805159 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 20:58:34.800309 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 20:58:34.801093 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 20:59:34.813293 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 20:59:34.813293 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:00:34.813804 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:00:34.814677 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:01:34.810458 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:01:34.811646 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:02:34.810441 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:02:34.811098 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:03:34.817303 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:03:34.818312 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:04:34.813452 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:04:34.814198 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:05:34.818768 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:05:34.819502 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:06:34.817216 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:06:34.817971 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:07:34.819367 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:07:34.821177 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:07:44.834408 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] Create backup started, backup: 9c644465-cf94-4bc3-9e83-5aa42b218df0 volume: 0d219623-5b71-4651-9d9b-3c7aadd4b36f. Mar 12 21:07:44.860580 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-03-12T21:07:44Z,data_timestamp=2022-03-12T21:07:44Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsTest-Backup-2068028501',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=9c644465-cf94-4bc3-9e83-5aa42b218df0,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='e2ea29aba17e41fcac0b47522cf48b4f',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='c5601cc6a01a4c35bceb33df0cad0acc',volume_id=0d219623-5b71-4651-9d9b-3c7aadd4b36f) Mar 12 21:07:45.003593 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.utils [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:07:45.005566 ncdv-master-1 cinder-backup[129443]: INFO oslo.privsep.daemon [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] Running privsep helper: ['sudo', 'cinder-rootwrap', '/etc/cinder/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/cinder/cinder.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmpqk_ryk18/privsep.sock'] Mar 12 21:07:45.209819 ncdv-master-1 sudo[157633]: ubuntu : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf privsep-helper --config-file /etc/cinder/cinder.conf --privsep_context os_brick.privileged.default --privsep_sock_path /tmp/tmpqk_ryk18/privsep.sock Mar 12 21:07:45.228409 ncdv-master-1 sudo[157633]: pam_unix(sudo:session): session opened for user root by (uid=0) Mar 12 21:07:51.025618 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_db.sqlalchemy.engines [None req-7708f48f-a412-494a-8980-681c55bd36ad None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129443) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 12 21:07:54.631163 ncdv-master-1 sudo[157633]: pam_unix(sudo:session): session closed for user root Mar 12 21:07:54.635260 ncdv-master-1 cinder-backup[129443]: INFO oslo.privsep.daemon [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] Spawned new privsep daemon via rootwrap Mar 12 21:07:54.636109 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] Accepted privsep connection to /tmp/tmpqk_ryk18/privsep.sock {{(pid=129443) __init__ /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:362}} Mar 12 21:07:54.638124 ncdv-master-1 cinder-backup[129443]: INFO oslo.privsep.daemon [-] privsep daemon starting Mar 12 21:07:54.638564 ncdv-master-1 cinder-backup[129443]: INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 Mar 12 21:07:54.638906 ncdv-master-1 cinder-backup[129443]: INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none Mar 12 21:07:54.639249 ncdv-master-1 cinder-backup[129443]: INFO oslo.privsep.daemon [-] privsep daemon running as pid 157799 Mar 12 21:07:54.642878 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[07045be3-7333-4d85-9424-de4aae91a3a8]: (2,) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:07:55.564729 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:07:55.613498 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.049s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:07:55.614107 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[cba614dc-8728-4649-94f4-6735f4cb493d]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:77472599036\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:07:55.615956 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:07:55.642927 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 12 21:07:55.667300 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[72ca8b23-b71e-466f-bba9-7736e5249aea]: [Errno 2] No such file or directory: 'systool' {{(pid=157799) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 12 21:07:55.667300 ncdv-master-1 cinder-backup[129443]: Traceback (most recent call last): Mar 12 21:07:55.667300 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 12 21:07:55.667300 ncdv-master-1 cinder-backup[129443]: ret = func(*f_args, **f_kwargs) Mar 12 21:07:55.667300 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 12 21:07:55.667300 ncdv-master-1 cinder-backup[129443]: return func(*args, **kwargs) Mar 12 21:07:55.667300 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 12 21:07:55.667300 ncdv-master-1 cinder-backup[129443]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 12 21:07:55.667300 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 12 21:07:55.667300 ncdv-master-1 cinder-backup[129443]: return putils.execute(on_execute=on_execute, Mar 12 21:07:55.667300 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 12 21:07:55.667300 ncdv-master-1 cinder-backup[129443]: obj = subprocess.Popen(cmd, Mar 12 21:07:55.667300 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 12 21:07:55.667300 ncdv-master-1 cinder-backup[129443]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 12 21:07:55.667300 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 12 21:07:55.667300 ncdv-master-1 cinder-backup[129443]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 12 21:07:55.667300 ncdv-master-1 cinder-backup[129443]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 12 21:07:55.669174 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[72ca8b23-b71e-466f-bba9-7736e5249aea]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:07:55.671032 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:07:55.709070 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 12 21:07:55.709711 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[e0d05bec-47f8-4645-91e1-b4187a3eb506]: [Errno 2] No such file or directory: 'systool' {{(pid=157799) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 12 21:07:55.709711 ncdv-master-1 cinder-backup[129443]: Traceback (most recent call last): Mar 12 21:07:55.709711 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 12 21:07:55.709711 ncdv-master-1 cinder-backup[129443]: ret = func(*f_args, **f_kwargs) Mar 12 21:07:55.709711 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 12 21:07:55.709711 ncdv-master-1 cinder-backup[129443]: return func(*args, **kwargs) Mar 12 21:07:55.709711 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 12 21:07:55.709711 ncdv-master-1 cinder-backup[129443]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 12 21:07:55.709711 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 12 21:07:55.709711 ncdv-master-1 cinder-backup[129443]: return putils.execute(on_execute=on_execute, Mar 12 21:07:55.709711 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 12 21:07:55.709711 ncdv-master-1 cinder-backup[129443]: obj = subprocess.Popen(cmd, Mar 12 21:07:55.709711 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 12 21:07:55.709711 ncdv-master-1 cinder-backup[129443]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 12 21:07:55.709711 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 12 21:07:55.709711 ncdv-master-1 cinder-backup[129443]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 12 21:07:55.709711 ncdv-master-1 cinder-backup[129443]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 12 21:07:55.712603 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[e0d05bec-47f8-4645-91e1-b4187a3eb506]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:07:55.717032 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:07:55.791369 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.076s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:07:55.791749 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[7349feb7-cdd1-41aa-aa37-d9c5b1cfa752]: (4, ('/dev/sda1\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:07:55.793293 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:07:55.867318 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.073s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:07:55.867613 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[cd79818e-c63d-4604-8d25-064a87d81ff4]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:07:55.869057 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:07:55.899250 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.030s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:07:55.899801 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[cf295cac-e85b-4426-aa4b-d2a7d0cfb087]: (4, ('55f4b1f4-b5b7-af4f-9adb-821e08822a61\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:07:55.901008 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] Running cmd (subprocess): nvme version {{(pid=129443) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:07:55.958706 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] CMD "nvme version" returned: 0 in 0.058s {{(pid=129443) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:07:56.000716 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129443) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 12 21:07:56.001435 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129443) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 12 21:07:56.002027 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:d9949f82-06c8-4f47-8c57-a55eecab76d2 dsc: {{(pid=129443) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 12 21:07:56.004537 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.utils [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] <== get_connector_properties: return (10999ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:77472599036', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '55f4b1f4-b5b7-af4f-9adb-821e08822a61', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:d9949f82-06c8-4f47-8c57-a55eecab76d2', 'found_dsc': ''} {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:07:57.440780 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connector [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] Factory for rbd on None {{(pid=129443) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 12 21:07:57.444030 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-0d219623-5b71-4651-9d9b-3c7aadd4b36f', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '0d219623-5b71-4651-9d9b-3c7aadd4b36f', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:07:57.447066 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.linuxrbd [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] opening connection to ceph cluster (timeout=-1). {{(pid=129443) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 12 21:07:57.547894 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] <== connect_volume: return (101ms) {'path': } {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:07:57.587565 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] Starting backup of volume='0d219623-5b71-4651-9d9b-3c7aadd4b36f'. {{(pid=129443) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Mar 12 21:07:57.588833 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] Volume file is RBD: attempting optimized backup {{(pid=129443) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Mar 12 21:07:57.660217 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] Creating base image 'volume-0d219623-5b71-4651-9d9b-3c7aadd4b36f.backup.9c644465-cf94-4bc3-9e83-5aa42b218df0' {{(pid=129443) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Mar 12 21:07:57.661080 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129443) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Mar 12 21:07:59.594859 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] Using --from-snap 'None' for incremental backup of volume 0d219623-5b71-4651-9d9b-3c7aadd4b36f. {{(pid=129443) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Mar 12 21:07:59.596069 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] Creating backup snapshot='backup.9c644465-cf94-4bc3-9e83-5aa42b218df0.snap.1647119279.5956738' {{(pid=129443) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Mar 12 21:08:00.555845 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] Performing differential transfer from 'volume-0d219623-5b71-4651-9d9b-3c7aadd4b36f' to 'volume-0d219623-5b71-4651-9d9b-3c7aadd4b36f.backup.9c644465-cf94-4bc3-9e83-5aa42b218df0' {{(pid=129443) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Mar 12 21:08:00.559290 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_hcdwryi9 --pool volumes volumes/volume-0d219623-5b71-4651-9d9b-3c7aadd4b36f@backup.9c644465-cf94-4bc3-9e83-5aa42b218df0.snap.1647119279.5956738 -' into... {{(pid=129443) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Mar 12 21:08:00.559481 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-0d219623-5b71-4651-9d9b-3c7aadd4b36f.backup.9c644465-cf94-4bc3-9e83-5aa42b218df0' {{(pid=129443) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Mar 12 21:08:01.793297 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] Differential backup transfer completed in 1.2351s {{(pid=129443) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Mar 12 21:08:01.795027 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] Getting metadata type 'volume-base-metadata' {{(pid=129443) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Mar 12 21:08:01.821880 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129443) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Mar 12 21:08:01.823008 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] Getting metadata type 'volume-metadata' {{(pid=129443) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Mar 12 21:08:01.873636 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] No metadata type 'volume-metadata' available {{(pid=129443) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Mar 12 21:08:01.927935 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] Getting metadata type 'volume-glance-metadata' {{(pid=129443) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Mar 12 21:08:01.927935 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] No metadata type 'volume-glance-metadata' available {{(pid=129443) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Mar 12 21:08:01.927935 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] Backing up metadata for volume 0d219623-5b71-4651-9d9b-3c7aadd4b36f. {{(pid=129443) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Mar 12 21:08:01.996122 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] Backup '9c644465-cf94-4bc3-9e83-5aa42b218df0' of volume 0d219623-5b71-4651-9d9b-3c7aadd4b36f finished. {{(pid=129443) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Mar 12 21:08:01.998216 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-0d219623-5b71-4651-9d9b-3c7aadd4b36f', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '0d219623-5b71-4651-9d9b-3c7aadd4b36f', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:08:02.023239 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] <== disconnect_volume: return (23ms) None {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:08:02.151164 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-76a7779f-a2d1-41a8-8cd3-fba7036f8380 tempest-VolumesBackupsTest-1109530946 None] Create backup finished. backup: 9c644465-cf94-4bc3-9e83-5aa42b218df0. Mar 12 21:08:02.796395 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-9408ef28-ce7a-417e-a520-6d34f16a70c7 tempest-VolumesBackupsTest-1109530946 None] Delete backup started, backup: 9c644465-cf94-4bc3-9e83-5aa42b218df0. Mar 12 21:08:02.798481 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-9408ef28-ce7a-417e-a520-6d34f16a70c7 tempest-VolumesBackupsTest-1109530946 None] Delete started for backup=9c644465-cf94-4bc3-9e83-5aa42b218df0 {{(pid=129443) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Mar 12 21:08:02.798921 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-9408ef28-ce7a-417e-a520-6d34f16a70c7 tempest-VolumesBackupsTest-1109530946 None] Trying diff format basename='volume-0d219623-5b71-4651-9d9b-3c7aadd4b36f.backup.9c644465-cf94-4bc3-9e83-5aa42b218df0' for backup base image of volume 0d219623-5b71-4651-9d9b-3c7aadd4b36f. {{(pid=129443) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Mar 12 21:08:02.904490 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-9408ef28-ce7a-417e-a520-6d34f16a70c7 tempest-VolumesBackupsTest-1109530946 None] Looking for snapshot of backup base 'volume-0d219623-5b71-4651-9d9b-3c7aadd4b36f.backup.9c644465-cf94-4bc3-9e83-5aa42b218df0' {{(pid=129443) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 12 21:08:02.905258 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-9408ef28-ce7a-417e-a520-6d34f16a70c7 tempest-VolumesBackupsTest-1109530946 None] Found snapshot 'backup.9c644465-cf94-4bc3-9e83-5aa42b218df0.snap.1647119279.5956738' {{(pid=129443) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Mar 12 21:08:02.905800 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-9408ef28-ce7a-417e-a520-6d34f16a70c7 tempest-VolumesBackupsTest-1109530946 None] Deleting backup snapshot='backup.9c644465-cf94-4bc3-9e83-5aa42b218df0.snap.1647119279.5956738' {{(pid=129443) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Mar 12 21:08:03.696227 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.drivers.ceph [None req-9408ef28-ce7a-417e-a520-6d34f16a70c7 tempest-VolumesBackupsTest-1109530946 None] Deleting backup base image='volume-0d219623-5b71-4651-9d9b-3c7aadd4b36f.backup.9c644465-cf94-4bc3-9e83-5aa42b218df0' of volume 0d219623-5b71-4651-9d9b-3c7aadd4b36f. Mar 12 21:08:04.123087 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-9408ef28-ce7a-417e-a520-6d34f16a70c7 tempest-VolumesBackupsTest-1109530946 None] Base backup image='volume-0d219623-5b71-4651-9d9b-3c7aadd4b36f.backup.9c644465-cf94-4bc3-9e83-5aa42b218df0' of volume 0d219623-5b71-4651-9d9b-3c7aadd4b36f deleted. {{(pid=129443) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Mar 12 21:08:04.173539 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-9408ef28-ce7a-417e-a520-6d34f16a70c7 tempest-VolumesBackupsTest-1109530946 None] Delete of backup '9c644465-cf94-4bc3-9e83-5aa42b218df0' for volume '0d219623-5b71-4651-9d9b-3c7aadd4b36f' finished. {{(pid=129443) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Mar 12 21:08:04.182363 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_db.sqlalchemy.engines [None req-9408ef28-ce7a-417e-a520-6d34f16a70c7 tempest-VolumesBackupsTest-1109530946 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129443) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 12 21:08:04.219572 ncdv-master-1 cinder-backup[129443]: WARNING cinder.quota [None req-9408ef28-ce7a-417e-a520-6d34f16a70c7 tempest-VolumesBackupsTest-1109530946 None] Deprecated: Default quota for resource: backups is set by the default quota flag: quota_backups, it is now deprecated. Please use the default quota class for default quota. Mar 12 21:08:04.220920 ncdv-master-1 cinder-backup[129443]: WARNING cinder.quota [None req-9408ef28-ce7a-417e-a520-6d34f16a70c7 tempest-VolumesBackupsTest-1109530946 None] Deprecated: Default quota for resource: backup_gigabytes is set by the default quota flag: quota_backup_gigabytes, it is now deprecated. Please use the default quota class for default quota. Mar 12 21:08:04.245298 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.quota [None req-9408ef28-ce7a-417e-a520-6d34f16a70c7 tempest-VolumesBackupsTest-1109530946 None] Created reservations ['3d23de2d-bbdb-4790-9bcd-5b4ca0657e42', 'a4e23b9c-a4f8-4f28-88ca-c88903a82f8b'] {{(pid=129443) reserve /opt/stack/cinder/cinder/quota.py:755}} Mar 12 21:08:04.292907 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-9408ef28-ce7a-417e-a520-6d34f16a70c7 tempest-VolumesBackupsTest-1109530946 None] Delete backup finished, backup 9c644465-cf94-4bc3-9e83-5aa42b218df0 deleted. Mar 12 21:08:18.123181 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] Create backup started, backup: be532a8a-f536-44e2-bb12-cb0e3ae303c1 volume: f03f9c6c-6781-4beb-b4e3-e9cab44747d1. Mar 12 21:08:18.152400 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-03-12T21:08:17Z,data_timestamp=2022-03-12T21:08:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsTest-Backup-1876115641',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=be532a8a-f536-44e2-bb12-cb0e3ae303c1,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='e2ea29aba17e41fcac0b47522cf48b4f',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='c5601cc6a01a4c35bceb33df0cad0acc',volume_id=f03f9c6c-6781-4beb-b4e3-e9cab44747d1) Mar 12 21:08:18.217659 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.utils [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:08:18.223172 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:08:18.267632 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.046s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:08:18.267933 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[1c677c9c-af90-4275-80ea-d0cdd96e8a1f]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:77472599036\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:08:18.274112 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:08:18.291171 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 12 21:08:18.291625 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[654d5791-d113-49b3-a4e9-c34c7b3f84f2]: [Errno 2] No such file or directory: 'systool' {{(pid=157799) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 12 21:08:18.291625 ncdv-master-1 cinder-backup[129443]: Traceback (most recent call last): Mar 12 21:08:18.291625 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 12 21:08:18.291625 ncdv-master-1 cinder-backup[129443]: ret = func(*f_args, **f_kwargs) Mar 12 21:08:18.291625 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 12 21:08:18.291625 ncdv-master-1 cinder-backup[129443]: return func(*args, **kwargs) Mar 12 21:08:18.291625 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 12 21:08:18.291625 ncdv-master-1 cinder-backup[129443]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 12 21:08:18.291625 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 12 21:08:18.291625 ncdv-master-1 cinder-backup[129443]: return putils.execute(on_execute=on_execute, Mar 12 21:08:18.291625 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 12 21:08:18.291625 ncdv-master-1 cinder-backup[129443]: obj = subprocess.Popen(cmd, Mar 12 21:08:18.291625 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 12 21:08:18.291625 ncdv-master-1 cinder-backup[129443]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 12 21:08:18.291625 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 12 21:08:18.291625 ncdv-master-1 cinder-backup[129443]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 12 21:08:18.291625 ncdv-master-1 cinder-backup[129443]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 12 21:08:18.292607 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[654d5791-d113-49b3-a4e9-c34c7b3f84f2]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:08:18.293222 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:08:18.307407 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 12 21:08:18.307988 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[289c3935-9ed2-4f9a-be40-2ffb8bf9fa62]: [Errno 2] No such file or directory: 'systool' {{(pid=157799) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 12 21:08:18.307988 ncdv-master-1 cinder-backup[129443]: Traceback (most recent call last): Mar 12 21:08:18.307988 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 12 21:08:18.307988 ncdv-master-1 cinder-backup[129443]: ret = func(*f_args, **f_kwargs) Mar 12 21:08:18.307988 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 12 21:08:18.307988 ncdv-master-1 cinder-backup[129443]: return func(*args, **kwargs) Mar 12 21:08:18.307988 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 12 21:08:18.307988 ncdv-master-1 cinder-backup[129443]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 12 21:08:18.307988 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 12 21:08:18.307988 ncdv-master-1 cinder-backup[129443]: return putils.execute(on_execute=on_execute, Mar 12 21:08:18.307988 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 12 21:08:18.307988 ncdv-master-1 cinder-backup[129443]: obj = subprocess.Popen(cmd, Mar 12 21:08:18.307988 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 12 21:08:18.307988 ncdv-master-1 cinder-backup[129443]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 12 21:08:18.307988 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 12 21:08:18.307988 ncdv-master-1 cinder-backup[129443]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 12 21:08:18.307988 ncdv-master-1 cinder-backup[129443]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 12 21:08:18.309210 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[289c3935-9ed2-4f9a-be40-2ffb8bf9fa62]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:08:18.310054 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:08:18.337684 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.027s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:08:18.338263 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[2eb0725c-e6e2-4cf9-abad-990489277837]: (4, ('/dev/sda1\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:08:18.339637 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:08:18.375499 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.035s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:08:18.376261 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[240170ee-ce1c-48a7-a864-7cc9a59dfa39]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:08:18.377642 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:08:18.409961 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.030s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:08:18.410465 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[d358d2e7-3436-41f4-b8bd-19c0324ca321]: (4, ('55f4b1f4-b5b7-af4f-9adb-821e08822a61\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:08:18.411972 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] Running cmd (subprocess): nvme version {{(pid=129443) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:08:18.458257 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] CMD "nvme version" returned: 0 in 0.046s {{(pid=129443) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:08:18.462333 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129443) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 12 21:08:18.462824 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129443) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 12 21:08:18.463337 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:d9949f82-06c8-4f47-8c57-a55eecab76d2 dsc: {{(pid=129443) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 12 21:08:18.465470 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.utils [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] <== get_connector_properties: return (246ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:77472599036', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '55f4b1f4-b5b7-af4f-9adb-821e08822a61', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:d9949f82-06c8-4f47-8c57-a55eecab76d2', 'found_dsc': ''} {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:08:19.958251 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connector [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] Factory for rbd on None {{(pid=129443) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 12 21:08:19.961243 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-f03f9c6c-6781-4beb-b4e3-e9cab44747d1', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'f03f9c6c-6781-4beb-b4e3-e9cab44747d1', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:08:19.964289 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.linuxrbd [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] opening connection to ceph cluster (timeout=-1). {{(pid=129443) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 12 21:08:20.087553 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] <== connect_volume: return (125ms) {'path': } {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:08:20.100310 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_db.sqlalchemy.engines [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129443) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 12 21:08:20.167240 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] Starting backup of volume='f03f9c6c-6781-4beb-b4e3-e9cab44747d1'. {{(pid=129443) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Mar 12 21:08:20.168979 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] Volume file is RBD: attempting optimized backup {{(pid=129443) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Mar 12 21:08:20.279493 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] Creating base image 'volume-f03f9c6c-6781-4beb-b4e3-e9cab44747d1.backup.be532a8a-f536-44e2-bb12-cb0e3ae303c1' {{(pid=129443) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Mar 12 21:08:20.280245 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129443) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Mar 12 21:08:20.369295 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] Using --from-snap 'None' for incremental backup of volume f03f9c6c-6781-4beb-b4e3-e9cab44747d1. {{(pid=129443) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Mar 12 21:08:20.369957 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] Creating backup snapshot='backup.be532a8a-f536-44e2-bb12-cb0e3ae303c1.snap.1647119300.3695323' {{(pid=129443) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Mar 12 21:08:20.733980 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] Performing differential transfer from 'volume-f03f9c6c-6781-4beb-b4e3-e9cab44747d1' to 'volume-f03f9c6c-6781-4beb-b4e3-e9cab44747d1.backup.be532a8a-f536-44e2-bb12-cb0e3ae303c1' {{(pid=129443) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Mar 12 21:08:20.734483 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_q0mirmrs --pool volumes volumes/volume-f03f9c6c-6781-4beb-b4e3-e9cab44747d1@backup.be532a8a-f536-44e2-bb12-cb0e3ae303c1.snap.1647119300.3695323 -' into... {{(pid=129443) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Mar 12 21:08:20.734869 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-f03f9c6c-6781-4beb-b4e3-e9cab44747d1.backup.be532a8a-f536-44e2-bb12-cb0e3ae303c1' {{(pid=129443) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Mar 12 21:08:29.807657 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] Differential backup transfer completed in 9.0732s {{(pid=129443) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Mar 12 21:08:29.808646 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] Getting metadata type 'volume-base-metadata' {{(pid=129443) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Mar 12 21:08:29.822613 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129443) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Mar 12 21:08:29.823014 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] Getting metadata type 'volume-metadata' {{(pid=129443) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Mar 12 21:08:29.839184 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] No metadata type 'volume-metadata' available {{(pid=129443) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Mar 12 21:08:29.841986 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] Getting metadata type 'volume-glance-metadata' {{(pid=129443) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Mar 12 21:08:29.858738 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=129443) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}} Mar 12 21:08:29.859921 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] Backing up metadata for volume f03f9c6c-6781-4beb-b4e3-e9cab44747d1. {{(pid=129443) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Mar 12 21:08:29.941149 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] Backup 'be532a8a-f536-44e2-bb12-cb0e3ae303c1' of volume f03f9c6c-6781-4beb-b4e3-e9cab44747d1 finished. {{(pid=129443) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Mar 12 21:08:29.942997 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-f03f9c6c-6781-4beb-b4e3-e9cab44747d1', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'f03f9c6c-6781-4beb-b4e3-e9cab44747d1', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:08:29.972176 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] <== disconnect_volume: return (27ms) None {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:08:30.064018 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-ec2d74ee-1852-4db0-988f-24f676b6b6b0 tempest-VolumesBackupsTest-1109530946 None] Create backup finished. backup: be532a8a-f536-44e2-bb12-cb0e3ae303c1. Mar 12 21:08:32.397878 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] Restore backup started, backup: be532a8a-f536-44e2-bb12-cb0e3ae303c1 volume: 2f3956a4-e9b1-4ded-ab9b-6e1370102687. Mar 12 21:08:32.419002 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.utils [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:08:32.438271 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:08:32.467258 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.029s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:08:32.467598 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[ef0c5c06-ac07-4f92-aba1-81e3f11f1da2]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:77472599036\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:08:32.471870 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:08:32.489683 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 12 21:08:32.490130 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[af05cb43-7229-47d9-985f-4366cff779fe]: [Errno 2] No such file or directory: 'systool' {{(pid=157799) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 12 21:08:32.490130 ncdv-master-1 cinder-backup[129443]: Traceback (most recent call last): Mar 12 21:08:32.490130 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 12 21:08:32.490130 ncdv-master-1 cinder-backup[129443]: ret = func(*f_args, **f_kwargs) Mar 12 21:08:32.490130 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 12 21:08:32.490130 ncdv-master-1 cinder-backup[129443]: return func(*args, **kwargs) Mar 12 21:08:32.490130 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 12 21:08:32.490130 ncdv-master-1 cinder-backup[129443]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 12 21:08:32.490130 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 12 21:08:32.490130 ncdv-master-1 cinder-backup[129443]: return putils.execute(on_execute=on_execute, Mar 12 21:08:32.490130 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 12 21:08:32.490130 ncdv-master-1 cinder-backup[129443]: obj = subprocess.Popen(cmd, Mar 12 21:08:32.490130 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 12 21:08:32.490130 ncdv-master-1 cinder-backup[129443]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 12 21:08:32.490130 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 12 21:08:32.490130 ncdv-master-1 cinder-backup[129443]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 12 21:08:32.490130 ncdv-master-1 cinder-backup[129443]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 12 21:08:32.491455 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[af05cb43-7229-47d9-985f-4366cff779fe]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:08:32.494734 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:08:32.505334 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 12 21:08:32.507266 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[f911ac9d-0ba3-4374-b1fb-02392ef8ea6f]: [Errno 2] No such file or directory: 'systool' {{(pid=157799) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 12 21:08:32.507266 ncdv-master-1 cinder-backup[129443]: Traceback (most recent call last): Mar 12 21:08:32.507266 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 12 21:08:32.507266 ncdv-master-1 cinder-backup[129443]: ret = func(*f_args, **f_kwargs) Mar 12 21:08:32.507266 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 12 21:08:32.507266 ncdv-master-1 cinder-backup[129443]: return func(*args, **kwargs) Mar 12 21:08:32.507266 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 12 21:08:32.507266 ncdv-master-1 cinder-backup[129443]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 12 21:08:32.507266 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 12 21:08:32.507266 ncdv-master-1 cinder-backup[129443]: return putils.execute(on_execute=on_execute, Mar 12 21:08:32.507266 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 12 21:08:32.507266 ncdv-master-1 cinder-backup[129443]: obj = subprocess.Popen(cmd, Mar 12 21:08:32.507266 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 12 21:08:32.507266 ncdv-master-1 cinder-backup[129443]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 12 21:08:32.507266 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 12 21:08:32.507266 ncdv-master-1 cinder-backup[129443]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 12 21:08:32.507266 ncdv-master-1 cinder-backup[129443]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 12 21:08:32.509398 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[f911ac9d-0ba3-4374-b1fb-02392ef8ea6f]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:08:32.515886 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:08:32.554952 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.040s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:08:32.556759 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[c875ad34-ed35-4c43-8ebd-0cea96ddccb4]: (4, ('/dev/sda1\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:08:32.558150 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:08:32.604121 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.045s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:08:32.604460 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[d45361d6-1428-4601-b41b-ef61989a340d]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:08:32.605659 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:08:32.636120 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.030s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:08:32.636406 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[076b9034-a6be-4b53-80d2-4b9ca88905db]: (4, ('55f4b1f4-b5b7-af4f-9adb-821e08822a61\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:08:32.637450 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] Running cmd (subprocess): nvme version {{(pid=129443) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:08:32.674459 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] CMD "nvme version" returned: 0 in 0.037s {{(pid=129443) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:08:32.678141 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129443) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 12 21:08:32.678716 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129443) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 12 21:08:32.679263 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:d9949f82-06c8-4f47-8c57-a55eecab76d2 dsc: {{(pid=129443) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 12 21:08:32.680753 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.utils [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] <== get_connector_properties: return (242ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:77472599036', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '55f4b1f4-b5b7-af4f-9adb-821e08822a61', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:d9949f82-06c8-4f47-8c57-a55eecab76d2', 'found_dsc': ''} {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:08:34.125604 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connector [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] Factory for rbd on None {{(pid=129443) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 12 21:08:34.127053 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-2f3956a4-e9b1-4ded-ab9b-6e1370102687', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '2f3956a4-e9b1-4ded-ab9b-6e1370102687', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:08:34.128503 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.linuxrbd [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] opening connection to ceph cluster (timeout=-1). {{(pid=129443) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 12 21:08:34.223365 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] <== connect_volume: return (95ms) {'path': } {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:08:34.235158 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_db.sqlalchemy.engines [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129443) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 12 21:08:34.248979 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] Starting restore from Ceph backup=be532a8a-f536-44e2-bb12-cb0e3ae303c1 to volume=volume-2f3956a4-e9b1-4ded-ab9b-6e1370102687 {{(pid=129443) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1274}} Mar 12 21:08:34.359898 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] Looking for snapshot of backup base 'volume-f03f9c6c-6781-4beb-b4e3-e9cab44747d1.backup.be532a8a-f536-44e2-bb12-cb0e3ae303c1' {{(pid=129443) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 12 21:08:34.360493 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] Found snapshot 'backup.be532a8a-f536-44e2-bb12-cb0e3ae303c1.snap.1647119300.3695323' {{(pid=129443) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Mar 12 21:08:34.366578 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] Volume file is RBD. {{(pid=129443) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1184}} Mar 12 21:08:34.435553 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] Attempting differential restore. {{(pid=129443) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1234}} Mar 12 21:08:34.436534 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] Attempting incremental restore from base='volume-f03f9c6c-6781-4beb-b4e3-e9cab44747d1.backup.be532a8a-f536-44e2-bb12-cb0e3ae303c1' snap='backup.be532a8a-f536-44e2-bb12-cb0e3ae303c1.snap.1647119300.3695323' {{(pid=129443) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1089}} Mar 12 21:08:34.437643 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] Performing differential transfer from 'volume-f03f9c6c-6781-4beb-b4e3-e9cab44747d1.backup.be532a8a-f536-44e2-bb12-cb0e3ae303c1' to 'volume-2f3956a4-e9b1-4ded-ab9b-6e1370102687' {{(pid=129443) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Mar 12 21:08:34.438370 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-f03f9c6c-6781-4beb-b4e3-e9cab44747d1.backup.be532a8a-f536-44e2-bb12-cb0e3ae303c1@backup.be532a8a-f536-44e2-bb12-cb0e3ae303c1.snap.1647119300.3695323 -' into... {{(pid=129443) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Mar 12 21:08:34.439002 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_qve3ysqa --pool volumes - volumes/volume-2f3956a4-e9b1-4ded-ab9b-6e1370102687' {{(pid=129443) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Mar 12 21:08:34.820648 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:08:34.821186 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:08:40.901878 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] Restore transfer completed in 6.4639s {{(pid=129443) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1111}} Mar 12 21:08:40.903015 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=129443) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1285}} Mar 12 21:08:40.987226 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] Restoring volume base metadata {{(pid=129443) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} Mar 12 21:08:41.040322 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] No metadata of type 'volume-metadata' to restore {{(pid=129443) put /opt/stack/cinder/cinder/backup/driver.py:345}} Mar 12 21:08:41.041037 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] Restoring volume glance metadata {{(pid=129443) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}} Mar 12 21:08:41.497794 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] Restore to volume 2f3956a4-e9b1-4ded-ab9b-6e1370102687 finished successfully. {{(pid=129443) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1292}} Mar 12 21:08:41.502169 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-2f3956a4-e9b1-4ded-ab9b-6e1370102687', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '2f3956a4-e9b1-4ded-ab9b-6e1370102687', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:08:41.530056 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] <== disconnect_volume: return (26ms) None {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:08:41.620970 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.manager [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] Encryption key ID for volume 2f3956a4-e9b1-4ded-ab9b-6e1370102687 already matches encryption key ID in backup be532a8a-f536-44e2-bb12-cb0e3ae303c1. {{(pid=129443) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} Mar 12 21:08:41.704689 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-a6970f7e-d78e-4c5f-81fe-0a2a82ae0382 tempest-VolumesBackupsTest-1109530946 None] Finished restoring backup be532a8a-f536-44e2-bb12-cb0e3ae303c1 to volume 2f3956a4-e9b1-4ded-ab9b-6e1370102687. Mar 12 21:08:45.778171 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-5d0aab05-f13e-4918-8f0e-6c5fd0122ff5 tempest-VolumesBackupsTest-1109530946 None] Delete backup started, backup: be532a8a-f536-44e2-bb12-cb0e3ae303c1. Mar 12 21:08:45.784081 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-5d0aab05-f13e-4918-8f0e-6c5fd0122ff5 tempest-VolumesBackupsTest-1109530946 None] Delete started for backup=be532a8a-f536-44e2-bb12-cb0e3ae303c1 {{(pid=129443) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Mar 12 21:08:45.784714 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-5d0aab05-f13e-4918-8f0e-6c5fd0122ff5 tempest-VolumesBackupsTest-1109530946 None] Trying diff format basename='volume-f03f9c6c-6781-4beb-b4e3-e9cab44747d1.backup.be532a8a-f536-44e2-bb12-cb0e3ae303c1' for backup base image of volume f03f9c6c-6781-4beb-b4e3-e9cab44747d1. {{(pid=129443) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Mar 12 21:08:45.918287 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-5d0aab05-f13e-4918-8f0e-6c5fd0122ff5 tempest-VolumesBackupsTest-1109530946 None] Looking for snapshot of backup base 'volume-f03f9c6c-6781-4beb-b4e3-e9cab44747d1.backup.be532a8a-f536-44e2-bb12-cb0e3ae303c1' {{(pid=129443) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 12 21:08:45.918983 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-5d0aab05-f13e-4918-8f0e-6c5fd0122ff5 tempest-VolumesBackupsTest-1109530946 None] Found snapshot 'backup.be532a8a-f536-44e2-bb12-cb0e3ae303c1.snap.1647119300.3695323' {{(pid=129443) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Mar 12 21:08:45.919911 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-5d0aab05-f13e-4918-8f0e-6c5fd0122ff5 tempest-VolumesBackupsTest-1109530946 None] Deleting backup snapshot='backup.be532a8a-f536-44e2-bb12-cb0e3ae303c1.snap.1647119300.3695323' {{(pid=129443) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Mar 12 21:08:46.753904 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.drivers.ceph [None req-5d0aab05-f13e-4918-8f0e-6c5fd0122ff5 tempest-VolumesBackupsTest-1109530946 None] Deleting backup base image='volume-f03f9c6c-6781-4beb-b4e3-e9cab44747d1.backup.be532a8a-f536-44e2-bb12-cb0e3ae303c1' of volume f03f9c6c-6781-4beb-b4e3-e9cab44747d1. Mar 12 21:08:47.558021 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-5d0aab05-f13e-4918-8f0e-6c5fd0122ff5 tempest-VolumesBackupsTest-1109530946 None] Base backup image='volume-f03f9c6c-6781-4beb-b4e3-e9cab44747d1.backup.be532a8a-f536-44e2-bb12-cb0e3ae303c1' of volume f03f9c6c-6781-4beb-b4e3-e9cab44747d1 deleted. {{(pid=129443) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Mar 12 21:08:47.637985 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-5d0aab05-f13e-4918-8f0e-6c5fd0122ff5 tempest-VolumesBackupsTest-1109530946 None] Delete of backup 'be532a8a-f536-44e2-bb12-cb0e3ae303c1' for volume 'f03f9c6c-6781-4beb-b4e3-e9cab44747d1' finished. {{(pid=129443) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Mar 12 21:08:47.644217 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_db.sqlalchemy.engines [None req-5d0aab05-f13e-4918-8f0e-6c5fd0122ff5 tempest-VolumesBackupsTest-1109530946 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129443) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 12 21:08:47.703809 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.quota [None req-5d0aab05-f13e-4918-8f0e-6c5fd0122ff5 tempest-VolumesBackupsTest-1109530946 None] Created reservations ['d0e79bf4-11b6-419c-a694-583e03c901f5', '1f8e8ddc-a5d9-457c-8439-78b44ac5a40e'] {{(pid=129443) reserve /opt/stack/cinder/cinder/quota.py:755}} Mar 12 21:08:47.763063 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-5d0aab05-f13e-4918-8f0e-6c5fd0122ff5 tempest-VolumesBackupsTest-1109530946 None] Delete backup finished, backup be532a8a-f536-44e2-bb12-cb0e3ae303c1 deleted. Mar 12 21:08:58.376093 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] Create backup started, backup: 460e2942-9409-42b0-b815-27d08e1c672a volume: 2b938d6a-5fce-41aa-a3d6-f0e32446feff. Mar 12 21:08:58.393325 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-03-12T21:08:58Z,data_timestamp=2022-03-12T21:08:58Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsV39Test-Backup-1682607007',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=460e2942-9409-42b0-b815-27d08e1c672a,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='0576147e4eaf414bbc49a330fd52a7e3',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='5b88500f5cbd458081b2574add4892b5',volume_id=2b938d6a-5fce-41aa-a3d6-f0e32446feff) Mar 12 21:08:58.490155 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.utils [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:08:58.494278 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:08:58.531263 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.037s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:08:58.531589 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[cff5ef7f-2e10-4cd8-b4e5-cf605eccb13a]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:77472599036\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:08:58.533590 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:08:58.556645 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 12 21:08:58.556944 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[eda557e5-a98e-480b-b803-6ea3232d5b72]: [Errno 2] No such file or directory: 'systool' {{(pid=157799) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 12 21:08:58.556944 ncdv-master-1 cinder-backup[129443]: Traceback (most recent call last): Mar 12 21:08:58.556944 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 12 21:08:58.556944 ncdv-master-1 cinder-backup[129443]: ret = func(*f_args, **f_kwargs) Mar 12 21:08:58.556944 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 12 21:08:58.556944 ncdv-master-1 cinder-backup[129443]: return func(*args, **kwargs) Mar 12 21:08:58.556944 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 12 21:08:58.556944 ncdv-master-1 cinder-backup[129443]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 12 21:08:58.556944 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 12 21:08:58.556944 ncdv-master-1 cinder-backup[129443]: return putils.execute(on_execute=on_execute, Mar 12 21:08:58.556944 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 12 21:08:58.556944 ncdv-master-1 cinder-backup[129443]: obj = subprocess.Popen(cmd, Mar 12 21:08:58.556944 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 12 21:08:58.556944 ncdv-master-1 cinder-backup[129443]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 12 21:08:58.556944 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 12 21:08:58.556944 ncdv-master-1 cinder-backup[129443]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 12 21:08:58.556944 ncdv-master-1 cinder-backup[129443]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 12 21:08:58.558309 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[eda557e5-a98e-480b-b803-6ea3232d5b72]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:08:58.560806 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:08:58.574532 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 12 21:08:58.574827 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[af81f50f-f221-48d2-8b09-b9d08ef5a06e]: [Errno 2] No such file or directory: 'systool' {{(pid=157799) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 12 21:08:58.574827 ncdv-master-1 cinder-backup[129443]: Traceback (most recent call last): Mar 12 21:08:58.574827 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 12 21:08:58.574827 ncdv-master-1 cinder-backup[129443]: ret = func(*f_args, **f_kwargs) Mar 12 21:08:58.574827 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 12 21:08:58.574827 ncdv-master-1 cinder-backup[129443]: return func(*args, **kwargs) Mar 12 21:08:58.574827 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 12 21:08:58.574827 ncdv-master-1 cinder-backup[129443]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 12 21:08:58.574827 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 12 21:08:58.574827 ncdv-master-1 cinder-backup[129443]: return putils.execute(on_execute=on_execute, Mar 12 21:08:58.574827 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 12 21:08:58.574827 ncdv-master-1 cinder-backup[129443]: obj = subprocess.Popen(cmd, Mar 12 21:08:58.574827 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 12 21:08:58.574827 ncdv-master-1 cinder-backup[129443]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 12 21:08:58.574827 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 12 21:08:58.574827 ncdv-master-1 cinder-backup[129443]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 12 21:08:58.574827 ncdv-master-1 cinder-backup[129443]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 12 21:08:58.576016 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[af81f50f-f221-48d2-8b09-b9d08ef5a06e]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:08:58.578799 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:08:58.624378 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.047s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:08:58.625805 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[36ef47a4-6b12-4c3a-a34a-169232e68578]: (4, ('/dev/sda1\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:08:58.627177 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:08:58.675383 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.048s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:08:58.676026 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[b6ce842a-4e7d-447e-b35a-962656f1afcf]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:08:58.680018 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:08:58.718738 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.040s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:08:58.721663 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[a86f90d9-a024-441c-adb6-c676aeaff70f]: (4, ('55f4b1f4-b5b7-af4f-9adb-821e08822a61\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:08:58.723124 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] Running cmd (subprocess): nvme version {{(pid=129443) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:08:58.794423 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] CMD "nvme version" returned: 0 in 0.071s {{(pid=129443) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:08:58.796762 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129443) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 12 21:08:58.797332 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129443) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 12 21:08:58.798013 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:d9949f82-06c8-4f47-8c57-a55eecab76d2 dsc: {{(pid=129443) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 12 21:08:58.798813 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.utils [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] <== get_connector_properties: return (305ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:77472599036', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '55f4b1f4-b5b7-af4f-9adb-821e08822a61', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:d9949f82-06c8-4f47-8c57-a55eecab76d2', 'found_dsc': ''} {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:09:00.000241 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connector [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] Factory for rbd on None {{(pid=129443) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 12 21:09:00.006754 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-2b938d6a-5fce-41aa-a3d6-f0e32446feff', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '2b938d6a-5fce-41aa-a3d6-f0e32446feff', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:09:00.007580 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.linuxrbd [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] opening connection to ceph cluster (timeout=-1). {{(pid=129443) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 12 21:09:00.093633 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] <== connect_volume: return (86ms) {'path': } {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:09:00.105657 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_db.sqlalchemy.engines [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129443) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 12 21:09:00.140318 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] Starting backup of volume='2b938d6a-5fce-41aa-a3d6-f0e32446feff'. {{(pid=129443) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Mar 12 21:09:00.142313 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] Volume file is RBD: attempting optimized backup {{(pid=129443) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Mar 12 21:09:00.197510 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] Creating base image 'volume-2b938d6a-5fce-41aa-a3d6-f0e32446feff.backup.460e2942-9409-42b0-b815-27d08e1c672a' {{(pid=129443) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Mar 12 21:09:00.198922 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129443) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Mar 12 21:09:00.260602 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] Using --from-snap 'None' for incremental backup of volume 2b938d6a-5fce-41aa-a3d6-f0e32446feff. {{(pid=129443) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Mar 12 21:09:00.261105 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] Creating backup snapshot='backup.460e2942-9409-42b0-b815-27d08e1c672a.snap.1647119340.260629' {{(pid=129443) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Mar 12 21:09:00.872298 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] Performing differential transfer from 'volume-2b938d6a-5fce-41aa-a3d6-f0e32446feff' to 'volume-2b938d6a-5fce-41aa-a3d6-f0e32446feff.backup.460e2942-9409-42b0-b815-27d08e1c672a' {{(pid=129443) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Mar 12 21:09:00.872715 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_uz4trome --pool volumes volumes/volume-2b938d6a-5fce-41aa-a3d6-f0e32446feff@backup.460e2942-9409-42b0-b815-27d08e1c672a.snap.1647119340.260629 -' into... {{(pid=129443) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Mar 12 21:09:00.873081 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-2b938d6a-5fce-41aa-a3d6-f0e32446feff.backup.460e2942-9409-42b0-b815-27d08e1c672a' {{(pid=129443) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Mar 12 21:09:02.028864 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] Differential backup transfer completed in 1.1561s {{(pid=129443) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Mar 12 21:09:02.038275 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] Getting metadata type 'volume-base-metadata' {{(pid=129443) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Mar 12 21:09:02.080109 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129443) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Mar 12 21:09:02.080858 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] Getting metadata type 'volume-metadata' {{(pid=129443) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Mar 12 21:09:02.105722 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] No metadata type 'volume-metadata' available {{(pid=129443) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Mar 12 21:09:02.106798 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] Getting metadata type 'volume-glance-metadata' {{(pid=129443) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Mar 12 21:09:02.134590 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] No metadata type 'volume-glance-metadata' available {{(pid=129443) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Mar 12 21:09:02.137037 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] Backing up metadata for volume 2b938d6a-5fce-41aa-a3d6-f0e32446feff. {{(pid=129443) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Mar 12 21:09:02.212180 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] Backup '460e2942-9409-42b0-b815-27d08e1c672a' of volume 2b938d6a-5fce-41aa-a3d6-f0e32446feff finished. {{(pid=129443) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Mar 12 21:09:02.214325 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-2b938d6a-5fce-41aa-a3d6-f0e32446feff', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '2b938d6a-5fce-41aa-a3d6-f0e32446feff', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:09:02.255036 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] <== disconnect_volume: return (39ms) None {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:09:02.396335 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-087464b7-e129-4688-bb47-80586dc06a31 tempest-VolumesBackupsV39Test-777707830 None] Create backup finished. backup: 460e2942-9409-42b0-b815-27d08e1c672a. Mar 12 21:09:02.967841 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-15e1cb18-d415-4296-a673-7d9935e20587 tempest-VolumesBackupsV39Test-777707830 None] Delete backup started, backup: 460e2942-9409-42b0-b815-27d08e1c672a. Mar 12 21:09:02.970253 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-15e1cb18-d415-4296-a673-7d9935e20587 tempest-VolumesBackupsV39Test-777707830 None] Delete started for backup=460e2942-9409-42b0-b815-27d08e1c672a {{(pid=129443) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Mar 12 21:09:02.971107 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-15e1cb18-d415-4296-a673-7d9935e20587 tempest-VolumesBackupsV39Test-777707830 None] Trying diff format basename='volume-2b938d6a-5fce-41aa-a3d6-f0e32446feff.backup.460e2942-9409-42b0-b815-27d08e1c672a' for backup base image of volume 2b938d6a-5fce-41aa-a3d6-f0e32446feff. {{(pid=129443) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Mar 12 21:09:03.090302 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-15e1cb18-d415-4296-a673-7d9935e20587 tempest-VolumesBackupsV39Test-777707830 None] Looking for snapshot of backup base 'volume-2b938d6a-5fce-41aa-a3d6-f0e32446feff.backup.460e2942-9409-42b0-b815-27d08e1c672a' {{(pid=129443) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 12 21:09:03.091467 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-15e1cb18-d415-4296-a673-7d9935e20587 tempest-VolumesBackupsV39Test-777707830 None] Found snapshot 'backup.460e2942-9409-42b0-b815-27d08e1c672a.snap.1647119340.260629' {{(pid=129443) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Mar 12 21:09:03.091994 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-15e1cb18-d415-4296-a673-7d9935e20587 tempest-VolumesBackupsV39Test-777707830 None] Deleting backup snapshot='backup.460e2942-9409-42b0-b815-27d08e1c672a.snap.1647119340.260629' {{(pid=129443) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Mar 12 21:09:04.037971 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.drivers.ceph [None req-15e1cb18-d415-4296-a673-7d9935e20587 tempest-VolumesBackupsV39Test-777707830 None] Deleting backup base image='volume-2b938d6a-5fce-41aa-a3d6-f0e32446feff.backup.460e2942-9409-42b0-b815-27d08e1c672a' of volume 2b938d6a-5fce-41aa-a3d6-f0e32446feff. Mar 12 21:09:04.501290 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-15e1cb18-d415-4296-a673-7d9935e20587 tempest-VolumesBackupsV39Test-777707830 None] Base backup image='volume-2b938d6a-5fce-41aa-a3d6-f0e32446feff.backup.460e2942-9409-42b0-b815-27d08e1c672a' of volume 2b938d6a-5fce-41aa-a3d6-f0e32446feff deleted. {{(pid=129443) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Mar 12 21:09:04.580377 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-15e1cb18-d415-4296-a673-7d9935e20587 tempest-VolumesBackupsV39Test-777707830 None] Delete of backup '460e2942-9409-42b0-b815-27d08e1c672a' for volume '2b938d6a-5fce-41aa-a3d6-f0e32446feff' finished. {{(pid=129443) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Mar 12 21:09:04.590227 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_db.sqlalchemy.engines [None req-15e1cb18-d415-4296-a673-7d9935e20587 tempest-VolumesBackupsV39Test-777707830 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129443) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 12 21:09:04.664072 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.quota [None req-15e1cb18-d415-4296-a673-7d9935e20587 tempest-VolumesBackupsV39Test-777707830 None] Created reservations ['5cf97148-a9ff-42d1-952b-85024eff4237', '80d92129-3fe7-4d7b-82d3-94c5224695f5'] {{(pid=129443) reserve /opt/stack/cinder/cinder/quota.py:755}} Mar 12 21:09:04.711583 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-15e1cb18-d415-4296-a673-7d9935e20587 tempest-VolumesBackupsV39Test-777707830 None] Delete backup finished, backup 460e2942-9409-42b0-b815-27d08e1c672a deleted. Mar 12 21:09:26.751875 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] Create backup started, backup: e93bc020-d569-4332-8c60-503b12f4d476 volume: acbf2fdf-7efb-4dc4-8ce1-64607b6441bf. Mar 12 21:09:26.781106 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-03-12T21:09:26Z,data_timestamp=2022-03-12T21:09:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsAdminTest-Backup-971827320',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=e93bc020-d569-4332-8c60-503b12f4d476,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='76614dd1780d41e99b3db646e59d928c',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='3d36ef1eff0649179221db760b119396',volume_id=acbf2fdf-7efb-4dc4-8ce1-64607b6441bf) Mar 12 21:09:27.164307 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.utils [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:09:27.170780 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:09:27.219241 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.052s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:09:27.220121 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[deb661e1-92e2-49f3-9c15-15f962c80be0]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:77472599036\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:09:27.225571 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:09:27.248537 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 12 21:09:27.248836 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[3b423b47-d4d6-48e6-b25f-3d1e8fc05629]: [Errno 2] No such file or directory: 'systool' {{(pid=157799) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 12 21:09:27.248836 ncdv-master-1 cinder-backup[129443]: Traceback (most recent call last): Mar 12 21:09:27.248836 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 12 21:09:27.248836 ncdv-master-1 cinder-backup[129443]: ret = func(*f_args, **f_kwargs) Mar 12 21:09:27.248836 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 12 21:09:27.248836 ncdv-master-1 cinder-backup[129443]: return func(*args, **kwargs) Mar 12 21:09:27.248836 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 12 21:09:27.248836 ncdv-master-1 cinder-backup[129443]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 12 21:09:27.248836 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 12 21:09:27.248836 ncdv-master-1 cinder-backup[129443]: return putils.execute(on_execute=on_execute, Mar 12 21:09:27.248836 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 12 21:09:27.248836 ncdv-master-1 cinder-backup[129443]: obj = subprocess.Popen(cmd, Mar 12 21:09:27.248836 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 12 21:09:27.248836 ncdv-master-1 cinder-backup[129443]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 12 21:09:27.248836 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 12 21:09:27.248836 ncdv-master-1 cinder-backup[129443]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 12 21:09:27.248836 ncdv-master-1 cinder-backup[129443]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 12 21:09:27.250131 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[3b423b47-d4d6-48e6-b25f-3d1e8fc05629]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:09:27.252706 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:09:27.286747 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 12 21:09:27.287061 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[a7111b2a-fc0b-4434-9b06-124e3037db3b]: [Errno 2] No such file or directory: 'systool' {{(pid=157799) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 12 21:09:27.287061 ncdv-master-1 cinder-backup[129443]: Traceback (most recent call last): Mar 12 21:09:27.287061 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 12 21:09:27.287061 ncdv-master-1 cinder-backup[129443]: ret = func(*f_args, **f_kwargs) Mar 12 21:09:27.287061 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 12 21:09:27.287061 ncdv-master-1 cinder-backup[129443]: return func(*args, **kwargs) Mar 12 21:09:27.287061 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 12 21:09:27.287061 ncdv-master-1 cinder-backup[129443]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 12 21:09:27.287061 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 12 21:09:27.287061 ncdv-master-1 cinder-backup[129443]: return putils.execute(on_execute=on_execute, Mar 12 21:09:27.287061 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 12 21:09:27.287061 ncdv-master-1 cinder-backup[129443]: obj = subprocess.Popen(cmd, Mar 12 21:09:27.287061 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 12 21:09:27.287061 ncdv-master-1 cinder-backup[129443]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 12 21:09:27.287061 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 12 21:09:27.287061 ncdv-master-1 cinder-backup[129443]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 12 21:09:27.287061 ncdv-master-1 cinder-backup[129443]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 12 21:09:27.288447 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[a7111b2a-fc0b-4434-9b06-124e3037db3b]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:09:27.289259 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:09:27.341168 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.051s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:09:27.341604 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[b60ea8d9-78a8-48f9-8aa6-1020302a74a7]: (4, ('/dev/sda1\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:09:27.346551 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:09:27.400434 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.054s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:09:27.400904 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[86317a8d-4244-4561-a1da-fb84d3ab94d5]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:09:27.405172 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:09:27.458724 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.055s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:09:27.458948 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[a651859f-09d7-4c57-aff0-1be39b948e40]: (4, ('55f4b1f4-b5b7-af4f-9adb-821e08822a61\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:09:27.459735 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] Running cmd (subprocess): nvme version {{(pid=129443) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:09:27.547787 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] CMD "nvme version" returned: 0 in 0.088s {{(pid=129443) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:09:27.550202 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129443) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 12 21:09:27.550793 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129443) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 12 21:09:27.551919 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:d9949f82-06c8-4f47-8c57-a55eecab76d2 dsc: {{(pid=129443) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 12 21:09:27.552490 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.utils [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] <== get_connector_properties: return (387ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:77472599036', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '55f4b1f4-b5b7-af4f-9adb-821e08822a61', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:d9949f82-06c8-4f47-8c57-a55eecab76d2', 'found_dsc': ''} {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:09:29.877541 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connector [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] Factory for rbd on None {{(pid=129443) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 12 21:09:29.881507 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-acbf2fdf-7efb-4dc4-8ce1-64607b6441bf', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'acbf2fdf-7efb-4dc4-8ce1-64607b6441bf', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:09:29.883480 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.linuxrbd [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] opening connection to ceph cluster (timeout=-1). {{(pid=129443) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 12 21:09:29.991761 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] <== connect_volume: return (109ms) {'path': } {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:09:30.006668 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_db.sqlalchemy.engines [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129443) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 12 21:09:30.042765 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] Starting backup of volume='acbf2fdf-7efb-4dc4-8ce1-64607b6441bf'. {{(pid=129443) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Mar 12 21:09:30.051292 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] Volume file is RBD: attempting optimized backup {{(pid=129443) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Mar 12 21:09:30.133708 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] Creating base image 'volume-acbf2fdf-7efb-4dc4-8ce1-64607b6441bf.backup.e93bc020-d569-4332-8c60-503b12f4d476' {{(pid=129443) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Mar 12 21:09:30.134718 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129443) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Mar 12 21:09:30.194249 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] Using --from-snap 'None' for incremental backup of volume acbf2fdf-7efb-4dc4-8ce1-64607b6441bf. {{(pid=129443) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Mar 12 21:09:30.194641 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] Creating backup snapshot='backup.e93bc020-d569-4332-8c60-503b12f4d476.snap.1647119370.194113' {{(pid=129443) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Mar 12 21:09:30.905432 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] Performing differential transfer from 'volume-acbf2fdf-7efb-4dc4-8ce1-64607b6441bf' to 'volume-acbf2fdf-7efb-4dc4-8ce1-64607b6441bf.backup.e93bc020-d569-4332-8c60-503b12f4d476' {{(pid=129443) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Mar 12 21:09:30.907201 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_y011tfhu --pool volumes volumes/volume-acbf2fdf-7efb-4dc4-8ce1-64607b6441bf@backup.e93bc020-d569-4332-8c60-503b12f4d476.snap.1647119370.194113 -' into... {{(pid=129443) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Mar 12 21:09:30.907201 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-acbf2fdf-7efb-4dc4-8ce1-64607b6441bf.backup.e93bc020-d569-4332-8c60-503b12f4d476' {{(pid=129443) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Mar 12 21:09:31.992070 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] Differential backup transfer completed in 1.0864s {{(pid=129443) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Mar 12 21:09:31.992750 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] Getting metadata type 'volume-base-metadata' {{(pid=129443) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Mar 12 21:09:32.016667 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129443) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Mar 12 21:09:32.017080 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] Getting metadata type 'volume-metadata' {{(pid=129443) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Mar 12 21:09:32.034576 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] No metadata type 'volume-metadata' available {{(pid=129443) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Mar 12 21:09:32.035308 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] Getting metadata type 'volume-glance-metadata' {{(pid=129443) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Mar 12 21:09:32.042327 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] No metadata type 'volume-glance-metadata' available {{(pid=129443) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Mar 12 21:09:32.043234 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] Backing up metadata for volume acbf2fdf-7efb-4dc4-8ce1-64607b6441bf. {{(pid=129443) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Mar 12 21:09:32.139696 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] Backup 'e93bc020-d569-4332-8c60-503b12f4d476' of volume acbf2fdf-7efb-4dc4-8ce1-64607b6441bf finished. {{(pid=129443) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Mar 12 21:09:32.141698 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-acbf2fdf-7efb-4dc4-8ce1-64607b6441bf', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'acbf2fdf-7efb-4dc4-8ce1-64607b6441bf', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:09:32.185283 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] <== disconnect_volume: return (42ms) None {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:09:32.572456 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-4e13fb3b-5f3a-4795-8289-b25eb64dd623 tempest-VolumesBackupsAdminTest-1513847769 None] Create backup finished. backup: e93bc020-d569-4332-8c60-503b12f4d476. Mar 12 21:09:33.818081 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-1a32f0ff-21be-4887-b011-b2533cfc13aa tempest-VolumesBackupsAdminTest-1519964800 None] Export record started, backup: e93bc020-d569-4332-8c60-503b12f4d476. Mar 12 21:09:33.821032 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-1a32f0ff-21be-4887-b011-b2533cfc13aa tempest-VolumesBackupsAdminTest-1519964800 None] Export record finished, backup e93bc020-d569-4332-8c60-503b12f4d476 exported. Mar 12 21:09:34.065197 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-937422f7-ce97-44a3-9145-2694d697ca79 tempest-VolumesBackupsAdminTest-1519964800 None] Import record started, backup_url: eyJkcml2ZXJfaW5mbyI6IHt9LCAiaWQiOiAiMDc5NjZhMjYtNWI2My00OGJkLTgxNjItODI2YThjZTJlM2VmIiwgInVzZXJfaWQiOiAiM2QzNmVmMWVmZjA2NDkxNzkyMjFkYjc2MGIxMTkzOTYiLCAicHJvamVjdF9pZCI6ICI3NjYxNGRkMTc4MGQ0MWU5OWIzZGI2NDZlNTlkOTI4YyIsICJ2b2x1bWVfaWQiOiAiYWNiZjJmZGYtN2VmYi00ZGM0LThjZTEtNjQ2MDdiNjQ0MWJmIiwgImhvc3QiOiAibmNkdi1tYXN0ZXItMSIsICJhdmFpbGFiaWxpdHlfem9uZSI6IG51bGwsICJjb250YWluZXIiOiAiYmFja3VwcyIsICJwYXJlbnRfaWQiOiBudWxsLCAic3RhdHVzIjogImF2YWlsYWJsZSIsICJmYWlsX3JlYXNvbiI6IG51bGwsICJzaXplIjogMSwgImRpc3BsYXlfbmFtZSI6ICJ0ZW1wZXN0LVZvbHVtZXNCYWNrdXBzQWRtaW5UZXN0LUJhY2t1cC05NzE4MjczMjAiLCAiZGlzcGxheV9kZXNjcmlwdGlvbiI6IG51bGwsICJzZXJ2aWNlX21ldGFkYXRhIjogIntcImJhc2VcIjogXCJ2b2x1bWUtYWNiZjJmZGYtN2VmYi00ZGM0LThjZTEtNjQ2MDdiNjQ0MWJmLmJhY2t1cC5lOTNiYzAyMC1kNTY5LTQzMzItOGM2MC01MDNiMTJmNGQ0NzZcIn0iLCAic2VydmljZSI6ICJjaW5kZXIuYmFja3VwLmRyaXZlcnMuY2VwaC5DZXBoQmFja3VwRHJpdmVyIiwgIm9iamVjdF9jb3VudCI6IDAsICJ0ZW1wX3ZvbHVtZV9pZCI6IG51bGwsICJ0ZW1wX3NuYXBzaG90X2lkIjogbnVsbCwgIm51bV9kZXBlbmRlbnRfYmFja3VwcyI6IDAsICJzbmFwc2hvdF9pZCI6IG51bGwsICJkYXRhX3RpbWVzdGFtcCI6ICIyMDIyLTAzLTEyVDIxOjA5OjI3WiIsICJyZXN0b3JlX3ZvbHVtZV9pZCI6IG51bGwsICJtZXRhZGF0YSI6IHt9LCAiZW5jcnlwdGlvbl9rZXlfaWQiOiBudWxsLCAiY3JlYXRlZF9hdCI6ICIyMDIyLTAzLTEyVDIxOjA5OjI3WiIsICJ1cGRhdGVkX2F0IjogIjIwMjItMDMtMTJUMjE6MDk6MzNaIiwgImRlbGV0ZWRfYXQiOiBudWxsLCAiZGVsZXRlZCI6IGZhbHNlfQ==. Mar 12 21:09:34.082762 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_db.sqlalchemy.engines [None req-937422f7-ce97-44a3-9145-2694d697ca79 tempest-VolumesBackupsAdminTest-1519964800 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129443) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 12 21:09:34.181567 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-937422f7-ce97-44a3-9145-2694d697ca79 tempest-VolumesBackupsAdminTest-1519964800 None] Import record id 07966a26-5b63-48bd-8162-826a8ce2e3ef metadata from driver finished. Mar 12 21:09:34.818926 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:09:34.819474 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:09:36.995364 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] Restore backup started, backup: e93bc020-d569-4332-8c60-503b12f4d476 volume: ec316201-71b8-4318-b927-930f7fb04d93. Mar 12 21:09:37.033963 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.utils [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:09:37.049828 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:09:37.084044 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.032s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:09:37.084544 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[cc30f3aa-29cc-4613-9b8e-5f6fde106279]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:77472599036\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:09:37.089944 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:09:37.109275 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 12 21:09:37.109965 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[8227a962-ea8f-41e1-a227-867092288ae8]: [Errno 2] No such file or directory: 'systool' {{(pid=157799) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 12 21:09:37.109965 ncdv-master-1 cinder-backup[129443]: Traceback (most recent call last): Mar 12 21:09:37.109965 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 12 21:09:37.109965 ncdv-master-1 cinder-backup[129443]: ret = func(*f_args, **f_kwargs) Mar 12 21:09:37.109965 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 12 21:09:37.109965 ncdv-master-1 cinder-backup[129443]: return func(*args, **kwargs) Mar 12 21:09:37.109965 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 12 21:09:37.109965 ncdv-master-1 cinder-backup[129443]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 12 21:09:37.109965 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 12 21:09:37.109965 ncdv-master-1 cinder-backup[129443]: return putils.execute(on_execute=on_execute, Mar 12 21:09:37.109965 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 12 21:09:37.109965 ncdv-master-1 cinder-backup[129443]: obj = subprocess.Popen(cmd, Mar 12 21:09:37.109965 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 12 21:09:37.109965 ncdv-master-1 cinder-backup[129443]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 12 21:09:37.109965 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 12 21:09:37.109965 ncdv-master-1 cinder-backup[129443]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 12 21:09:37.109965 ncdv-master-1 cinder-backup[129443]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 12 21:09:37.111952 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[8227a962-ea8f-41e1-a227-867092288ae8]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:09:37.114178 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:09:37.126373 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 12 21:09:37.128389 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[309679a7-8aa9-4a8b-9407-39a990fb207b]: [Errno 2] No such file or directory: 'systool' {{(pid=157799) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 12 21:09:37.128389 ncdv-master-1 cinder-backup[129443]: Traceback (most recent call last): Mar 12 21:09:37.128389 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 12 21:09:37.128389 ncdv-master-1 cinder-backup[129443]: ret = func(*f_args, **f_kwargs) Mar 12 21:09:37.128389 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 12 21:09:37.128389 ncdv-master-1 cinder-backup[129443]: return func(*args, **kwargs) Mar 12 21:09:37.128389 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 12 21:09:37.128389 ncdv-master-1 cinder-backup[129443]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 12 21:09:37.128389 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 12 21:09:37.128389 ncdv-master-1 cinder-backup[129443]: return putils.execute(on_execute=on_execute, Mar 12 21:09:37.128389 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 12 21:09:37.128389 ncdv-master-1 cinder-backup[129443]: obj = subprocess.Popen(cmd, Mar 12 21:09:37.128389 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 12 21:09:37.128389 ncdv-master-1 cinder-backup[129443]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 12 21:09:37.128389 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 12 21:09:37.128389 ncdv-master-1 cinder-backup[129443]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 12 21:09:37.128389 ncdv-master-1 cinder-backup[129443]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 12 21:09:37.130468 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[309679a7-8aa9-4a8b-9407-39a990fb207b]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:09:37.134316 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:09:37.159743 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.027s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:09:37.160670 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[b2e52c0a-0c8b-4fce-adbe-bdc4218a607f]: (4, ('/dev/sda1\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:09:37.165826 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:09:37.199901 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.037s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:09:37.200658 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[6b435ccf-b6d1-4bae-a10e-6034b22acdae]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:09:37.202065 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:09:37.269924 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.065s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:09:37.270530 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[be4455d1-2409-4a33-af3b-93d7dac38a79]: (4, ('55f4b1f4-b5b7-af4f-9adb-821e08822a61\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:09:37.271444 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] Running cmd (subprocess): nvme version {{(pid=129443) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:09:37.358659 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] CMD "nvme version" returned: 0 in 0.087s {{(pid=129443) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:09:37.366867 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129443) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 12 21:09:37.367330 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129443) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 12 21:09:37.367699 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:d9949f82-06c8-4f47-8c57-a55eecab76d2 dsc: {{(pid=129443) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 12 21:09:37.370480 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.utils [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] <== get_connector_properties: return (334ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:77472599036', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '55f4b1f4-b5b7-af4f-9adb-821e08822a61', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:d9949f82-06c8-4f47-8c57-a55eecab76d2', 'found_dsc': ''} {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:09:39.263245 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connector [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] Factory for rbd on None {{(pid=129443) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 12 21:09:39.268616 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-ec316201-71b8-4318-b927-930f7fb04d93', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'ec316201-71b8-4318-b927-930f7fb04d93', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:09:39.270916 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.linuxrbd [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] opening connection to ceph cluster (timeout=-1). {{(pid=129443) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 12 21:09:39.387230 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] <== connect_volume: return (118ms) {'path': } {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:09:39.398230 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_db.sqlalchemy.engines [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129443) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 12 21:09:39.424501 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] Starting restore from Ceph backup=e93bc020-d569-4332-8c60-503b12f4d476 to volume=volume-ec316201-71b8-4318-b927-930f7fb04d93 {{(pid=129443) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1274}} Mar 12 21:09:39.593130 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] Looking for snapshot of backup base 'volume-acbf2fdf-7efb-4dc4-8ce1-64607b6441bf.backup.e93bc020-d569-4332-8c60-503b12f4d476' {{(pid=129443) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 12 21:09:39.594178 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] Found snapshot 'backup.e93bc020-d569-4332-8c60-503b12f4d476.snap.1647119370.194113' {{(pid=129443) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Mar 12 21:09:39.605712 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] Volume file is RBD. {{(pid=129443) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1184}} Mar 12 21:09:39.680115 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] Attempting differential restore. {{(pid=129443) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1234}} Mar 12 21:09:39.680608 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] Attempting incremental restore from base='volume-acbf2fdf-7efb-4dc4-8ce1-64607b6441bf.backup.e93bc020-d569-4332-8c60-503b12f4d476' snap='backup.e93bc020-d569-4332-8c60-503b12f4d476.snap.1647119370.194113' {{(pid=129443) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1089}} Mar 12 21:09:39.681008 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] Performing differential transfer from 'volume-acbf2fdf-7efb-4dc4-8ce1-64607b6441bf.backup.e93bc020-d569-4332-8c60-503b12f4d476' to 'volume-ec316201-71b8-4318-b927-930f7fb04d93' {{(pid=129443) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Mar 12 21:09:39.681402 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-acbf2fdf-7efb-4dc4-8ce1-64607b6441bf.backup.e93bc020-d569-4332-8c60-503b12f4d476@backup.e93bc020-d569-4332-8c60-503b12f4d476.snap.1647119370.194113 -' into... {{(pid=129443) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Mar 12 21:09:39.681783 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_j45a8mv2 --pool volumes - volumes/volume-ec316201-71b8-4318-b927-930f7fb04d93' {{(pid=129443) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Mar 12 21:09:41.023050 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] Restore transfer completed in 1.3414s {{(pid=129443) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1111}} Mar 12 21:09:41.024898 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=129443) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1285}} Mar 12 21:09:41.108661 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] Restoring volume base metadata {{(pid=129443) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} Mar 12 21:09:41.150303 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] No metadata of type 'volume-metadata' to restore {{(pid=129443) put /opt/stack/cinder/cinder/backup/driver.py:345}} Mar 12 21:09:41.151068 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] No metadata of type 'volume-glance-metadata' to restore {{(pid=129443) put /opt/stack/cinder/cinder/backup/driver.py:345}} Mar 12 21:09:41.155000 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] Restore to volume ec316201-71b8-4318-b927-930f7fb04d93 finished successfully. {{(pid=129443) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1292}} Mar 12 21:09:41.156705 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-ec316201-71b8-4318-b927-930f7fb04d93', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'ec316201-71b8-4318-b927-930f7fb04d93', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:09:41.177022 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] <== disconnect_volume: return (19ms) None {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:09:41.284395 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.manager [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] Encryption key ID for volume ec316201-71b8-4318-b927-930f7fb04d93 already matches encryption key ID in backup e93bc020-d569-4332-8c60-503b12f4d476. {{(pid=129443) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} Mar 12 21:09:41.353625 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-ad0d5dcd-5d43-491e-91d8-1b842a91d82f tempest-VolumesBackupsAdminTest-1513847769 None] Finished restoring backup e93bc020-d569-4332-8c60-503b12f4d476 to volume ec316201-71b8-4318-b927-930f7fb04d93. Mar 12 21:09:42.613801 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-13c92f38-35c6-415b-8170-f9a3a3e09721 tempest-VolumesBackupsAdminTest-1519964800 None] Delete backup started, backup: 07966a26-5b63-48bd-8162-826a8ce2e3ef. Mar 12 21:09:42.615814 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-13c92f38-35c6-415b-8170-f9a3a3e09721 tempest-VolumesBackupsAdminTest-1519964800 None] Delete started for backup=07966a26-5b63-48bd-8162-826a8ce2e3ef {{(pid=129443) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Mar 12 21:09:42.616330 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-13c92f38-35c6-415b-8170-f9a3a3e09721 tempest-VolumesBackupsAdminTest-1519964800 None] Trying diff format basename='volume-acbf2fdf-7efb-4dc4-8ce1-64607b6441bf.backup.e93bc020-d569-4332-8c60-503b12f4d476' for backup base image of volume acbf2fdf-7efb-4dc4-8ce1-64607b6441bf. {{(pid=129443) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Mar 12 21:09:42.709664 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-13c92f38-35c6-415b-8170-f9a3a3e09721 tempest-VolumesBackupsAdminTest-1519964800 None] Looking for snapshot of backup base 'volume-acbf2fdf-7efb-4dc4-8ce1-64607b6441bf.backup.e93bc020-d569-4332-8c60-503b12f4d476' {{(pid=129443) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 12 21:09:42.710312 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-13c92f38-35c6-415b-8170-f9a3a3e09721 tempest-VolumesBackupsAdminTest-1519964800 None] Backup '07966a26-5b63-48bd-8162-826a8ce2e3ef' has no snapshot {{(pid=129443) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:915}} Mar 12 21:09:42.711447 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-13c92f38-35c6-415b-8170-f9a3a3e09721 tempest-VolumesBackupsAdminTest-1519964800 None] No backup snapshot to delete {{(pid=129443) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:476}} Mar 12 21:09:42.730685 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.drivers.ceph [None req-13c92f38-35c6-415b-8170-f9a3a3e09721 tempest-VolumesBackupsAdminTest-1519964800 None] Backup base image of volume acbf2fdf-7efb-4dc4-8ce1-64607b6441bf still has 1 snapshots so skipping base image delete. Mar 12 21:09:42.818128 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-13c92f38-35c6-415b-8170-f9a3a3e09721 tempest-VolumesBackupsAdminTest-1519964800 None] Metadata backup object 'backup.07966a26-5b63-48bd-8162-826a8ce2e3ef.meta' not found - ignoring {{(pid=129443) remove_if_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:160}} Mar 12 21:09:42.822637 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-13c92f38-35c6-415b-8170-f9a3a3e09721 tempest-VolumesBackupsAdminTest-1519964800 None] Delete of backup '07966a26-5b63-48bd-8162-826a8ce2e3ef' for volume 'acbf2fdf-7efb-4dc4-8ce1-64607b6441bf' finished. {{(pid=129443) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Mar 12 21:09:42.842156 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_db.sqlalchemy.engines [None req-13c92f38-35c6-415b-8170-f9a3a3e09721 tempest-VolumesBackupsAdminTest-1519964800 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129443) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 12 21:09:42.893569 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.quota [None req-13c92f38-35c6-415b-8170-f9a3a3e09721 tempest-VolumesBackupsAdminTest-1519964800 None] Created reservations ['e2a04e8a-f3f6-4542-9b2f-81d7503d074b', '88a307f0-cee4-415b-8df4-ef1bfc0dbd2e'] {{(pid=129443) reserve /opt/stack/cinder/cinder/quota.py:755}} Mar 12 21:09:42.971609 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-13c92f38-35c6-415b-8170-f9a3a3e09721 tempest-VolumesBackupsAdminTest-1519964800 None] Delete backup finished, backup 07966a26-5b63-48bd-8162-826a8ce2e3ef deleted. Mar 12 21:09:43.881990 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-a028308c-54ec-47a2-b757-e0a733b8c00b tempest-VolumesBackupsAdminTest-1513847769 None] Delete backup started, backup: e93bc020-d569-4332-8c60-503b12f4d476. Mar 12 21:09:43.884591 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-a028308c-54ec-47a2-b757-e0a733b8c00b tempest-VolumesBackupsAdminTest-1513847769 None] Delete started for backup=e93bc020-d569-4332-8c60-503b12f4d476 {{(pid=129443) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Mar 12 21:09:43.885262 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-a028308c-54ec-47a2-b757-e0a733b8c00b tempest-VolumesBackupsAdminTest-1513847769 None] Trying diff format basename='volume-acbf2fdf-7efb-4dc4-8ce1-64607b6441bf.backup.e93bc020-d569-4332-8c60-503b12f4d476' for backup base image of volume acbf2fdf-7efb-4dc4-8ce1-64607b6441bf. {{(pid=129443) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Mar 12 21:09:44.022791 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-a028308c-54ec-47a2-b757-e0a733b8c00b tempest-VolumesBackupsAdminTest-1513847769 None] Looking for snapshot of backup base 'volume-acbf2fdf-7efb-4dc4-8ce1-64607b6441bf.backup.e93bc020-d569-4332-8c60-503b12f4d476' {{(pid=129443) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 12 21:09:44.023297 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-a028308c-54ec-47a2-b757-e0a733b8c00b tempest-VolumesBackupsAdminTest-1513847769 None] Found snapshot 'backup.e93bc020-d569-4332-8c60-503b12f4d476.snap.1647119370.194113' {{(pid=129443) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Mar 12 21:09:44.023829 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-a028308c-54ec-47a2-b757-e0a733b8c00b tempest-VolumesBackupsAdminTest-1513847769 None] Deleting backup snapshot='backup.e93bc020-d569-4332-8c60-503b12f4d476.snap.1647119370.194113' {{(pid=129443) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Mar 12 21:09:44.931194 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.drivers.ceph [None req-a028308c-54ec-47a2-b757-e0a733b8c00b tempest-VolumesBackupsAdminTest-1513847769 None] Deleting backup base image='volume-acbf2fdf-7efb-4dc4-8ce1-64607b6441bf.backup.e93bc020-d569-4332-8c60-503b12f4d476' of volume acbf2fdf-7efb-4dc4-8ce1-64607b6441bf. Mar 12 21:09:45.385818 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-a028308c-54ec-47a2-b757-e0a733b8c00b tempest-VolumesBackupsAdminTest-1513847769 None] Base backup image='volume-acbf2fdf-7efb-4dc4-8ce1-64607b6441bf.backup.e93bc020-d569-4332-8c60-503b12f4d476' of volume acbf2fdf-7efb-4dc4-8ce1-64607b6441bf deleted. {{(pid=129443) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Mar 12 21:09:45.496105 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-a028308c-54ec-47a2-b757-e0a733b8c00b tempest-VolumesBackupsAdminTest-1513847769 None] Delete of backup 'e93bc020-d569-4332-8c60-503b12f4d476' for volume 'acbf2fdf-7efb-4dc4-8ce1-64607b6441bf' finished. {{(pid=129443) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Mar 12 21:09:45.503084 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_db.sqlalchemy.engines [None req-a028308c-54ec-47a2-b757-e0a733b8c00b tempest-VolumesBackupsAdminTest-1513847769 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129443) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 12 21:09:45.541223 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.quota [None req-a028308c-54ec-47a2-b757-e0a733b8c00b tempest-VolumesBackupsAdminTest-1513847769 None] Created reservations ['41019d43-0674-48ed-bebe-6c1a64cd5406', '2af3f21e-d5fe-4c89-bcec-75cc1694977f'] {{(pid=129443) reserve /opt/stack/cinder/cinder/quota.py:755}} Mar 12 21:09:45.620902 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-a028308c-54ec-47a2-b757-e0a733b8c00b tempest-VolumesBackupsAdminTest-1513847769 None] Delete backup finished, backup e93bc020-d569-4332-8c60-503b12f4d476 deleted. Mar 12 21:09:48.098857 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] Create backup started, backup: e48f9fbe-d6aa-4178-bee3-d082b456689b volume: b763224b-f5d9-41ae-9177-904edf177d28. Mar 12 21:09:48.125453 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-03-12T21:09:47Z,data_timestamp=2022-03-12T21:09:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsAdminTest-Backup-265713057',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=e48f9fbe-d6aa-4178-bee3-d082b456689b,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='76614dd1780d41e99b3db646e59d928c',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='3d36ef1eff0649179221db760b119396',volume_id=b763224b-f5d9-41ae-9177-904edf177d28) Mar 12 21:09:48.231418 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.utils [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:09:48.238490 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:09:48.262626 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.029s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:09:48.263218 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[2b2e8738-1af1-4a5f-b74e-2f78fdd9ff0e]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:77472599036\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:09:48.270432 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:09:48.289965 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 12 21:09:48.290489 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[e11e3036-1080-46cc-8e9c-86bf1b09f0ef]: [Errno 2] No such file or directory: 'systool' {{(pid=157799) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 12 21:09:48.290489 ncdv-master-1 cinder-backup[129443]: Traceback (most recent call last): Mar 12 21:09:48.290489 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 12 21:09:48.290489 ncdv-master-1 cinder-backup[129443]: ret = func(*f_args, **f_kwargs) Mar 12 21:09:48.290489 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 12 21:09:48.290489 ncdv-master-1 cinder-backup[129443]: return func(*args, **kwargs) Mar 12 21:09:48.290489 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 12 21:09:48.290489 ncdv-master-1 cinder-backup[129443]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 12 21:09:48.290489 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 12 21:09:48.290489 ncdv-master-1 cinder-backup[129443]: return putils.execute(on_execute=on_execute, Mar 12 21:09:48.290489 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 12 21:09:48.290489 ncdv-master-1 cinder-backup[129443]: obj = subprocess.Popen(cmd, Mar 12 21:09:48.290489 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 12 21:09:48.290489 ncdv-master-1 cinder-backup[129443]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 12 21:09:48.290489 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 12 21:09:48.290489 ncdv-master-1 cinder-backup[129443]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 12 21:09:48.290489 ncdv-master-1 cinder-backup[129443]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 12 21:09:48.291721 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[e11e3036-1080-46cc-8e9c-86bf1b09f0ef]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:09:48.293652 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:09:48.301319 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 12 21:09:48.301956 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[f9c694dd-819f-46db-8fe1-8f382434025c]: [Errno 2] No such file or directory: 'systool' {{(pid=157799) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 12 21:09:48.301956 ncdv-master-1 cinder-backup[129443]: Traceback (most recent call last): Mar 12 21:09:48.301956 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 12 21:09:48.301956 ncdv-master-1 cinder-backup[129443]: ret = func(*f_args, **f_kwargs) Mar 12 21:09:48.301956 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 12 21:09:48.301956 ncdv-master-1 cinder-backup[129443]: return func(*args, **kwargs) Mar 12 21:09:48.301956 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 12 21:09:48.301956 ncdv-master-1 cinder-backup[129443]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 12 21:09:48.301956 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 12 21:09:48.301956 ncdv-master-1 cinder-backup[129443]: return putils.execute(on_execute=on_execute, Mar 12 21:09:48.301956 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 12 21:09:48.301956 ncdv-master-1 cinder-backup[129443]: obj = subprocess.Popen(cmd, Mar 12 21:09:48.301956 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 12 21:09:48.301956 ncdv-master-1 cinder-backup[129443]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 12 21:09:48.301956 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 12 21:09:48.301956 ncdv-master-1 cinder-backup[129443]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 12 21:09:48.301956 ncdv-master-1 cinder-backup[129443]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 12 21:09:48.303774 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[f9c694dd-819f-46db-8fe1-8f382434025c]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:09:48.309604 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:09:48.334244 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.023s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:09:48.335110 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[cf3da2fc-e0b9-47eb-9e62-31c643db8483]: (4, ('/dev/sda1\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:09:48.338590 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:09:48.371228 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.034s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:09:48.372092 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[3a947852-2163-4397-aa03-a3af72b76189]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:09:48.373523 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:09:48.416511 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.042s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:09:48.416900 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[f6a9c850-695e-40f4-9253-674b3e41bfe9]: (4, ('55f4b1f4-b5b7-af4f-9adb-821e08822a61\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:09:48.417913 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] Running cmd (subprocess): nvme version {{(pid=129443) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:09:48.475666 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] CMD "nvme version" returned: 0 in 0.057s {{(pid=129443) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:09:48.478770 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129443) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 12 21:09:48.479426 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129443) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 12 21:09:48.479901 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:d9949f82-06c8-4f47-8c57-a55eecab76d2 dsc: {{(pid=129443) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 12 21:09:48.481601 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.utils [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] <== get_connector_properties: return (248ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:77472599036', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '55f4b1f4-b5b7-af4f-9adb-821e08822a61', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:d9949f82-06c8-4f47-8c57-a55eecab76d2', 'found_dsc': ''} {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:09:50.448237 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connector [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] Factory for rbd on None {{(pid=129443) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 12 21:09:50.449852 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-b763224b-f5d9-41ae-9177-904edf177d28', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'b763224b-f5d9-41ae-9177-904edf177d28', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:09:50.451471 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.linuxrbd [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] opening connection to ceph cluster (timeout=-1). {{(pid=129443) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 12 21:09:50.551781 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] <== connect_volume: return (100ms) {'path': } {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:09:50.559522 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_db.sqlalchemy.engines [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129443) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 12 21:09:50.599508 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] Starting backup of volume='b763224b-f5d9-41ae-9177-904edf177d28'. {{(pid=129443) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Mar 12 21:09:50.601264 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] Volume file is RBD: attempting optimized backup {{(pid=129443) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Mar 12 21:09:50.677387 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] Creating base image 'volume-b763224b-f5d9-41ae-9177-904edf177d28.backup.e48f9fbe-d6aa-4178-bee3-d082b456689b' {{(pid=129443) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Mar 12 21:09:50.677909 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129443) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Mar 12 21:09:50.770409 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] Using --from-snap 'None' for incremental backup of volume b763224b-f5d9-41ae-9177-904edf177d28. {{(pid=129443) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Mar 12 21:09:50.771004 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] Creating backup snapshot='backup.e48f9fbe-d6aa-4178-bee3-d082b456689b.snap.1647119390.7706165' {{(pid=129443) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Mar 12 21:09:51.350126 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] Performing differential transfer from 'volume-b763224b-f5d9-41ae-9177-904edf177d28' to 'volume-b763224b-f5d9-41ae-9177-904edf177d28.backup.e48f9fbe-d6aa-4178-bee3-d082b456689b' {{(pid=129443) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Mar 12 21:09:51.350894 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_v440mvb2 --pool volumes volumes/volume-b763224b-f5d9-41ae-9177-904edf177d28@backup.e48f9fbe-d6aa-4178-bee3-d082b456689b.snap.1647119390.7706165 -' into... {{(pid=129443) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Mar 12 21:09:51.351431 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-b763224b-f5d9-41ae-9177-904edf177d28.backup.e48f9fbe-d6aa-4178-bee3-d082b456689b' {{(pid=129443) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Mar 12 21:09:52.434802 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] Differential backup transfer completed in 1.0840s {{(pid=129443) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Mar 12 21:09:52.435489 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] Getting metadata type 'volume-base-metadata' {{(pid=129443) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Mar 12 21:09:52.464189 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129443) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Mar 12 21:09:52.469255 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] Getting metadata type 'volume-metadata' {{(pid=129443) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Mar 12 21:09:52.500805 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] No metadata type 'volume-metadata' available {{(pid=129443) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Mar 12 21:09:52.501199 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] Getting metadata type 'volume-glance-metadata' {{(pid=129443) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Mar 12 21:09:52.521937 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] No metadata type 'volume-glance-metadata' available {{(pid=129443) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Mar 12 21:09:52.523367 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] Backing up metadata for volume b763224b-f5d9-41ae-9177-904edf177d28. {{(pid=129443) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Mar 12 21:09:52.631398 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] Backup 'e48f9fbe-d6aa-4178-bee3-d082b456689b' of volume b763224b-f5d9-41ae-9177-904edf177d28 finished. {{(pid=129443) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Mar 12 21:09:52.632963 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-b763224b-f5d9-41ae-9177-904edf177d28', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'b763224b-f5d9-41ae-9177-904edf177d28', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:09:52.682305 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] <== disconnect_volume: return (48ms) None {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:09:52.844111 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-891473e1-5cd7-4ea0-99da-919caa69591c tempest-VolumesBackupsAdminTest-1513847769 None] Create backup finished. backup: e48f9fbe-d6aa-4178-bee3-d082b456689b. Mar 12 21:09:53.547004 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-7736b00c-1ad7-4627-accb-2f46b8d23727 tempest-VolumesBackupsAdminTest-1519964800 None] Reset backup status started, backup_id: e48f9fbe-d6aa-4178-bee3-d082b456689b, status: error. Mar 12 21:09:53.547583 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-7736b00c-1ad7-4627-accb-2f46b8d23727 tempest-VolumesBackupsAdminTest-1519964800 None] Backup service: cinder.backup.drivers.ceph.CephBackupDriver. Mar 12 21:09:54.733993 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-c94446a9-1e28-4408-88cf-27d0a59912a3 tempest-VolumesBackupsAdminTest-1513847769 None] Delete backup started, backup: e48f9fbe-d6aa-4178-bee3-d082b456689b. Mar 12 21:09:54.735762 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-c94446a9-1e28-4408-88cf-27d0a59912a3 tempest-VolumesBackupsAdminTest-1513847769 None] Delete started for backup=e48f9fbe-d6aa-4178-bee3-d082b456689b {{(pid=129443) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Mar 12 21:09:54.736078 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-c94446a9-1e28-4408-88cf-27d0a59912a3 tempest-VolumesBackupsAdminTest-1513847769 None] Trying diff format basename='volume-b763224b-f5d9-41ae-9177-904edf177d28.backup.e48f9fbe-d6aa-4178-bee3-d082b456689b' for backup base image of volume b763224b-f5d9-41ae-9177-904edf177d28. {{(pid=129443) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Mar 12 21:09:54.840336 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-c94446a9-1e28-4408-88cf-27d0a59912a3 tempest-VolumesBackupsAdminTest-1513847769 None] Looking for snapshot of backup base 'volume-b763224b-f5d9-41ae-9177-904edf177d28.backup.e48f9fbe-d6aa-4178-bee3-d082b456689b' {{(pid=129443) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 12 21:09:54.841073 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-c94446a9-1e28-4408-88cf-27d0a59912a3 tempest-VolumesBackupsAdminTest-1513847769 None] Found snapshot 'backup.e48f9fbe-d6aa-4178-bee3-d082b456689b.snap.1647119390.7706165' {{(pid=129443) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Mar 12 21:09:54.841687 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-c94446a9-1e28-4408-88cf-27d0a59912a3 tempest-VolumesBackupsAdminTest-1513847769 None] Deleting backup snapshot='backup.e48f9fbe-d6aa-4178-bee3-d082b456689b.snap.1647119390.7706165' {{(pid=129443) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Mar 12 21:09:55.580835 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.drivers.ceph [None req-c94446a9-1e28-4408-88cf-27d0a59912a3 tempest-VolumesBackupsAdminTest-1513847769 None] Deleting backup base image='volume-b763224b-f5d9-41ae-9177-904edf177d28.backup.e48f9fbe-d6aa-4178-bee3-d082b456689b' of volume b763224b-f5d9-41ae-9177-904edf177d28. Mar 12 21:09:56.118553 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-c94446a9-1e28-4408-88cf-27d0a59912a3 tempest-VolumesBackupsAdminTest-1513847769 None] Base backup image='volume-b763224b-f5d9-41ae-9177-904edf177d28.backup.e48f9fbe-d6aa-4178-bee3-d082b456689b' of volume b763224b-f5d9-41ae-9177-904edf177d28 deleted. {{(pid=129443) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Mar 12 21:09:56.214080 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-c94446a9-1e28-4408-88cf-27d0a59912a3 tempest-VolumesBackupsAdminTest-1513847769 None] Delete of backup 'e48f9fbe-d6aa-4178-bee3-d082b456689b' for volume 'b763224b-f5d9-41ae-9177-904edf177d28' finished. {{(pid=129443) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Mar 12 21:09:56.222514 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_db.sqlalchemy.engines [None req-c94446a9-1e28-4408-88cf-27d0a59912a3 tempest-VolumesBackupsAdminTest-1513847769 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129443) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 12 21:09:56.296016 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.quota [None req-c94446a9-1e28-4408-88cf-27d0a59912a3 tempest-VolumesBackupsAdminTest-1513847769 None] Created reservations ['86267837-54f9-4b99-a428-dd94d0e7eaff', 'ded9a34a-b054-4777-8e9c-b31243270d0c'] {{(pid=129443) reserve /opt/stack/cinder/cinder/quota.py:755}} Mar 12 21:09:56.349334 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-c94446a9-1e28-4408-88cf-27d0a59912a3 tempest-VolumesBackupsAdminTest-1513847769 None] Delete backup finished, backup e48f9fbe-d6aa-4178-bee3-d082b456689b deleted. Mar 12 21:10:34.815632 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:10:34.816330 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:11:01.216199 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] Create backup started, backup: 2f1b3292-256a-4bcd-82e8-5d941cc56f39 volume: 7aa69d6a-de21-48b0-a279-c3e51e53b549 snapshot: c2e42604-fc83-4591-a5a0-fc6e27633e22. Mar 12 21:11:01.229819 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-03-12T21:11:01Z,data_timestamp=2022-03-12T21:11:00Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Backup-1621937425',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=2f1b3292-256a-4bcd-82e8-5d941cc56f39,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='d30a0195c6334b39a246a88b1f4cc81f',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id='c2e42604-fc83-4591-a5a0-fc6e27633e22',status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='96d0c07fcae34c48b4c0ed61c4cec1c2',volume_id=7aa69d6a-de21-48b0-a279-c3e51e53b549) Mar 12 21:11:01.862773 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.utils [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:11:01.864349 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:11:01.887289 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.021s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:11:01.888685 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[30467770-581c-45e7-8d1a-e2b2180ae508]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:77472599036\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:11:01.890757 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:11:01.903904 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 12 21:11:01.904215 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[6652ce9b-c363-477e-baef-10f5e013a87d]: [Errno 2] No such file or directory: 'systool' {{(pid=157799) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 12 21:11:01.904215 ncdv-master-1 cinder-backup[129443]: Traceback (most recent call last): Mar 12 21:11:01.904215 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 12 21:11:01.904215 ncdv-master-1 cinder-backup[129443]: ret = func(*f_args, **f_kwargs) Mar 12 21:11:01.904215 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 12 21:11:01.904215 ncdv-master-1 cinder-backup[129443]: return func(*args, **kwargs) Mar 12 21:11:01.904215 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 12 21:11:01.904215 ncdv-master-1 cinder-backup[129443]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 12 21:11:01.904215 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 12 21:11:01.904215 ncdv-master-1 cinder-backup[129443]: return putils.execute(on_execute=on_execute, Mar 12 21:11:01.904215 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 12 21:11:01.904215 ncdv-master-1 cinder-backup[129443]: obj = subprocess.Popen(cmd, Mar 12 21:11:01.904215 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 12 21:11:01.904215 ncdv-master-1 cinder-backup[129443]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 12 21:11:01.904215 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 12 21:11:01.904215 ncdv-master-1 cinder-backup[129443]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 12 21:11:01.904215 ncdv-master-1 cinder-backup[129443]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 12 21:11:01.906224 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[6652ce9b-c363-477e-baef-10f5e013a87d]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:11:01.908095 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:11:01.916919 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 12 21:11:01.918046 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[b61bf59a-9479-4960-ab26-aeab92c6991c]: [Errno 2] No such file or directory: 'systool' {{(pid=157799) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 12 21:11:01.918046 ncdv-master-1 cinder-backup[129443]: Traceback (most recent call last): Mar 12 21:11:01.918046 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 12 21:11:01.918046 ncdv-master-1 cinder-backup[129443]: ret = func(*f_args, **f_kwargs) Mar 12 21:11:01.918046 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 12 21:11:01.918046 ncdv-master-1 cinder-backup[129443]: return func(*args, **kwargs) Mar 12 21:11:01.918046 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 12 21:11:01.918046 ncdv-master-1 cinder-backup[129443]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 12 21:11:01.918046 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 12 21:11:01.918046 ncdv-master-1 cinder-backup[129443]: return putils.execute(on_execute=on_execute, Mar 12 21:11:01.918046 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 12 21:11:01.918046 ncdv-master-1 cinder-backup[129443]: obj = subprocess.Popen(cmd, Mar 12 21:11:01.918046 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 12 21:11:01.918046 ncdv-master-1 cinder-backup[129443]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 12 21:11:01.918046 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 12 21:11:01.918046 ncdv-master-1 cinder-backup[129443]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 12 21:11:01.918046 ncdv-master-1 cinder-backup[129443]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 12 21:11:01.920754 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[b61bf59a-9479-4960-ab26-aeab92c6991c]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:11:01.924917 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:11:01.959425 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.034s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:11:01.959958 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[fa54d158-4323-4e0f-845c-78e90c2ffbc0]: (4, ('/dev/sda1\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:11:01.962082 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:11:01.982606 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.020s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:11:01.982893 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[5a18ab3f-a229-4ba9-aa75-cf69233a4f49]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:11:01.984530 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:11:02.025632 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.040s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:11:02.027252 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[61c50d78-e4a3-4236-bcf8-e40c5a23bc07]: (4, ('55f4b1f4-b5b7-af4f-9adb-821e08822a61\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:11:02.028299 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] Running cmd (subprocess): nvme version {{(pid=129443) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:11:02.059409 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] CMD "nvme version" returned: 0 in 0.031s {{(pid=129443) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:11:02.061218 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129443) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 12 21:11:02.061615 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129443) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 12 21:11:02.062295 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:d9949f82-06c8-4f47-8c57-a55eecab76d2 dsc: {{(pid=129443) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 12 21:11:02.063136 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.utils [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] <== get_connector_properties: return (200ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:77472599036', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '55f4b1f4-b5b7-af4f-9adb-821e08822a61', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:d9949f82-06c8-4f47-8c57-a55eecab76d2', 'found_dsc': ''} {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:11:03.062876 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connector [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] Factory for rbd on None {{(pid=129443) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 12 21:11:03.065055 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-aa8bfaab-68ad-47f3-bc65-a53db22c6bc4', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'aa8bfaab-68ad-47f3-bc65-a53db22c6bc4', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:11:03.066144 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.linuxrbd [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] opening connection to ceph cluster (timeout=-1). {{(pid=129443) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 12 21:11:03.138512 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] <== connect_volume: return (72ms) {'path': } {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:11:03.147954 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_db.sqlalchemy.engines [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129443) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 12 21:11:03.170049 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] Starting backup of volume='7aa69d6a-de21-48b0-a279-c3e51e53b549'. {{(pid=129443) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Mar 12 21:11:03.231148 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129443) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Mar 12 21:11:03.231148 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] Creating backup base image='volume-7aa69d6a-de21-48b0-a279-c3e51e53b549.backup.base' for volume 7aa69d6a-de21-48b0-a279-c3e51e53b549. {{(pid=129443) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:830}} Mar 12 21:11:03.268197 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] Copying data from volume 7aa69d6a-de21-48b0-a279-c3e51e53b549. {{(pid=129443) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:842}} Mar 12 21:11:03.292835 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] Transferring data between 'volume-7aa69d6a-de21-48b0-a279-c3e51e53b549' and 'volume-7aa69d6a-de21-48b0-a279-c3e51e53b549.backup.base' {{(pid=129443) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:399}} Mar 12 21:11:03.293225 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] 8 chunks of 134217728 bytes to be transferred {{(pid=129443) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:403}} Mar 12 21:11:04.881308 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] Transferred chunk 1 of 8 (82566K/s) {{(pid=129443) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Mar 12 21:11:06.455149 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] Transferred chunk 2 of 8 (83363K/s) {{(pid=129443) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Mar 12 21:11:08.090685 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] Transferred chunk 3 of 8 (80346K/s) {{(pid=129443) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Mar 12 21:11:09.715123 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] Transferred chunk 4 of 8 (80717K/s) {{(pid=129443) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Mar 12 21:11:11.279379 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] Transferred chunk 5 of 8 (83914K/s) {{(pid=129443) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Mar 12 21:11:12.748688 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] Transferred chunk 6 of 8 (89243K/s) {{(pid=129443) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Mar 12 21:11:14.193315 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] Transferred chunk 7 of 8 (90816K/s) {{(pid=129443) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Mar 12 21:11:15.518450 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] Transferred chunk 8 of 8 (98963K/s) {{(pid=129443) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Mar 12 21:11:15.605013 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] Getting metadata type 'volume-base-metadata' {{(pid=129443) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Mar 12 21:11:15.621427 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129443) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Mar 12 21:11:15.622162 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] Getting metadata type 'volume-metadata' {{(pid=129443) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Mar 12 21:11:15.630687 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] No metadata type 'volume-metadata' available {{(pid=129443) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Mar 12 21:11:15.631427 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] Getting metadata type 'volume-glance-metadata' {{(pid=129443) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Mar 12 21:11:15.639816 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] No metadata type 'volume-glance-metadata' available {{(pid=129443) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Mar 12 21:11:15.640661 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] Backing up metadata for volume 7aa69d6a-de21-48b0-a279-c3e51e53b549. {{(pid=129443) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Mar 12 21:11:15.686268 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] Backup '2f1b3292-256a-4bcd-82e8-5d941cc56f39' of volume 7aa69d6a-de21-48b0-a279-c3e51e53b549 finished. {{(pid=129443) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Mar 12 21:11:15.687653 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-aa8bfaab-68ad-47f3-bc65-a53db22c6bc4', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'aa8bfaab-68ad-47f3-bc65-a53db22c6bc4', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:11:15.716833 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] <== disconnect_volume: return (28ms) None {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:11:15.884010 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-f676cf1e-f857-4a13-ba94-ace771b1ca5d tempest-VolumesSnapshotTestJSON-1127165440 None] Create backup finished. backup: 2f1b3292-256a-4bcd-82e8-5d941cc56f39. Mar 12 21:11:16.977007 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-2153d400-7f67-40d4-a818-3d0e2e87b37c tempest-VolumesSnapshotTestJSON-1127165440 None] Delete backup started, backup: 2f1b3292-256a-4bcd-82e8-5d941cc56f39. Mar 12 21:11:16.978937 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-2153d400-7f67-40d4-a818-3d0e2e87b37c tempest-VolumesSnapshotTestJSON-1127165440 None] Delete started for backup=2f1b3292-256a-4bcd-82e8-5d941cc56f39 {{(pid=129443) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Mar 12 21:11:16.979438 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-2153d400-7f67-40d4-a818-3d0e2e87b37c tempest-VolumesSnapshotTestJSON-1127165440 None] Trying diff format basename='volume-7aa69d6a-de21-48b0-a279-c3e51e53b549.backup.2f1b3292-256a-4bcd-82e8-5d941cc56f39' for backup base image of volume 7aa69d6a-de21-48b0-a279-c3e51e53b549. {{(pid=129443) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Mar 12 21:11:17.043622 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-2153d400-7f67-40d4-a818-3d0e2e87b37c tempest-VolumesSnapshotTestJSON-1127165440 None] Image 'volume-7aa69d6a-de21-48b0-a279-c3e51e53b549.backup.2f1b3292-256a-4bcd-82e8-5d941cc56f39' not found - trying diff format name {{(pid=129443) _rbd_image_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:661}} Mar 12 21:11:17.066790 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-2153d400-7f67-40d4-a818-3d0e2e87b37c tempest-VolumesSnapshotTestJSON-1127165440 None] Looking for snapshot of backup base 'volume-7aa69d6a-de21-48b0-a279-c3e51e53b549.backup.base' {{(pid=129443) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 12 21:11:17.067355 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-2153d400-7f67-40d4-a818-3d0e2e87b37c tempest-VolumesSnapshotTestJSON-1127165440 None] Backup base 'volume-7aa69d6a-de21-48b0-a279-c3e51e53b549.backup.base' has no snapshots {{(pid=129443) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:908}} Mar 12 21:11:17.067794 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-2153d400-7f67-40d4-a818-3d0e2e87b37c tempest-VolumesSnapshotTestJSON-1127165440 None] No backup snapshot to delete {{(pid=129443) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:476}} Mar 12 21:11:17.075318 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.drivers.ceph [None req-2153d400-7f67-40d4-a818-3d0e2e87b37c tempest-VolumesSnapshotTestJSON-1127165440 None] Deleting backup base image='volume-7aa69d6a-de21-48b0-a279-c3e51e53b549.backup.base' of volume 7aa69d6a-de21-48b0-a279-c3e51e53b549. Mar 12 21:11:17.890527 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-2153d400-7f67-40d4-a818-3d0e2e87b37c tempest-VolumesSnapshotTestJSON-1127165440 None] Base backup image='volume-7aa69d6a-de21-48b0-a279-c3e51e53b549.backup.base' of volume 7aa69d6a-de21-48b0-a279-c3e51e53b549 deleted. {{(pid=129443) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Mar 12 21:11:17.947133 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-2153d400-7f67-40d4-a818-3d0e2e87b37c tempest-VolumesSnapshotTestJSON-1127165440 None] Delete of backup '2f1b3292-256a-4bcd-82e8-5d941cc56f39' for volume '7aa69d6a-de21-48b0-a279-c3e51e53b549' finished. {{(pid=129443) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Mar 12 21:11:17.953502 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_db.sqlalchemy.engines [None req-2153d400-7f67-40d4-a818-3d0e2e87b37c tempest-VolumesSnapshotTestJSON-1127165440 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129443) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 12 21:11:17.983082 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.quota [None req-2153d400-7f67-40d4-a818-3d0e2e87b37c tempest-VolumesSnapshotTestJSON-1127165440 None] Created reservations ['7c1a74a5-b6bc-460b-80b9-348480f37f96', '84e0ac96-9cff-4dd9-9347-891625918577'] {{(pid=129443) reserve /opt/stack/cinder/cinder/quota.py:755}} Mar 12 21:11:18.020258 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-2153d400-7f67-40d4-a818-3d0e2e87b37c tempest-VolumesSnapshotTestJSON-1127165440 None] Delete backup finished, backup 2f1b3292-256a-4bcd-82e8-5d941cc56f39 deleted. Mar 12 21:11:34.813003 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:11:34.813553 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:12:34.810440 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:12:34.811143 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:13:34.811687 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:13:34.813014 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:14:34.808878 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:14:34.808878 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:15:34.814310 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:15:34.814964 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:16:34.814523 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:16:34.814523 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:17:34.819393 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:17:34.819926 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:18:34.825153 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:18:34.825750 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:19:21.113190 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] Create backup started, backup: 6f2a8acc-32b2-451c-98bb-63cbbae2c432 volume: b68846b1-e2e8-4551-9e39-b8fa771ce5ff. Mar 12 21:19:21.122984 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-03-12T21:19:20Z,data_timestamp=2022-03-12T21:19:20Z,deleted=False,deleted_at=None,display_description=None,display_name=None,encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=6f2a8acc-32b2-451c-98bb-63cbbae2c432,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='2e6a0660e45d4aea9ff4576abd1f0e56',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='62b74fe4246c415bbb401849ce000f9b',volume_id=b68846b1-e2e8-4551-9e39-b8fa771ce5ff) Mar 12 21:19:21.347231 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.utils [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:19:21.351368 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:19:21.367901 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.016s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:19:21.368231 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[9423f110-d6b5-4ac1-9afe-6423753713a3]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:77472599036\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:19:21.372574 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:19:21.381973 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 12 21:19:21.383946 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[ef0605eb-e207-435b-b35f-8a5a8751ae46]: [Errno 2] No such file or directory: 'systool' {{(pid=157799) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 12 21:19:21.383946 ncdv-master-1 cinder-backup[129443]: Traceback (most recent call last): Mar 12 21:19:21.383946 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 12 21:19:21.383946 ncdv-master-1 cinder-backup[129443]: ret = func(*f_args, **f_kwargs) Mar 12 21:19:21.383946 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 12 21:19:21.383946 ncdv-master-1 cinder-backup[129443]: return func(*args, **kwargs) Mar 12 21:19:21.383946 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 12 21:19:21.383946 ncdv-master-1 cinder-backup[129443]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 12 21:19:21.383946 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 12 21:19:21.383946 ncdv-master-1 cinder-backup[129443]: return putils.execute(on_execute=on_execute, Mar 12 21:19:21.383946 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 12 21:19:21.383946 ncdv-master-1 cinder-backup[129443]: obj = subprocess.Popen(cmd, Mar 12 21:19:21.383946 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 12 21:19:21.383946 ncdv-master-1 cinder-backup[129443]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 12 21:19:21.383946 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 12 21:19:21.383946 ncdv-master-1 cinder-backup[129443]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 12 21:19:21.383946 ncdv-master-1 cinder-backup[129443]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 12 21:19:21.386240 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[ef0605eb-e207-435b-b35f-8a5a8751ae46]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:19:21.389296 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:19:21.400445 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 12 21:19:21.402025 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[621e3a75-e78c-48e8-bcc1-483dc72c48f6]: [Errno 2] No such file or directory: 'systool' {{(pid=157799) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 12 21:19:21.402025 ncdv-master-1 cinder-backup[129443]: Traceback (most recent call last): Mar 12 21:19:21.402025 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 12 21:19:21.402025 ncdv-master-1 cinder-backup[129443]: ret = func(*f_args, **f_kwargs) Mar 12 21:19:21.402025 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 12 21:19:21.402025 ncdv-master-1 cinder-backup[129443]: return func(*args, **kwargs) Mar 12 21:19:21.402025 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 12 21:19:21.402025 ncdv-master-1 cinder-backup[129443]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 12 21:19:21.402025 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 12 21:19:21.402025 ncdv-master-1 cinder-backup[129443]: return putils.execute(on_execute=on_execute, Mar 12 21:19:21.402025 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 12 21:19:21.402025 ncdv-master-1 cinder-backup[129443]: obj = subprocess.Popen(cmd, Mar 12 21:19:21.402025 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 12 21:19:21.402025 ncdv-master-1 cinder-backup[129443]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 12 21:19:21.402025 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 12 21:19:21.402025 ncdv-master-1 cinder-backup[129443]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 12 21:19:21.402025 ncdv-master-1 cinder-backup[129443]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 12 21:19:21.403043 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[621e3a75-e78c-48e8-bcc1-483dc72c48f6]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:19:21.404328 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:19:21.427645 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.023s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:19:21.429021 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[0eaafc3c-7258-416c-a2d8-6d7651d1e1a6]: (4, ('/dev/sda1\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:19:21.430398 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:19:21.447013 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.015s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:19:21.447339 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[9528ba16-2f2b-4ee4-84a8-ae8ac08fde66]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:19:21.448927 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:19:21.459397 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.010s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:19:21.459813 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[11273ece-afe4-44e0-8ff3-5aa12eb73c70]: (4, ('55f4b1f4-b5b7-af4f-9adb-821e08822a61\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:19:21.460727 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] Running cmd (subprocess): nvme version {{(pid=129443) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:19:21.484087 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] CMD "nvme version" returned: 0 in 0.023s {{(pid=129443) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:19:21.486039 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129443) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 12 21:19:21.486425 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129443) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 12 21:19:21.486832 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:d9949f82-06c8-4f47-8c57-a55eecab76d2 dsc: {{(pid=129443) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 12 21:19:21.487533 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.utils [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] <== get_connector_properties: return (139ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:77472599036', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '55f4b1f4-b5b7-af4f-9adb-821e08822a61', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:d9949f82-06c8-4f47-8c57-a55eecab76d2', 'found_dsc': ''} {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:19:22.272503 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connector [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] Factory for rbd on None {{(pid=129443) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 12 21:19:22.276368 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-b68846b1-e2e8-4551-9e39-b8fa771ce5ff', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'b68846b1-e2e8-4551-9e39-b8fa771ce5ff', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:19:22.278521 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.linuxrbd [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] opening connection to ceph cluster (timeout=-1). {{(pid=129443) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 12 21:19:22.386201 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] <== connect_volume: return (108ms) {'path': } {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:19:22.398652 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_db.sqlalchemy.engines [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129443) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 12 21:19:22.451123 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] Starting backup of volume='b68846b1-e2e8-4551-9e39-b8fa771ce5ff'. {{(pid=129443) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Mar 12 21:19:22.452079 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] Volume file is RBD: attempting optimized backup {{(pid=129443) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Mar 12 21:19:22.499300 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] Creating base image 'volume-b68846b1-e2e8-4551-9e39-b8fa771ce5ff.backup.6f2a8acc-32b2-451c-98bb-63cbbae2c432' {{(pid=129443) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Mar 12 21:19:22.500004 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129443) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Mar 12 21:19:22.582295 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] Using --from-snap 'None' for incremental backup of volume b68846b1-e2e8-4551-9e39-b8fa771ce5ff. {{(pid=129443) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Mar 12 21:19:22.583267 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] Creating backup snapshot='backup.6f2a8acc-32b2-451c-98bb-63cbbae2c432.snap.1647119962.5825202' {{(pid=129443) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Mar 12 21:19:22.798867 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] Performing differential transfer from 'volume-b68846b1-e2e8-4551-9e39-b8fa771ce5ff' to 'volume-b68846b1-e2e8-4551-9e39-b8fa771ce5ff.backup.6f2a8acc-32b2-451c-98bb-63cbbae2c432' {{(pid=129443) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Mar 12 21:19:22.799442 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_50d0052j --pool volumes volumes/volume-b68846b1-e2e8-4551-9e39-b8fa771ce5ff@backup.6f2a8acc-32b2-451c-98bb-63cbbae2c432.snap.1647119962.5825202 -' into... {{(pid=129443) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Mar 12 21:19:22.800107 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-b68846b1-e2e8-4551-9e39-b8fa771ce5ff.backup.6f2a8acc-32b2-451c-98bb-63cbbae2c432' {{(pid=129443) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Mar 12 21:19:29.597610 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] Differential backup transfer completed in 6.7985s {{(pid=129443) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Mar 12 21:19:29.598460 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] Getting metadata type 'volume-base-metadata' {{(pid=129443) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Mar 12 21:19:29.613600 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129443) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Mar 12 21:19:29.614057 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] Getting metadata type 'volume-metadata' {{(pid=129443) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Mar 12 21:19:29.621431 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] No metadata type 'volume-metadata' available {{(pid=129443) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Mar 12 21:19:29.621849 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] Getting metadata type 'volume-glance-metadata' {{(pid=129443) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Mar 12 21:19:29.629249 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=129443) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}} Mar 12 21:19:29.629858 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] Backing up metadata for volume b68846b1-e2e8-4551-9e39-b8fa771ce5ff. {{(pid=129443) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Mar 12 21:19:29.678415 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] Backup '6f2a8acc-32b2-451c-98bb-63cbbae2c432' of volume b68846b1-e2e8-4551-9e39-b8fa771ce5ff finished. {{(pid=129443) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Mar 12 21:19:29.681566 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-b68846b1-e2e8-4551-9e39-b8fa771ce5ff', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'b68846b1-e2e8-4551-9e39-b8fa771ce5ff', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:19:29.718460 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] <== disconnect_volume: return (35ms) None {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:19:29.781799 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-631cfd3f-790f-408d-a264-d886ab852c96 tempest-TestVolumeBackupRestore-1490532501 None] Create backup finished. backup: 6f2a8acc-32b2-451c-98bb-63cbbae2c432. Mar 12 21:19:31.782866 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] Restore backup started, backup: 6f2a8acc-32b2-451c-98bb-63cbbae2c432 volume: e5912a2c-aeb4-40f4-a05e-781c6b342f11. Mar 12 21:19:31.795127 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.utils [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:19:31.797068 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:19:31.806978 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.010s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:19:31.807314 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[4fe421fe-305f-47f3-b284-22d98cf700cb]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:77472599036\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:19:31.808559 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:19:31.814734 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 12 21:19:31.815719 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[32cb2dd7-28c8-45bb-819e-7d5d36f7b10f]: [Errno 2] No such file or directory: 'systool' {{(pid=157799) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 12 21:19:31.815719 ncdv-master-1 cinder-backup[129443]: Traceback (most recent call last): Mar 12 21:19:31.815719 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 12 21:19:31.815719 ncdv-master-1 cinder-backup[129443]: ret = func(*f_args, **f_kwargs) Mar 12 21:19:31.815719 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 12 21:19:31.815719 ncdv-master-1 cinder-backup[129443]: return func(*args, **kwargs) Mar 12 21:19:31.815719 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 12 21:19:31.815719 ncdv-master-1 cinder-backup[129443]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 12 21:19:31.815719 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 12 21:19:31.815719 ncdv-master-1 cinder-backup[129443]: return putils.execute(on_execute=on_execute, Mar 12 21:19:31.815719 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 12 21:19:31.815719 ncdv-master-1 cinder-backup[129443]: obj = subprocess.Popen(cmd, Mar 12 21:19:31.815719 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 12 21:19:31.815719 ncdv-master-1 cinder-backup[129443]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 12 21:19:31.815719 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 12 21:19:31.815719 ncdv-master-1 cinder-backup[129443]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 12 21:19:31.815719 ncdv-master-1 cinder-backup[129443]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 12 21:19:31.816890 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[32cb2dd7-28c8-45bb-819e-7d5d36f7b10f]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:19:31.817247 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:19:31.822482 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 12 21:19:31.823311 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[bfa75e31-c475-489f-85ad-576f7b71cccb]: [Errno 2] No such file or directory: 'systool' {{(pid=157799) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 12 21:19:31.823311 ncdv-master-1 cinder-backup[129443]: Traceback (most recent call last): Mar 12 21:19:31.823311 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 12 21:19:31.823311 ncdv-master-1 cinder-backup[129443]: ret = func(*f_args, **f_kwargs) Mar 12 21:19:31.823311 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 12 21:19:31.823311 ncdv-master-1 cinder-backup[129443]: return func(*args, **kwargs) Mar 12 21:19:31.823311 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 12 21:19:31.823311 ncdv-master-1 cinder-backup[129443]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 12 21:19:31.823311 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 12 21:19:31.823311 ncdv-master-1 cinder-backup[129443]: return putils.execute(on_execute=on_execute, Mar 12 21:19:31.823311 ncdv-master-1 cinder-backup[129443]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 12 21:19:31.823311 ncdv-master-1 cinder-backup[129443]: obj = subprocess.Popen(cmd, Mar 12 21:19:31.823311 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 12 21:19:31.823311 ncdv-master-1 cinder-backup[129443]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 12 21:19:31.823311 ncdv-master-1 cinder-backup[129443]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 12 21:19:31.823311 ncdv-master-1 cinder-backup[129443]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 12 21:19:31.823311 ncdv-master-1 cinder-backup[129443]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 12 21:19:31.824418 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[bfa75e31-c475-489f-85ad-576f7b71cccb]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:19:31.824981 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:19:31.833279 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.008s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:19:31.833609 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[4c170cee-9f1e-4d87-b131-f7e3f0831345]: (4, ('/dev/sda1\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:19:31.834734 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:19:31.843301 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.008s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:19:31.843626 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[88c67e73-d019-4346-ad14-488b1e55ad89]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:19:31.844674 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:19:31.852267 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.006s {{(pid=157799) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:19:31.852575 ncdv-master-1 cinder-backup[129443]: DEBUG oslo.privsep.daemon [-] privsep: reply[5be318df-346c-45f0-998c-e5d74956c8d3]: (4, ('55f4b1f4-b5b7-af4f-9adb-821e08822a61\n', '')) {{(pid=157799) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 12 21:19:31.853352 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] Running cmd (subprocess): nvme version {{(pid=129443) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 12 21:19:31.871884 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_concurrency.processutils [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] CMD "nvme version" returned: 0 in 0.018s {{(pid=129443) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 12 21:19:31.873676 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129443) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 12 21:19:31.874101 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129443) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 12 21:19:31.874478 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.lightos [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:d9949f82-06c8-4f47-8c57-a55eecab76d2 dsc: {{(pid=129443) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 12 21:19:31.875120 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.utils [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] <== get_connector_properties: return (79ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:77472599036', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '55f4b1f4-b5b7-af4f-9adb-821e08822a61', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:d9949f82-06c8-4f47-8c57-a55eecab76d2', 'found_dsc': ''} {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:19:32.549548 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connector [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] Factory for rbd on None {{(pid=129443) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 12 21:19:32.556763 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-e5912a2c-aeb4-40f4-a05e-781c6b342f11', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'e5912a2c-aeb4-40f4-a05e-781c6b342f11', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:19:32.557930 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.linuxrbd [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] opening connection to ceph cluster (timeout=-1). {{(pid=129443) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 12 21:19:32.600183 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] <== connect_volume: return (43ms) {'path': } {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:19:32.608614 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_db.sqlalchemy.engines [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129443) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 12 21:19:32.619765 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] Starting restore from Ceph backup=6f2a8acc-32b2-451c-98bb-63cbbae2c432 to volume=volume-e5912a2c-aeb4-40f4-a05e-781c6b342f11 {{(pid=129443) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1274}} Mar 12 21:19:32.686925 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] Looking for snapshot of backup base 'volume-b68846b1-e2e8-4551-9e39-b8fa771ce5ff.backup.6f2a8acc-32b2-451c-98bb-63cbbae2c432' {{(pid=129443) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 12 21:19:32.687310 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] Found snapshot 'backup.6f2a8acc-32b2-451c-98bb-63cbbae2c432.snap.1647119962.5825202' {{(pid=129443) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Mar 12 21:19:32.691853 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] Volume file is RBD. {{(pid=129443) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1184}} Mar 12 21:19:32.729968 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] Attempting differential restore. {{(pid=129443) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1234}} Mar 12 21:19:32.730466 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] Attempting incremental restore from base='volume-b68846b1-e2e8-4551-9e39-b8fa771ce5ff.backup.6f2a8acc-32b2-451c-98bb-63cbbae2c432' snap='backup.6f2a8acc-32b2-451c-98bb-63cbbae2c432.snap.1647119962.5825202' {{(pid=129443) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1089}} Mar 12 21:19:32.730862 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] Performing differential transfer from 'volume-b68846b1-e2e8-4551-9e39-b8fa771ce5ff.backup.6f2a8acc-32b2-451c-98bb-63cbbae2c432' to 'volume-e5912a2c-aeb4-40f4-a05e-781c6b342f11' {{(pid=129443) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Mar 12 21:19:32.731255 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-b68846b1-e2e8-4551-9e39-b8fa771ce5ff.backup.6f2a8acc-32b2-451c-98bb-63cbbae2c432@backup.6f2a8acc-32b2-451c-98bb-63cbbae2c432.snap.1647119962.5825202 -' into... {{(pid=129443) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Mar 12 21:19:32.731619 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_ixzan1wn --pool volumes - volumes/volume-e5912a2c-aeb4-40f4-a05e-781c6b342f11' {{(pid=129443) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Mar 12 21:19:34.832519 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:19:34.838477 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:19:36.718608 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] Restore transfer completed in 3.9873s {{(pid=129443) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1111}} Mar 12 21:19:36.719770 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=129443) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1285}} Mar 12 21:19:36.744588 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] Restoring volume base metadata {{(pid=129443) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} Mar 12 21:19:36.765206 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] No metadata of type 'volume-metadata' to restore {{(pid=129443) put /opt/stack/cinder/cinder/backup/driver.py:345}} Mar 12 21:19:36.765494 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.driver [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] Restoring volume glance metadata {{(pid=129443) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}} Mar 12 21:19:36.905019 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] Restore to volume e5912a2c-aeb4-40f4-a05e-781c6b342f11 finished successfully. {{(pid=129443) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1292}} Mar 12 21:19:36.906782 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-e5912a2c-aeb4-40f4-a05e-781c6b342f11', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'e5912a2c-aeb4-40f4-a05e-781c6b342f11', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 12 21:19:36.917841 ncdv-master-1 cinder-backup[129443]: DEBUG os_brick.initiator.connectors.rbd [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] <== disconnect_volume: return (10ms) None {{(pid=129443) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 12 21:19:36.968477 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.manager [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] Encryption key ID for volume e5912a2c-aeb4-40f4-a05e-781c6b342f11 already matches encryption key ID in backup 6f2a8acc-32b2-451c-98bb-63cbbae2c432. {{(pid=129443) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} Mar 12 21:19:36.999921 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-ad9dbc4f-fe22-4273-9487-f77e91bbe1f7 tempest-TestVolumeBackupRestore-1490532501 None] Finished restoring backup 6f2a8acc-32b2-451c-98bb-63cbbae2c432 to volume e5912a2c-aeb4-40f4-a05e-781c6b342f11. Mar 12 21:20:47.934599 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-e9963fd8-5cd0-49b3-885f-e6cd249e6bf4 tempest-TestVolumeBackupRestore-1490532501 None] Delete backup started, backup: 6f2a8acc-32b2-451c-98bb-63cbbae2c432. Mar 12 21:20:47.937208 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-e9963fd8-5cd0-49b3-885f-e6cd249e6bf4 tempest-TestVolumeBackupRestore-1490532501 None] Delete started for backup=6f2a8acc-32b2-451c-98bb-63cbbae2c432 {{(pid=129443) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Mar 12 21:20:47.937843 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-e9963fd8-5cd0-49b3-885f-e6cd249e6bf4 tempest-TestVolumeBackupRestore-1490532501 None] Trying diff format basename='volume-b68846b1-e2e8-4551-9e39-b8fa771ce5ff.backup.6f2a8acc-32b2-451c-98bb-63cbbae2c432' for backup base image of volume b68846b1-e2e8-4551-9e39-b8fa771ce5ff. {{(pid=129443) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Mar 12 21:20:48.015253 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-e9963fd8-5cd0-49b3-885f-e6cd249e6bf4 tempest-TestVolumeBackupRestore-1490532501 None] Looking for snapshot of backup base 'volume-b68846b1-e2e8-4551-9e39-b8fa771ce5ff.backup.6f2a8acc-32b2-451c-98bb-63cbbae2c432' {{(pid=129443) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 12 21:20:48.015715 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-e9963fd8-5cd0-49b3-885f-e6cd249e6bf4 tempest-TestVolumeBackupRestore-1490532501 None] Found snapshot 'backup.6f2a8acc-32b2-451c-98bb-63cbbae2c432.snap.1647119962.5825202' {{(pid=129443) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Mar 12 21:20:48.016375 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-e9963fd8-5cd0-49b3-885f-e6cd249e6bf4 tempest-TestVolumeBackupRestore-1490532501 None] Deleting backup snapshot='backup.6f2a8acc-32b2-451c-98bb-63cbbae2c432.snap.1647119962.5825202' {{(pid=129443) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Mar 12 21:20:48.817908 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.drivers.ceph [None req-e9963fd8-5cd0-49b3-885f-e6cd249e6bf4 tempest-TestVolumeBackupRestore-1490532501 None] Deleting backup base image='volume-b68846b1-e2e8-4551-9e39-b8fa771ce5ff.backup.6f2a8acc-32b2-451c-98bb-63cbbae2c432' of volume b68846b1-e2e8-4551-9e39-b8fa771ce5ff. Mar 12 21:20:49.606798 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-e9963fd8-5cd0-49b3-885f-e6cd249e6bf4 tempest-TestVolumeBackupRestore-1490532501 None] Base backup image='volume-b68846b1-e2e8-4551-9e39-b8fa771ce5ff.backup.6f2a8acc-32b2-451c-98bb-63cbbae2c432' of volume b68846b1-e2e8-4551-9e39-b8fa771ce5ff deleted. {{(pid=129443) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Mar 12 21:20:49.664507 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.backup.drivers.ceph [None req-e9963fd8-5cd0-49b3-885f-e6cd249e6bf4 tempest-TestVolumeBackupRestore-1490532501 None] Delete of backup '6f2a8acc-32b2-451c-98bb-63cbbae2c432' for volume 'b68846b1-e2e8-4551-9e39-b8fa771ce5ff' finished. {{(pid=129443) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Mar 12 21:20:49.673086 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_db.sqlalchemy.engines [None req-e9963fd8-5cd0-49b3-885f-e6cd249e6bf4 tempest-TestVolumeBackupRestore-1490532501 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129443) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 12 21:20:49.704666 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.quota [None req-e9963fd8-5cd0-49b3-885f-e6cd249e6bf4 tempest-TestVolumeBackupRestore-1490532501 None] Created reservations ['8c0911fd-212d-4a4c-b3bb-a5fcdc1d01eb', '5f7377a3-817a-4177-88c5-6d14f4ffa051'] {{(pid=129443) reserve /opt/stack/cinder/cinder/quota.py:755}} Mar 12 21:20:49.733985 ncdv-master-1 cinder-backup[129443]: INFO cinder.backup.manager [None req-e9963fd8-5cd0-49b3-885f-e6cd249e6bf4 tempest-TestVolumeBackupRestore-1490532501 None] Delete backup finished, backup 6f2a8acc-32b2-451c-98bb-63cbbae2c432 deleted. Mar 12 21:21:34.833214 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:21:34.833214 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:22:34.830507 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:22:34.830507 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:23:34.833582 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:23:34.834239 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:24:34.830582 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:24:34.831253 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:25:34.828264 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:25:34.828948 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:26:34.834052 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:26:34.834729 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:27:34.838694 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:27:34.839261 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:28:34.835648 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:28:34.836326 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:29:34.833213 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:29:34.833909 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:30:34.839859 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:30:34.840893 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:31:34.836663 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:31:34.837329 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:32:34.833631 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:32:34.834373 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:33:34.830829 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:33:34.831496 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:34:34.835601 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:34:34.836451 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:35:34.833484 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:35:34.834230 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:36:34.838496 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:36:34.839217 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:37:34.843812 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:37:34.844489 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:38:34.848507 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:38:34.849142 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:39:34.849629 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:39:34.850302 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:40:34.854131 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:40:34.854814 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:41:34.859551 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:41:34.860217 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:42:34.855919 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:42:34.856599 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:43:34.860952 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:43:34.861554 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:44:34.858195 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:44:34.858871 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:45:34.863861 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:45:34.864384 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:46:34.868910 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:46:34.870077 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:47:34.866833 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:47:34.867444 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:48:34.872646 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:48:34.873223 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:49:34.878259 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:49:34.878871 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:50:34.884317 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:50:34.885020 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:51:34.879911 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:51:34.880494 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:52:34.885043 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:52:34.885635 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:53:34.882131 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:53:34.882733 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:54:34.886493 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:54:34.887082 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:55:34.891334 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:55:34.892081 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:56:34.890992 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:56:34.891667 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:57:34.893491 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:57:34.894126 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:58:34.898811 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:58:34.899436 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 21:59:34.894491 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 21:59:34.895098 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 22:00:34.891200 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 22:00:34.891804 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 22:01:34.897145 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 22:01:34.898157 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 22:02:34.898671 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 22:02:34.899218 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 22:03:34.903850 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 22:03:34.904432 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 22:04:34.908617 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 22:04:34.909235 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 22:05:34.914905 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 22:05:34.915455 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 22:06:34.920283 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 22:06:34.920830 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 22:07:34.917125 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 22:07:34.917699 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 22:08:34.922172 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 22:08:34.922760 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 12 22:09:34.927501 ncdv-master-1 cinder-backup[129443]: DEBUG oslo_service.periodic_task [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129443) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 12 22:09:34.928046 ncdv-master-1 cinder-backup[129443]: DEBUG cinder.manager [None req-da5b1524-1f38-4890-bcfd-494f42ff45e1 None None] Notifying Schedulers of capabilities ... {{(pid=129443) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}