-- Logs begin at Sat 2022-03-26 20:01:24 UTC, end at Sat 2022-03-26 22:09:11 UTC. -- Mar 26 20:40:03.393128 ncdv-master-1 systemd[1]: Started Devstack devstack@c-bak.service. Mar 26 20:40:06.796261 ncdv-master-1 cinder-backup[134311]: INFO cinder.cmd.backup [-] Backup running with 1 processes. Mar 26 20:40:06.798024 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_concurrency.lockutils [-] Acquired lock "singleton_lock" {{(pid=134311) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:294}} Mar 26 20:40:06.798598 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_concurrency.lockutils [-] Releasing lock "singleton_lock" {{(pid=134311) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:312}} Mar 26 20:40:06.946217 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_db.sqlalchemy.engines [None req-65d40358-d574-424f-940d-832f68e29aef None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134311) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 26 20:40:07.184715 ncdv-master-1 cinder-backup[134311]: INFO cinder.rpc [None req-65d40358-d574-424f-940d-832f68e29aef None None] Automatically selected cinder-backup objects version 1.39 as minimum service version. Mar 26 20:40:07.190205 ncdv-master-1 cinder-backup[134311]: INFO cinder.rpc [None req-65d40358-d574-424f-940d-832f68e29aef None None] Automatically selected cinder-backup RPC version 2.3 as minimum service version. Mar 26 20:40:07.195077 ncdv-master-1 cinder-backup[134311]: INFO cinder.rpc [None req-65d40358-d574-424f-940d-832f68e29aef None None] Automatically selected cinder-volume objects version 1.39 as minimum service version. Mar 26 20:40:07.204742 ncdv-master-1 cinder-backup[134311]: INFO cinder.rpc [None req-65d40358-d574-424f-940d-832f68e29aef None None] Automatically selected cinder-volume RPC version 3.18 as minimum service version. Mar 26 20:40:07.210837 ncdv-master-1 cinder-backup[134311]: INFO cinder.rpc [None req-65d40358-d574-424f-940d-832f68e29aef None None] Automatically selected cinder-scheduler objects version 1.39 as minimum service version. Mar 26 20:40:07.215858 ncdv-master-1 cinder-backup[134311]: INFO cinder.rpc [None req-65d40358-d574-424f-940d-832f68e29aef None None] Automatically selected cinder-scheduler RPC version 3.12 as minimum service version. Mar 26 20:40:07.278076 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_db.sqlalchemy.engines [None req-65d40358-d574-424f-940d-832f68e29aef None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134311) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 26 20:40:07.303813 ncdv-master-1 cinder-backup[134311]: INFO oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] Starting 1 workers Mar 26 20:40:07.309612 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] Started child 134755 {{(pid=134311) _start_child /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:575}} Mar 26 20:40:07.312265 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] Full set of CONF: {{(pid=134311) wait /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:649}} Mar 26 20:40:07.312890 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] ******************************************************************************** {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Mar 26 20:40:07.313341 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] Configuration options gathered from: {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Mar 26 20:40:07.313847 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] command line args: ['--config-file', '/etc/cinder/cinder.conf'] {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Mar 26 20:40:07.314754 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] config files: ['/etc/cinder/cinder.conf'] {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Mar 26 20:40:07.315236 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] ================================================================================ {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Mar 26 20:40:07.316159 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] allow_availability_zone_fallback = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.316714 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] allow_compression_on_image_upload = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.317422 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] allowed_direct_url_schemes = [] {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.317993 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] api_rate_limit = True {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.318764 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] auth_strategy = keystone {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.321243 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_availability_zone = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.321919 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backup_api_class = cinder.backup.api.API {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.322514 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backup_ceph_chunk_size = 134217728 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.323529 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backup_ceph_conf = /etc/ceph/ceph.conf {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.324866 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backup_ceph_image_journals = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.325453 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backup_ceph_pool = backups {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.328228 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backup_ceph_stripe_count = 0 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.328937 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backup_ceph_stripe_unit = 0 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.329499 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backup_ceph_user = cinder-bak {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.331165 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backup_driver = cinder.backup.drivers.ceph.CephBackupDriver {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.331767 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backup_driver_init_check_interval = 60 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.333008 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backup_driver_stats_polling_interval = 60 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.335049 ncdv-master-1 cinder-backup[134755]: INFO cinder.service [-] Starting cinder-backup node (version 20.1.0) Mar 26 20:40:07.336074 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backup_manager = cinder.backup.manager.BackupManager {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.336641 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backup_max_operations = 15 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.337190 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backup_metadata_version = 2 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.337881 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backup_name_template = backup-%s {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.339048 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backup_native_threads_pool_size = 60 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.339878 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backup_object_number_per_notification = 10 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.341712 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backup_service_inithost_offload = True {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.342884 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backup_timer_interval = 120 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.344043 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backup_use_same_host = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.344444 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backup_use_temp_snapshot = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.344874 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backup_workers = 1 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.345277 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] chap_password = **** {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.345722 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] chap_username = {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.346124 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.346554 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cinder_internal_tenant_project_id = 5d311a58db4c44ebbd88fbdf463fab47 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.346987 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cinder_internal_tenant_user_id = 416ac158078b4b678c236e0ad811acad {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.347380 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] compression_format = gzip {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.347799 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] config_dir = [] {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.348115 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] config_file = ['/etc/cinder/cinder.conf'] {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.348511 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] config_source = [] {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.348924 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] consistencygroup_api_class = cinder.consistencygroup.api.API {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.349312 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] control_exchange = openstack {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.349806 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] debug = True {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.350152 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] default_availability_zone = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.350488 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] default_group_type = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.351247 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'swiftclient=WARN', 'botocore=WARN'] {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.352033 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] default_volume_type = ceph {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.353447 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] driver_client_cert = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.353810 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] driver_client_cert_key = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.354178 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] driver_data_namespace = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.354541 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] driver_ssl_cert_path = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.355222 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] driver_ssl_cert_verify = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.355222 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] driver_use_ssl = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.355599 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] enable_new_services = True {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.355988 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] enable_unsupported_driver = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.356398 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] enabled_backends = ['ceph'] {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.356750 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] enforce_multipath_for_image_xfer = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.357094 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] filter_function = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.357475 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] glance_api_insecure = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.358374 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] glance_api_servers = ['http://10.222.0.16/image'] {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.358722 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] glance_api_ssl_compression = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.359064 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] glance_ca_certificates_file = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.359493 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] glance_catalog_info = image:glance:publicURL {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.359888 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] glance_certfile = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.360393 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] glance_core_properties = ['checksum', 'container_format', 'disk_format', 'image_name', 'image_id', 'min_disk', 'min_ram', 'name', 'size'] {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.363944 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] glance_keyfile = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.363944 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] glance_num_retries = 3 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.363944 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] glance_request_timeout = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.363944 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] goodness_function = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.363944 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] graceful_shutdown_timeout = 5 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.363944 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] group_api_class = cinder.group.api.API {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.363944 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] host = ncdv-master-1 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.364499 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] iet_conf = /etc/iet/ietd.conf {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.364499 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] image_compress_on_upload = True {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.364499 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] image_conversion_address_space_limit = 1 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.364499 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] image_conversion_cpu_limit = 60 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.364982 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] image_conversion_dir = /opt/stack/data/cinder/conversion {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.365352 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] image_upload_use_cinder_backend = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.365780 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] image_upload_use_internal_tenant = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.366135 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] image_volume_cache_enabled = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.366483 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] image_volume_cache_max_count = 0 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.366830 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] image_volume_cache_max_size_gb = 0 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.367213 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] instance_format = [instance: %(uuid)s] {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.367562 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] instance_uuid_format = [instance: %(uuid)s] {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.368003 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] iscsi_iotype = fileio {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.368393 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] iscsi_secondary_ip_addresses = [] {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.368745 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] iscsi_target_flags = {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.369121 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] iscsi_write_cache = on {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.369470 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] iser_helper = tgtadm {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.370120 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] iser_ip_address = 10.222.0.16 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.374941 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] iser_port = 3260 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.374941 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.374941 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] log_config_append = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.374941 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.374941 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] log_dir = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.374941 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] log_file = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.375488 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] log_options = True {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.375488 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] log_rotate_interval = 1 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.375488 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] log_rotate_interval_type = days {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.375488 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] log_rotation_type = none {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.375488 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.375488 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.375982 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.376407 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.376703 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] logging_user_identity_format = %(user)s %(project)s %(domain)s %(user_domain)s %(project_domain)s {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.377081 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] max_age = 0 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.377447 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] max_logfile_count = 30 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.377945 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] max_logfile_size_mb = 200 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.378378 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] max_over_subscription_ratio = 20.0 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.378757 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] message_reap_interval = 86400 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.379113 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] message_ttl = 2592000 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.379452 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] monkey_patch = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.379960 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] monkey_patch_modules = [] {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.380228 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] my_ip = 10.222.0.16 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.380567 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] no_snapshot_gb_quota = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.380929 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] num_iser_scan_tries = 3 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.381304 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] num_shell_tries = 3 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.381642 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] num_volume_device_scan_tries = 3 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.381984 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] nvmet_ns_id = 10 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.382321 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] nvmet_port_id = 1 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.382712 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] osapi_max_limit = 1000 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.383079 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] osapi_volume_ext_list = [] {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.383483 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] osapi_volume_extension = ['cinder.api.contrib.standard_extensions'] {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.383849 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] osapi_volume_listen = 0.0.0.0 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.389703 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] osapi_volume_listen_port = 8776 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.389703 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] osapi_volume_use_ssl = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.389703 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] osapi_volume_workers = 2 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.389703 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] per_volume_size_limit = -1 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.389703 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] periodic_fuzzy_delay = 60 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.389703 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] periodic_interval = 60 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.389703 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] project_id_regex = [0-9a-f\-]+ {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.390279 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] public_endpoint = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.390279 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] publish_errors = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.390279 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] quota_backup_gigabytes = 1000 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.390279 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] quota_backups = 10 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.390279 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] quota_consistencygroups = 10 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.390279 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] quota_driver = cinder.quota.DbQuotaDriver {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.390279 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] quota_gigabytes = 1000 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.390699 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] quota_groups = 10 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.390699 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] quota_snapshots = 10 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.390699 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] quota_volumes = 10 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.390699 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] rate_limit_burst = 0 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.390935 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] rate_limit_except_level = CRITICAL {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.391092 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] rate_limit_interval = 0 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.391450 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] replication_device = **** {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.391789 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] report_discard_supported = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.392154 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] report_interval = 10 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.392590 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] reservation_clean_interval = 86400 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.392874 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] reservation_expire = 86400 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.393217 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] reserved_percentage = 0 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.393581 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] resource_query_filters_file = /etc/cinder/resource_filters.json {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.393927 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] restore_discard_excess_bytes = True {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.394523 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] rootwrap_config = /etc/cinder/rootwrap.conf {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.399636 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] rpc_response_timeout = 120 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.399636 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] run_external_periodic_tasks = True {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.399636 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] scheduler_manager = cinder.scheduler.manager.SchedulerManager {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.399636 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] scst_target_driver = iscsi {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.399636 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] scst_target_iqn_name = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.399636 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] service_down_time = 60 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.400229 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] snapshot_name_template = snapshot-%s {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.401257 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] split_loggers = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.401586 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] state_path = /opt/stack/data/cinder {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.402257 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] storage_availability_zone = nova {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.402709 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] storage_protocol = iscsi {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.403087 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] syslog_log_facility = LOG_USER {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.403452 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] target_helper = lioadm {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.403903 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] target_ip_address = 10.222.0.16 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.404310 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] target_port = 3260 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.404709 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] target_prefix = iqn.2010-10.org.openstack: {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.405108 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] target_protocol = iscsi {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.405457 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] trace_flags = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.405831 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] transfer_api_class = cinder.transfer.api.API {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.406193 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] transport_url = **** {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.406537 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] until_refresh = 0 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.406913 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] use_chap_auth = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.407260 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] use_default_quota_class = True {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.407600 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] use_eventlog = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.407955 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] use_journal = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.408303 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] use_json = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.408645 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] use_multipath_for_image_xfer = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.408989 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] use_stderr = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.409326 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] use_syslog = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.409706 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] verify_glance_signatures = enabled {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.410064 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] volume_api_class = cinder.volume.api.API {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.410446 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] volume_backend_name = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.410836 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] volume_clear = zero {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.411170 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] volume_clear_ionice = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.411508 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] volume_clear_size = 0 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.411879 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.412293 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] volume_copy_bps_limit = 0 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.412517 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.lockutils [None req-62fcbc4c-a281-4594-b0d1-afc4a8f811ee None None] Lock "cleanup_incomplete_backups_134311" acquired by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: waited 0.002s {{(pid=134755) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:386}} Mar 26 20:40:07.412827 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] volume_dd_blocksize = 1M {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.419962 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-62fcbc4c-a281-4594-b0d1-afc4a8f811ee None None] Cleaning up incomplete backup operations. Mar 26 20:40:07.420093 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] volume_manager = cinder.volume.manager.VolumeManager {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.420093 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] volume_name_template = volume-%s {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.420093 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] volume_usage_audit_period = month {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.420093 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] volumes_dir = /opt/stack/data/cinder/volumes {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.420093 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] watch_log_file = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 26 20:40:07.420093 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.421030 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.421030 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_policy.enforce_new_defaults = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.421030 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_policy.enforce_scope = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.421030 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_policy.policy_default_rule = default {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.421030 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_policy.policy_dirs = ['policy.d'] {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.421030 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_policy.policy_file = policy.yaml {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.421394 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.421394 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.421394 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.421394 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.421394 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.421394 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_concurrency.disable_process_locking = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.421757 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_concurrency.lock_path = /opt/stack/data/cinder {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.421757 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.admin_password = **** {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.421757 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.admin_tenant_name = admin {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.421757 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.admin_token = **** {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.422031 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.admin_user = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.422196 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.auth_admin_prefix = {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.422541 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.422906 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.auth_port = 35357 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.423281 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.auth_protocol = https {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.423687 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.auth_section = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.424094 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.auth_type = password {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.424456 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.auth_uri = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.424825 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.auth_version = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.425166 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.cache = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.425522 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.cafile = /opt/stack/data/ca-bundle.pem {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.425858 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.certfile = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.426201 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.delay_auth_decision = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.427683 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.enforce_token_bind = permissive {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.428099 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.http_connect_timeout = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.428574 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.http_request_max_retries = 3 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.428929 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.identity_uri = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.429359 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.include_service_catalog = True {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.429711 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.insecure = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.430130 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.interface = public {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.430485 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.keyfile = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.430891 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.431320 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.431806 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.432252 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.432681 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.433163 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.memcache_secret_key = **** {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.433541 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.memcache_security_strategy = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.433934 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.434345 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.memcached_servers = ['localhost:11211'] {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.435582 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.region_name = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.436713 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.service_token_roles = ['service'] {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.437209 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.service_token_roles_required = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.437572 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.service_type = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.437963 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.token_cache_time = 300 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.454115 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] keystone_authtoken.www_authenticate_uri = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.455112 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.backend = dogpile.cache.null {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.456008 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.backend_argument = **** {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.456674 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.config_prefix = cache.oslo {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.457290 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.dead_timeout = 60.0 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.457969 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.debug_cache_backend = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.458534 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.enable_retry_client = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.462007 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.enable_socket_keepalive = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.462552 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.enabled = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.463173 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.expiration_time = 600 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.463795 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.hashclient_retry_attempts = 2 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.464518 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.hashclient_retry_delay = 1.0 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.465239 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.memcache_dead_retry = 300 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.465839 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.memcache_pool_connection_get_timeout = 10 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.466334 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.memcache_pool_flush_on_reconnect = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.466902 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.memcache_pool_maxsize = 10 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.467828 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.memcache_pool_unused_timeout = 60 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.468390 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.memcache_servers = ['localhost:11211'] {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.468970 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.memcache_socket_timeout = 1.0 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.469486 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.proxies = [] {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.470027 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.retry_attempts = 2 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.470531 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.retry_delay = 0.0 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.471083 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.socket_keepalive_count = 1 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.471572 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.socket_keepalive_idle = 1 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.472153 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.socket_keepalive_interval = 1 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.472644 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.tls_allowed_ciphers = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.473183 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.tls_cafile = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.473688 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.tls_certfile = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.474214 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.tls_enabled = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.474713 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cache.tls_keyfile = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.475304 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] coordination.backend_url = etcd3+http://10.222.0.16:2379 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.476064 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] database.backend = sqlalchemy {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.476530 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] database.connection = **** {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.477117 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] database.connection_debug = 0 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.477631 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] database.connection_parameters = {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.478120 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] database.connection_recycle_time = 3600 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.478623 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] database.connection_trace = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.479113 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] database.db_inc_retry_interval = True {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.479615 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] database.db_max_retries = 20 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.480209 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] database.db_max_retry_interval = 10 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.480722 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] database.db_retry_interval = 1 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.481266 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] database.max_overflow = 50 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.481761 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] database.max_pool_size = 5 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.482279 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] database.max_retries = 10 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.483391 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] database.mysql_enable_ndb = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.484091 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] database.mysql_sql_mode = TRADITIONAL {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.484598 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] database.pool_timeout = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.485102 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] database.retry_interval = 10 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.485635 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] database.slave_connection = **** {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.486163 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] database.sqlite_synchronous = True {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.486861 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] database.use_db_reconnect = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.487398 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.487910 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.488504 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_metrics.metrics_process_name = {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.489016 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.489554 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.490107 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] profiler.connection_string = messaging:// {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.490554 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] profiler.enabled = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.491061 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] profiler.es_doc_type = notification {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.491553 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] profiler.es_scroll_size = 10000 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.492121 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] profiler.es_scroll_time = 2m {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.493867 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] profiler.filter_error_trace = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.494401 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] profiler.hmac_keys = SECRET_KEY {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.494907 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] profiler.sentinel_service_name = mymaster {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.495414 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] profiler.socket_timeout = 0.1 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.495940 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] profiler.trace_sqlalchemy = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.496504 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] privsep_osbrick.capabilities = [21] {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.497039 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] privsep_osbrick.group = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.497541 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] privsep_osbrick.helper_command = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.498050 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] privsep_osbrick.logger_name = oslo_privsep.daemon {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.499104 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] privsep_osbrick.thread_pool_size = 4 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.499612 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] privsep_osbrick.user = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.500178 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cinder_sys_admin.capabilities = [0, 1, 2, 3, 12, 21] {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.500709 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cinder_sys_admin.group = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.501204 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cinder_sys_admin.helper_command = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.501742 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cinder_sys_admin.logger_name = oslo_privsep.daemon {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.502244 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cinder_sys_admin.thread_pool_size = 4 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.502723 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] cinder_sys_admin.user = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.503528 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] service_user.auth_section = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.504140 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] service_user.auth_type = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.504624 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] service_user.cafile = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.505119 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] service_user.certfile = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.505631 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] service_user.collect_timing = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.506269 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] service_user.insecure = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.506750 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] service_user.keyfile = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.507269 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] service_user.send_service_user_token = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.507793 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] service_user.split_loggers = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.508324 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] service_user.timeout = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.508952 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.backend_availability_zone = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.509463 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.chap_password = **** {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.510012 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.chap_username = {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.510516 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.511023 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.deferred_deletion_delay = 0 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.511519 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.deferred_deletion_purge_interval = 60 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.512074 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.driver_client_cert = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.512556 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.driver_client_cert_key = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.513056 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.driver_data_namespace = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.513553 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.driver_ssl_cert_path = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.514085 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.driver_ssl_cert_verify = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.514681 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.driver_use_ssl = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.515172 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.enable_deferred_deletion = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.515658 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.enable_unsupported_driver = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.516230 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.enforce_multipath_for_image_xfer = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.516782 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.filter_function = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.517277 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.goodness_function = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.517849 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.iet_conf = /etc/iet/ietd.conf {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.518356 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.image_upload_use_cinder_backend = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.518857 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.image_upload_use_internal_tenant = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.519376 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.image_volume_cache_enabled = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.520127 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.image_volume_cache_max_count = 0 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.520646 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.image_volume_cache_max_size_gb = 0 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.521168 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.iscsi_iotype = fileio {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.521675 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.iscsi_secondary_ip_addresses = [] {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.522188 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.iscsi_target_flags = {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.522692 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.iscsi_write_cache = on {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.523197 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.iser_helper = tgtadm {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.523877 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.iser_ip_address = 10.222.0.16 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.524424 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.iser_port = 3260 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.525007 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.525522 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.max_over_subscription_ratio = 20.0 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.526085 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.num_iser_scan_tries = 3 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.526583 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.num_shell_tries = 3 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.527077 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.num_volume_device_scan_tries = 3 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.527593 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.nvmet_ns_id = 10 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.528125 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.nvmet_port_id = 1 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.528691 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.rados_connect_timeout = -1 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.529188 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.rados_connection_interval = 5 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.530221 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.rados_connection_retries = 3 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.530726 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.rbd_ceph_conf = {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.531247 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.rbd_cluster_name = ceph {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.531741 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.rbd_exclusive_cinder_pool = True {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.532263 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.rbd_flatten_volume_from_snapshot = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.532773 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.rbd_max_clone_depth = 5 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.533299 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.rbd_pool = rbd {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.533793 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.rbd_secret_uuid = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.534422 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.rbd_store_chunk_size = 4 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.535133 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.rbd_user = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.535644 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.replication_connect_timeout = 5 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.536407 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.replication_device = **** {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.536982 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.report_discard_supported = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.537596 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.report_dynamic_total_capacity = True {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.538206 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.reserved_percentage = 0 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.538718 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.scst_target_driver = iscsi {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.539392 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.scst_target_iqn_name = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.539900 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.storage_protocol = iscsi {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.540478 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.target_helper = tgtadm {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.541363 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.target_ip_address = 10.222.0.16 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.542026 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.target_port = 3260 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.542608 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.target_prefix = iqn.2010-10.org.openstack: {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.543344 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.target_protocol = iscsi {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.543884 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.trace_flags = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.544792 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.unique_fqdn_network = True {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.546043 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.use_chap_auth = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.546721 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.use_multipath_for_image_xfer = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.547439 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.volume_backend_name = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.548300 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.volume_clear = zero {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.548888 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.volume_clear_ionice = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.549460 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.volume_clear_size = 0 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.550135 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.550697 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.volume_copy_bps_limit = 0 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.551198 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.volume_dd_blocksize = 1M {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.551865 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] backend_defaults.volumes_dir = /opt/stack/data/cinder/volumes {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.552473 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_reports.file_event_handler = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.553016 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_reports.file_event_handler_interval = 1 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.553784 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_reports.log_dir = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.554369 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.554918 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.555445 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.556033 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.556598 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.557105 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.557678 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_rabbit.heartbeat_in_pthread = True {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.558163 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_rabbit.heartbeat_rate = 2 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.558650 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.559130 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.559700 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.560247 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.560814 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.561349 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.561904 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.562436 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.562924 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.563459 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.563979 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.564462 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.565009 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.565503 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.566109 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_rabbit.ssl = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.566606 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_rabbit.ssl_ca_file = {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.567122 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_rabbit.ssl_cert_file = {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.567784 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_rabbit.ssl_key_file = {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.568325 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_rabbit.ssl_version = {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.568957 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_notifications.driver = [] {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.569474 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_notifications.retry = -1 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.570062 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.570565 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] oslo_messaging_notifications.transport_url = **** {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.571152 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] key_manager.backend = cinder.keymgr.conf_key_mgr.ConfKeyManager {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.571726 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] barbican.auth_endpoint = http://localhost/identity/v3 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.572260 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] barbican.barbican_api_version = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.572757 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] barbican.barbican_endpoint = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.573280 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] barbican.barbican_endpoint_type = public {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.573788 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] barbican.barbican_region_name = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.574269 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] barbican.cafile = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.574765 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] barbican.certfile = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.575257 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] barbican.collect_timing = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.575751 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] barbican.insecure = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.576269 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] barbican.keyfile = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.576797 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] barbican.number_of_retries = 60 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.577319 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] barbican.retry_delay = 1 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.577801 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] barbican.send_service_user_token = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.578324 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] barbican.split_loggers = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.578800 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] barbican.timeout = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.579289 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] barbican.verify_ssl = True {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.579788 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] barbican.verify_ssl_path = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.580342 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] barbican_service_user.auth_section = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.580861 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] barbican_service_user.auth_type = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.581350 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] barbican_service_user.cafile = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.581829 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] barbican_service_user.certfile = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.582343 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] barbican_service_user.collect_timing = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.582857 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] barbican_service_user.insecure = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.583354 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] barbican_service_user.keyfile = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.583840 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] barbican_service_user.split_loggers = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.584355 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] barbican_service_user.timeout = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.584875 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] vault.approle_role_id = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.585422 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] vault.approle_secret_id = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.585903 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] vault.cafile = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.586417 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] vault.certfile = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.587063 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] vault.collect_timing = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.587551 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] vault.insecure = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.588070 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] vault.keyfile = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.588594 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] vault.kv_mountpoint = secret {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.589100 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] vault.kv_version = 2 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.589601 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] vault.namespace = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.590093 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] vault.root_token_id = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.590591 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] vault.split_loggers = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.591093 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] vault.ssl_ca_crt_file = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.591583 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] vault.timeout = None {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.592635 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] vault.use_ssl = False {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.593191 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] vault.vault_url = http://127.0.0.1:8200 {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 26 20:40:07.593616 ncdv-master-1 cinder-backup[134311]: DEBUG oslo_service.service [None req-65d40358-d574-424f-940d-832f68e29aef None None] ******************************************************************************** {{(pid=134311) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Mar 26 20:40:07.628162 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_db.sqlalchemy.engines [None req-62fcbc4c-a281-4594-b0d1-afc4a8f811ee None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134755) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 26 20:40:07.671815 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.lockutils [None req-62fcbc4c-a281-4594-b0d1-afc4a8f811ee None None] Lock "cleanup_incomplete_backups_134311" "released" by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: held 0.261s {{(pid=134755) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:400}} Mar 26 20:40:07.677812 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_db.sqlalchemy.engines [None req-62fcbc4c-a281-4594-b0d1-afc4a8f811ee None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134755) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 26 20:40:07.683783 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-62fcbc4c-a281-4594-b0d1-afc4a8f811ee None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 20:40:07.704256 ncdv-master-1 cinder-backup[134755]: INFO cinder.keymgr.migration [None req-f1530833-63f5-422e-89c3-a661e517104f None None] Not migrating encryption keys because the ConfKeyManager is still in use. Mar 26 20:40:07.723045 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.service [None req-62fcbc4c-a281-4594-b0d1-afc4a8f811ee None None] Creating RPC server for service cinder-backup {{(pid=134755) start /opt/stack/cinder/cinder/service.py:231}} Mar 26 20:40:07.739010 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.service [None req-62fcbc4c-a281-4594-b0d1-afc4a8f811ee None None] Pinning object versions for RPC server serializer to 1.39 {{(pid=134755) start /opt/stack/cinder/cinder/service.py:237}} Mar 26 20:41:35.771670 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 20:41:35.772464 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 20:42:35.776381 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 20:42:35.777089 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 20:43:35.781953 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 20:43:35.782563 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 20:44:35.787115 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 20:44:35.787842 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 20:45:35.784038 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 20:45:35.784823 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 20:46:35.789484 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 20:46:35.790175 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 20:47:35.794358 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 20:47:35.794986 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 20:48:35.799856 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 20:48:35.800599 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 20:49:35.805189 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 20:49:35.806338 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 20:50:35.802069 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 20:50:35.802796 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 20:51:35.806341 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 20:51:35.806979 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 20:52:35.813349 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 20:52:35.815196 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 20:53:35.818443 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 20:53:35.819057 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 20:54:35.817169 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 20:54:35.817890 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 20:55:35.814579 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 20:55:35.815150 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 20:56:35.817782 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 20:56:35.819063 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 20:57:35.816429 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 20:57:35.817079 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 20:58:35.814334 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 20:58:35.814948 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 20:59:35.819393 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 20:59:35.820162 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:00:35.817163 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:00:35.817777 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:01:35.816702 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:01:35.817445 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:02:35.821937 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:02:35.823153 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:03:35.821842 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:03:35.823033 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:04:35.822323 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:04:35.823006 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:05:35.818675 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:05:35.819185 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:06:35.816170 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:06:35.817151 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:07:35.812660 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:07:35.813326 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:08:35.810564 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:08:35.811594 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:09:07.185542 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] Create backup started, backup: 16792d45-36c2-4f5c-9c44-f15f89a575d6 volume: ee76d1ab-6f43-41ff-ab83-be8282dc01ad. Mar 26 21:09:07.225046 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-03-26T21:09:06Z,data_timestamp=2022-03-26T21:09:06Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsV39Test-Backup-1135655536',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=16792d45-36c2-4f5c-9c44-f15f89a575d6,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='8657daeb436044378b8b124a0f9d5c15',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='9d7e99c126f3435ea890fa21c1614041',volume_id=ee76d1ab-6f43-41ff-ab83-be8282dc01ad) Mar 26 21:09:07.324516 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.utils [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.16', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:09:07.328803 ncdv-master-1 cinder-backup[134755]: INFO oslo.privsep.daemon [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] Running privsep helper: ['sudo', 'cinder-rootwrap', '/etc/cinder/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/cinder/cinder.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmps9bl8c_d/privsep.sock'] Mar 26 21:09:07.513831 ncdv-master-1 sudo[163473]: ubuntu : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf privsep-helper --config-file /etc/cinder/cinder.conf --privsep_context os_brick.privileged.default --privsep_sock_path /tmp/tmps9bl8c_d/privsep.sock Mar 26 21:09:07.548239 ncdv-master-1 sudo[163473]: pam_unix(sudo:session): session opened for user root by (uid=0) Mar 26 21:09:07.909968 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_db.sqlalchemy.engines [None req-3e2636c3-f868-49d5-9eb0-1c88dcfc918b None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134755) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 26 21:09:14.701195 ncdv-master-1 sudo[163473]: pam_unix(sudo:session): session closed for user root Mar 26 21:09:14.716471 ncdv-master-1 cinder-backup[134755]: INFO oslo.privsep.daemon [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] Spawned new privsep daemon via rootwrap Mar 26 21:09:14.717777 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] Accepted privsep connection to /tmp/tmps9bl8c_d/privsep.sock {{(pid=134755) __init__ /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:362}} Mar 26 21:09:14.719503 ncdv-master-1 cinder-backup[134755]: INFO oslo.privsep.daemon [-] privsep daemon starting Mar 26 21:09:14.720512 ncdv-master-1 cinder-backup[134755]: INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 Mar 26 21:09:14.721459 ncdv-master-1 cinder-backup[134755]: INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none Mar 26 21:09:14.721898 ncdv-master-1 cinder-backup[134755]: INFO oslo.privsep.daemon [-] privsep daemon running as pid 163609 Mar 26 21:09:14.723880 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[1575d3d1-3999-4441-87d3-3e6d6d55fd63]: (2,) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:09:15.237418 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:09:15.289336 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.051s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:09:15.289706 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[77a1fe51-717c-4d49-9817-06b9295c3111]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:92f8692cf1e\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:09:15.296537 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:09:15.321518 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 26 21:09:15.344191 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[d348cc36-9862-4f52-955a-88c96cc784fd]: [Errno 2] No such file or directory: 'systool' {{(pid=163609) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 26 21:09:15.344191 ncdv-master-1 cinder-backup[134755]: Traceback (most recent call last): Mar 26 21:09:15.344191 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 26 21:09:15.344191 ncdv-master-1 cinder-backup[134755]: ret = func(*f_args, **f_kwargs) Mar 26 21:09:15.344191 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 26 21:09:15.344191 ncdv-master-1 cinder-backup[134755]: return func(*args, **kwargs) Mar 26 21:09:15.344191 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 26 21:09:15.344191 ncdv-master-1 cinder-backup[134755]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 26 21:09:15.344191 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 26 21:09:15.344191 ncdv-master-1 cinder-backup[134755]: return putils.execute(on_execute=on_execute, Mar 26 21:09:15.344191 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 26 21:09:15.344191 ncdv-master-1 cinder-backup[134755]: obj = subprocess.Popen(cmd, Mar 26 21:09:15.344191 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 26 21:09:15.344191 ncdv-master-1 cinder-backup[134755]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 26 21:09:15.344191 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 26 21:09:15.344191 ncdv-master-1 cinder-backup[134755]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 26 21:09:15.344191 ncdv-master-1 cinder-backup[134755]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 26 21:09:15.345578 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[d348cc36-9862-4f52-955a-88c96cc784fd]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:09:15.348304 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:09:15.364340 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 26 21:09:15.364944 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[b0cb2957-3155-4ae3-abc3-b8a5bc2576c4]: [Errno 2] No such file or directory: 'systool' {{(pid=163609) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 26 21:09:15.364944 ncdv-master-1 cinder-backup[134755]: Traceback (most recent call last): Mar 26 21:09:15.364944 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 26 21:09:15.364944 ncdv-master-1 cinder-backup[134755]: ret = func(*f_args, **f_kwargs) Mar 26 21:09:15.364944 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 26 21:09:15.364944 ncdv-master-1 cinder-backup[134755]: return func(*args, **kwargs) Mar 26 21:09:15.364944 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 26 21:09:15.364944 ncdv-master-1 cinder-backup[134755]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 26 21:09:15.364944 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 26 21:09:15.364944 ncdv-master-1 cinder-backup[134755]: return putils.execute(on_execute=on_execute, Mar 26 21:09:15.364944 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 26 21:09:15.364944 ncdv-master-1 cinder-backup[134755]: obj = subprocess.Popen(cmd, Mar 26 21:09:15.364944 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 26 21:09:15.364944 ncdv-master-1 cinder-backup[134755]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 26 21:09:15.364944 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 26 21:09:15.364944 ncdv-master-1 cinder-backup[134755]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 26 21:09:15.364944 ncdv-master-1 cinder-backup[134755]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 26 21:09:15.365914 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[b0cb2957-3155-4ae3-abc3-b8a5bc2576c4]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:09:15.367315 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:09:15.411847 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.044s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:09:15.412398 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[c55b74d2-3ce6-4763-9f49-6832a30dfad8]: (4, ('/dev/sda1\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:09:15.413726 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:09:15.502557 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.088s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:09:15.503250 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[d095b335-e7e2-46d1-8718-48b9fa90167f]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:09:15.506161 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:09:15.550961 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.045s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:09:15.551992 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[476f7140-391e-4a7d-aac7-061ccb07a868]: (4, ('c6a66835-8686-c942-a93c-00f6321af5e6\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:09:15.553289 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] Running cmd (subprocess): nvme version {{(pid=134755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:09:15.641014 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] CMD "nvme version" returned: 0 in 0.088s {{(pid=134755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:09:15.677987 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134755) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 26 21:09:15.678368 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134755) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 26 21:09:15.678742 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:44f9348f-d915-47a9-a95c-57bf517d9528 dsc: {{(pid=134755) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 26 21:09:15.683959 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.utils [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] <== get_connector_properties: return (8352ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.16', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:92f8692cf1e', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c6a66835-8686-c942-a93c-00f6321af5e6', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:44f9348f-d915-47a9-a95c-57bf517d9528', 'found_dsc': ''} {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:09:16.999368 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connector [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] Factory for rbd on None {{(pid=134755) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 26 21:09:17.002481 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-ee76d1ab-6f43-41ff-ab83-be8282dc01ad', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'ee76d1ab-6f43-41ff-ab83-be8282dc01ad', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:09:17.004417 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.linuxrbd [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] opening connection to ceph cluster (timeout=-1). {{(pid=134755) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 26 21:09:17.107093 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] <== connect_volume: return (103ms) {'path': } {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:09:17.145087 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] Starting backup of volume='ee76d1ab-6f43-41ff-ab83-be8282dc01ad'. {{(pid=134755) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Mar 26 21:09:17.146267 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] Volume file is RBD: attempting optimized backup {{(pid=134755) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Mar 26 21:09:17.230023 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] Creating base image 'volume-ee76d1ab-6f43-41ff-ab83-be8282dc01ad.backup.16792d45-36c2-4f5c-9c44-f15f89a575d6' {{(pid=134755) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Mar 26 21:09:17.230582 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134755) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Mar 26 21:09:18.841456 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] Using --from-snap 'None' for incremental backup of volume ee76d1ab-6f43-41ff-ab83-be8282dc01ad. {{(pid=134755) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Mar 26 21:09:18.842100 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] Creating backup snapshot='backup.16792d45-36c2-4f5c-9c44-f15f89a575d6.snap.1648328958.8414764' {{(pid=134755) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Mar 26 21:09:19.810998 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] Performing differential transfer from 'volume-ee76d1ab-6f43-41ff-ab83-be8282dc01ad' to 'volume-ee76d1ab-6f43-41ff-ab83-be8282dc01ad.backup.16792d45-36c2-4f5c-9c44-f15f89a575d6' {{(pid=134755) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Mar 26 21:09:19.816121 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_bp1phqxf --pool volumes volumes/volume-ee76d1ab-6f43-41ff-ab83-be8282dc01ad@backup.16792d45-36c2-4f5c-9c44-f15f89a575d6.snap.1648328958.8414764 -' into... {{(pid=134755) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Mar 26 21:09:19.816895 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-ee76d1ab-6f43-41ff-ab83-be8282dc01ad.backup.16792d45-36c2-4f5c-9c44-f15f89a575d6' {{(pid=134755) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Mar 26 21:09:21.248989 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] Differential backup transfer completed in 1.4366s {{(pid=134755) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Mar 26 21:09:21.250088 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] Getting metadata type 'volume-base-metadata' {{(pid=134755) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Mar 26 21:09:21.283658 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134755) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Mar 26 21:09:21.283658 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] Getting metadata type 'volume-metadata' {{(pid=134755) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Mar 26 21:09:21.327799 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] No metadata type 'volume-metadata' available {{(pid=134755) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Mar 26 21:09:21.329533 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] Getting metadata type 'volume-glance-metadata' {{(pid=134755) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Mar 26 21:09:21.357168 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] No metadata type 'volume-glance-metadata' available {{(pid=134755) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Mar 26 21:09:21.359286 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] Backing up metadata for volume ee76d1ab-6f43-41ff-ab83-be8282dc01ad. {{(pid=134755) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Mar 26 21:09:21.485862 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] Backup '16792d45-36c2-4f5c-9c44-f15f89a575d6' of volume ee76d1ab-6f43-41ff-ab83-be8282dc01ad finished. {{(pid=134755) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Mar 26 21:09:21.488466 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-ee76d1ab-6f43-41ff-ab83-be8282dc01ad', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'ee76d1ab-6f43-41ff-ab83-be8282dc01ad', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:09:21.536444 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] <== disconnect_volume: return (46ms) None {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:09:21.907300 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-41ac08c9-7051-44d9-9508-8a4792a3e2b2 tempest-VolumesBackupsV39Test-511055746 None] Create backup finished. backup: 16792d45-36c2-4f5c-9c44-f15f89a575d6. Mar 26 21:09:23.155489 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-c28bc6a4-8cc9-4c9a-9e98-d5b84357955d tempest-VolumesBackupsV39Test-511055746 None] Delete backup started, backup: 16792d45-36c2-4f5c-9c44-f15f89a575d6. Mar 26 21:09:23.160372 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-c28bc6a4-8cc9-4c9a-9e98-d5b84357955d tempest-VolumesBackupsV39Test-511055746 None] Delete started for backup=16792d45-36c2-4f5c-9c44-f15f89a575d6 {{(pid=134755) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Mar 26 21:09:23.161040 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-c28bc6a4-8cc9-4c9a-9e98-d5b84357955d tempest-VolumesBackupsV39Test-511055746 None] Trying diff format basename='volume-ee76d1ab-6f43-41ff-ab83-be8282dc01ad.backup.16792d45-36c2-4f5c-9c44-f15f89a575d6' for backup base image of volume ee76d1ab-6f43-41ff-ab83-be8282dc01ad. {{(pid=134755) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Mar 26 21:09:23.287711 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-c28bc6a4-8cc9-4c9a-9e98-d5b84357955d tempest-VolumesBackupsV39Test-511055746 None] Looking for snapshot of backup base 'volume-ee76d1ab-6f43-41ff-ab83-be8282dc01ad.backup.16792d45-36c2-4f5c-9c44-f15f89a575d6' {{(pid=134755) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 26 21:09:23.288576 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-c28bc6a4-8cc9-4c9a-9e98-d5b84357955d tempest-VolumesBackupsV39Test-511055746 None] Found snapshot 'backup.16792d45-36c2-4f5c-9c44-f15f89a575d6.snap.1648328958.8414764' {{(pid=134755) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Mar 26 21:09:23.288669 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-c28bc6a4-8cc9-4c9a-9e98-d5b84357955d tempest-VolumesBackupsV39Test-511055746 None] Deleting backup snapshot='backup.16792d45-36c2-4f5c-9c44-f15f89a575d6.snap.1648328958.8414764' {{(pid=134755) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Mar 26 21:09:23.978690 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.drivers.ceph [None req-c28bc6a4-8cc9-4c9a-9e98-d5b84357955d tempest-VolumesBackupsV39Test-511055746 None] Deleting backup base image='volume-ee76d1ab-6f43-41ff-ab83-be8282dc01ad.backup.16792d45-36c2-4f5c-9c44-f15f89a575d6' of volume ee76d1ab-6f43-41ff-ab83-be8282dc01ad. Mar 26 21:09:24.376695 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-c28bc6a4-8cc9-4c9a-9e98-d5b84357955d tempest-VolumesBackupsV39Test-511055746 None] Base backup image='volume-ee76d1ab-6f43-41ff-ab83-be8282dc01ad.backup.16792d45-36c2-4f5c-9c44-f15f89a575d6' of volume ee76d1ab-6f43-41ff-ab83-be8282dc01ad deleted. {{(pid=134755) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Mar 26 21:09:24.454771 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-c28bc6a4-8cc9-4c9a-9e98-d5b84357955d tempest-VolumesBackupsV39Test-511055746 None] Delete of backup '16792d45-36c2-4f5c-9c44-f15f89a575d6' for volume 'ee76d1ab-6f43-41ff-ab83-be8282dc01ad' finished. {{(pid=134755) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Mar 26 21:09:24.462006 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_db.sqlalchemy.engines [None req-c28bc6a4-8cc9-4c9a-9e98-d5b84357955d tempest-VolumesBackupsV39Test-511055746 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134755) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 26 21:09:24.529502 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.quota [None req-c28bc6a4-8cc9-4c9a-9e98-d5b84357955d tempest-VolumesBackupsV39Test-511055746 None] Created reservations ['a410f310-555d-48ca-9a6f-1ff448c76b3a', '5a28270d-a5af-46fb-9dc6-50449b67c30b'] {{(pid=134755) reserve /opt/stack/cinder/cinder/quota.py:755}} Mar 26 21:09:24.599529 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-c28bc6a4-8cc9-4c9a-9e98-d5b84357955d tempest-VolumesBackupsV39Test-511055746 None] Delete backup finished, backup 16792d45-36c2-4f5c-9c44-f15f89a575d6 deleted. Mar 26 21:09:57.946672 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] Create backup started, backup: ff7f4420-2114-4db6-bf7c-74de9ffea3d5 volume: a1a81c96-3fa3-4167-be2b-3237aa8faf35 snapshot: 25988537-311e-45e1-b94f-782168ebae7b. Mar 26 21:09:57.961654 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-03-26T21:09:57Z,data_timestamp=2022-03-26T21:09:55Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Backup-764574221',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=ff7f4420-2114-4db6-bf7c-74de9ffea3d5,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='52b4455ec1a2447ebb1f737c0d97fc83',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id='25988537-311e-45e1-b94f-782168ebae7b',status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='54a09bedd0ad4328b1e687fc5f7e3967',volume_id=a1a81c96-3fa3-4167-be2b-3237aa8faf35) Mar 26 21:09:58.650646 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.utils [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.16', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:09:58.652963 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:09:58.675290 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.022s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:09:58.675780 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[2adca3b5-9f25-48fe-80a3-4dbc5c86abe3]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:92f8692cf1e\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:09:58.688037 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:09:58.688037 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 26 21:09:58.688037 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[27775158-b8f8-4c8c-9b80-f8dec6210211]: [Errno 2] No such file or directory: 'systool' {{(pid=163609) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 26 21:09:58.688037 ncdv-master-1 cinder-backup[134755]: Traceback (most recent call last): Mar 26 21:09:58.688037 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 26 21:09:58.688037 ncdv-master-1 cinder-backup[134755]: ret = func(*f_args, **f_kwargs) Mar 26 21:09:58.688037 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 26 21:09:58.688037 ncdv-master-1 cinder-backup[134755]: return func(*args, **kwargs) Mar 26 21:09:58.688037 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 26 21:09:58.688037 ncdv-master-1 cinder-backup[134755]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 26 21:09:58.688037 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 26 21:09:58.688037 ncdv-master-1 cinder-backup[134755]: return putils.execute(on_execute=on_execute, Mar 26 21:09:58.688037 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 26 21:09:58.688037 ncdv-master-1 cinder-backup[134755]: obj = subprocess.Popen(cmd, Mar 26 21:09:58.688037 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 26 21:09:58.688037 ncdv-master-1 cinder-backup[134755]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 26 21:09:58.688037 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 26 21:09:58.688037 ncdv-master-1 cinder-backup[134755]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 26 21:09:58.688037 ncdv-master-1 cinder-backup[134755]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 26 21:09:58.689436 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[27775158-b8f8-4c8c-9b80-f8dec6210211]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:09:58.689436 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:09:58.714241 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 26 21:09:58.714241 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[7d798a5d-9cfc-463d-93fc-cf69f2be4493]: [Errno 2] No such file or directory: 'systool' {{(pid=163609) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 26 21:09:58.714241 ncdv-master-1 cinder-backup[134755]: Traceback (most recent call last): Mar 26 21:09:58.714241 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 26 21:09:58.714241 ncdv-master-1 cinder-backup[134755]: ret = func(*f_args, **f_kwargs) Mar 26 21:09:58.714241 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 26 21:09:58.714241 ncdv-master-1 cinder-backup[134755]: return func(*args, **kwargs) Mar 26 21:09:58.714241 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 26 21:09:58.714241 ncdv-master-1 cinder-backup[134755]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 26 21:09:58.714241 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 26 21:09:58.714241 ncdv-master-1 cinder-backup[134755]: return putils.execute(on_execute=on_execute, Mar 26 21:09:58.714241 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 26 21:09:58.714241 ncdv-master-1 cinder-backup[134755]: obj = subprocess.Popen(cmd, Mar 26 21:09:58.714241 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 26 21:09:58.714241 ncdv-master-1 cinder-backup[134755]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 26 21:09:58.714241 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 26 21:09:58.714241 ncdv-master-1 cinder-backup[134755]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 26 21:09:58.714241 ncdv-master-1 cinder-backup[134755]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 26 21:09:58.714241 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[7d798a5d-9cfc-463d-93fc-cf69f2be4493]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:09:58.720071 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:09:58.751153 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.035s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:09:58.752484 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[2d1e79d7-303c-423d-8842-311499399f96]: (4, ('/dev/sda1\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:09:58.754112 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:09:58.793631 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.039s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:09:58.793984 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[8ef6f828-cc72-4391-afe6-6266d0be5f6d]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:09:58.798597 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:09:58.809641 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.013s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:09:58.809946 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[7aee45d0-eb40-4090-846f-3ac81370015c]: (4, ('c6a66835-8686-c942-a93c-00f6321af5e6\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:09:58.810917 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] Running cmd (subprocess): nvme version {{(pid=134755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:09:58.859898 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] CMD "nvme version" returned: 0 in 0.048s {{(pid=134755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:09:58.862221 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134755) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 26 21:09:58.862494 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134755) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 26 21:09:58.864623 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:44f9348f-d915-47a9-a95c-57bf517d9528 dsc: {{(pid=134755) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 26 21:09:58.865559 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.utils [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] <== get_connector_properties: return (214ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.16', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:92f8692cf1e', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c6a66835-8686-c942-a93c-00f6321af5e6', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:44f9348f-d915-47a9-a95c-57bf517d9528', 'found_dsc': ''} {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:09:59.790444 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connector [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] Factory for rbd on None {{(pid=134755) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 26 21:09:59.791886 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-9c60118f-0692-48ef-8ed1-39fa11bbbe87', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '9c60118f-0692-48ef-8ed1-39fa11bbbe87', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:09:59.792981 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.linuxrbd [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] opening connection to ceph cluster (timeout=-1). {{(pid=134755) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 26 21:09:59.895625 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] <== connect_volume: return (103ms) {'path': } {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:09:59.903958 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_db.sqlalchemy.engines [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134755) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 26 21:09:59.943384 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] Starting backup of volume='a1a81c96-3fa3-4167-be2b-3237aa8faf35'. {{(pid=134755) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Mar 26 21:10:00.039269 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134755) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Mar 26 21:10:00.039682 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] Creating backup base image='volume-a1a81c96-3fa3-4167-be2b-3237aa8faf35.backup.base' for volume a1a81c96-3fa3-4167-be2b-3237aa8faf35. {{(pid=134755) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:830}} Mar 26 21:10:00.105067 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] Copying data from volume a1a81c96-3fa3-4167-be2b-3237aa8faf35. {{(pid=134755) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:842}} Mar 26 21:10:00.153347 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] Transferring data between 'volume-a1a81c96-3fa3-4167-be2b-3237aa8faf35' and 'volume-a1a81c96-3fa3-4167-be2b-3237aa8faf35.backup.base' {{(pid=134755) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:399}} Mar 26 21:10:00.153844 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] 8 chunks of 134217728 bytes to be transferred {{(pid=134755) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:403}} Mar 26 21:10:02.201591 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] Transferred chunk 1 of 8 (64031K/s) {{(pid=134755) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Mar 26 21:10:03.798721 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] Transferred chunk 2 of 8 (82178K/s) {{(pid=134755) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Mar 26 21:10:05.518484 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] Transferred chunk 3 of 8 (76282K/s) {{(pid=134755) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Mar 26 21:10:07.220172 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] Transferred chunk 4 of 8 (77048K/s) {{(pid=134755) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Mar 26 21:10:08.843695 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] Transferred chunk 5 of 8 (80795K/s) {{(pid=134755) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Mar 26 21:10:10.411448 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] Transferred chunk 6 of 8 (83827K/s) {{(pid=134755) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Mar 26 21:10:12.087000 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] Transferred chunk 7 of 8 (78255K/s) {{(pid=134755) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Mar 26 21:10:13.802440 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] Transferred chunk 8 of 8 (76472K/s) {{(pid=134755) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Mar 26 21:10:13.872748 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] Getting metadata type 'volume-base-metadata' {{(pid=134755) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Mar 26 21:10:13.891000 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134755) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Mar 26 21:10:13.891783 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] Getting metadata type 'volume-metadata' {{(pid=134755) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Mar 26 21:10:13.900458 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] No metadata type 'volume-metadata' available {{(pid=134755) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Mar 26 21:10:13.901527 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] Getting metadata type 'volume-glance-metadata' {{(pid=134755) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Mar 26 21:10:13.914349 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] No metadata type 'volume-glance-metadata' available {{(pid=134755) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Mar 26 21:10:13.915349 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] Backing up metadata for volume a1a81c96-3fa3-4167-be2b-3237aa8faf35. {{(pid=134755) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Mar 26 21:10:13.975436 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] Backup 'ff7f4420-2114-4db6-bf7c-74de9ffea3d5' of volume a1a81c96-3fa3-4167-be2b-3237aa8faf35 finished. {{(pid=134755) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Mar 26 21:10:13.977035 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-9c60118f-0692-48ef-8ed1-39fa11bbbe87', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '9c60118f-0692-48ef-8ed1-39fa11bbbe87', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:10:14.007564 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] <== disconnect_volume: return (29ms) None {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:10:14.148091 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-8c8edda3-6d19-4257-a2da-00e15d65adac tempest-VolumesSnapshotTestJSON-1730121226 None] Create backup finished. backup: ff7f4420-2114-4db6-bf7c-74de9ffea3d5. Mar 26 21:10:14.781384 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-8a22e252-6d41-4a5f-9d97-0d916d14681e tempest-VolumesSnapshotTestJSON-1730121226 None] Delete backup started, backup: ff7f4420-2114-4db6-bf7c-74de9ffea3d5. Mar 26 21:10:14.788984 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-8a22e252-6d41-4a5f-9d97-0d916d14681e tempest-VolumesSnapshotTestJSON-1730121226 None] Delete started for backup=ff7f4420-2114-4db6-bf7c-74de9ffea3d5 {{(pid=134755) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Mar 26 21:10:14.789655 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-8a22e252-6d41-4a5f-9d97-0d916d14681e tempest-VolumesSnapshotTestJSON-1730121226 None] Trying diff format basename='volume-a1a81c96-3fa3-4167-be2b-3237aa8faf35.backup.ff7f4420-2114-4db6-bf7c-74de9ffea3d5' for backup base image of volume a1a81c96-3fa3-4167-be2b-3237aa8faf35. {{(pid=134755) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Mar 26 21:10:14.843660 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-8a22e252-6d41-4a5f-9d97-0d916d14681e tempest-VolumesSnapshotTestJSON-1730121226 None] Image 'volume-a1a81c96-3fa3-4167-be2b-3237aa8faf35.backup.ff7f4420-2114-4db6-bf7c-74de9ffea3d5' not found - trying diff format name {{(pid=134755) _rbd_image_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:661}} Mar 26 21:10:14.873773 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-8a22e252-6d41-4a5f-9d97-0d916d14681e tempest-VolumesSnapshotTestJSON-1730121226 None] Looking for snapshot of backup base 'volume-a1a81c96-3fa3-4167-be2b-3237aa8faf35.backup.base' {{(pid=134755) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 26 21:10:14.875056 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-8a22e252-6d41-4a5f-9d97-0d916d14681e tempest-VolumesSnapshotTestJSON-1730121226 None] Backup base 'volume-a1a81c96-3fa3-4167-be2b-3237aa8faf35.backup.base' has no snapshots {{(pid=134755) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:908}} Mar 26 21:10:14.875656 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-8a22e252-6d41-4a5f-9d97-0d916d14681e tempest-VolumesSnapshotTestJSON-1730121226 None] No backup snapshot to delete {{(pid=134755) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:476}} Mar 26 21:10:14.884532 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.drivers.ceph [None req-8a22e252-6d41-4a5f-9d97-0d916d14681e tempest-VolumesSnapshotTestJSON-1730121226 None] Deleting backup base image='volume-a1a81c96-3fa3-4167-be2b-3237aa8faf35.backup.base' of volume a1a81c96-3fa3-4167-be2b-3237aa8faf35. Mar 26 21:10:16.034640 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-8a22e252-6d41-4a5f-9d97-0d916d14681e tempest-VolumesSnapshotTestJSON-1730121226 None] Base backup image='volume-a1a81c96-3fa3-4167-be2b-3237aa8faf35.backup.base' of volume a1a81c96-3fa3-4167-be2b-3237aa8faf35 deleted. {{(pid=134755) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Mar 26 21:10:16.133628 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-8a22e252-6d41-4a5f-9d97-0d916d14681e tempest-VolumesSnapshotTestJSON-1730121226 None] Delete of backup 'ff7f4420-2114-4db6-bf7c-74de9ffea3d5' for volume 'a1a81c96-3fa3-4167-be2b-3237aa8faf35' finished. {{(pid=134755) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Mar 26 21:10:16.140934 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_db.sqlalchemy.engines [None req-8a22e252-6d41-4a5f-9d97-0d916d14681e tempest-VolumesSnapshotTestJSON-1730121226 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134755) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 26 21:10:16.223578 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.quota [None req-8a22e252-6d41-4a5f-9d97-0d916d14681e tempest-VolumesSnapshotTestJSON-1730121226 None] Created reservations ['fe4933c8-806c-45fd-9b9a-52b6b90d1455', '01825f00-1021-4abb-bce2-0fd88258f31b'] {{(pid=134755) reserve /opt/stack/cinder/cinder/quota.py:755}} Mar 26 21:10:16.306331 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-8a22e252-6d41-4a5f-9d97-0d916d14681e tempest-VolumesSnapshotTestJSON-1730121226 None] Delete backup finished, backup ff7f4420-2114-4db6-bf7c-74de9ffea3d5 deleted. Mar 26 21:10:35.813432 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:10:35.813432 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:11:31.540585 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] Create backup started, backup: 516180f7-6a66-4457-8a3c-08aefebb3c78 volume: f15536f9-5d63-4bc4-8237-4141ad30b570. Mar 26 21:11:31.550535 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-03-26T21:11:31Z,data_timestamp=2022-03-26T21:11:31Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsTest-Backup-1782643102',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=516180f7-6a66-4457-8a3c-08aefebb3c78,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='13edcc6f41ce42ff8b88edb21a3ae4ec',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='579df559018e45808c3886ac22ecc165',volume_id=f15536f9-5d63-4bc4-8237-4141ad30b570) Mar 26 21:11:31.600398 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.utils [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.16', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:11:31.604691 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:11:31.641561 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.039s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:11:31.642121 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[72ee07c4-7fbb-44f7-a49a-ead266b6c255]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:92f8692cf1e\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:11:31.643533 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:11:31.650883 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 26 21:11:31.651453 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[88e3c930-8cb4-4b49-951d-2b06ca9eb162]: [Errno 2] No such file or directory: 'systool' {{(pid=163609) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 26 21:11:31.651453 ncdv-master-1 cinder-backup[134755]: Traceback (most recent call last): Mar 26 21:11:31.651453 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 26 21:11:31.651453 ncdv-master-1 cinder-backup[134755]: ret = func(*f_args, **f_kwargs) Mar 26 21:11:31.651453 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 26 21:11:31.651453 ncdv-master-1 cinder-backup[134755]: return func(*args, **kwargs) Mar 26 21:11:31.651453 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 26 21:11:31.651453 ncdv-master-1 cinder-backup[134755]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 26 21:11:31.651453 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 26 21:11:31.651453 ncdv-master-1 cinder-backup[134755]: return putils.execute(on_execute=on_execute, Mar 26 21:11:31.651453 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 26 21:11:31.651453 ncdv-master-1 cinder-backup[134755]: obj = subprocess.Popen(cmd, Mar 26 21:11:31.651453 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 26 21:11:31.651453 ncdv-master-1 cinder-backup[134755]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 26 21:11:31.651453 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 26 21:11:31.651453 ncdv-master-1 cinder-backup[134755]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 26 21:11:31.651453 ncdv-master-1 cinder-backup[134755]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 26 21:11:31.653267 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[88e3c930-8cb4-4b49-951d-2b06ca9eb162]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:11:31.653829 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:11:31.661116 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 26 21:11:31.661684 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[aa7135b6-a8b6-4a56-aa28-796b4986dabd]: [Errno 2] No such file or directory: 'systool' {{(pid=163609) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 26 21:11:31.661684 ncdv-master-1 cinder-backup[134755]: Traceback (most recent call last): Mar 26 21:11:31.661684 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 26 21:11:31.661684 ncdv-master-1 cinder-backup[134755]: ret = func(*f_args, **f_kwargs) Mar 26 21:11:31.661684 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 26 21:11:31.661684 ncdv-master-1 cinder-backup[134755]: return func(*args, **kwargs) Mar 26 21:11:31.661684 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 26 21:11:31.661684 ncdv-master-1 cinder-backup[134755]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 26 21:11:31.661684 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 26 21:11:31.661684 ncdv-master-1 cinder-backup[134755]: return putils.execute(on_execute=on_execute, Mar 26 21:11:31.661684 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 26 21:11:31.661684 ncdv-master-1 cinder-backup[134755]: obj = subprocess.Popen(cmd, Mar 26 21:11:31.661684 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 26 21:11:31.661684 ncdv-master-1 cinder-backup[134755]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 26 21:11:31.661684 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 26 21:11:31.661684 ncdv-master-1 cinder-backup[134755]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 26 21:11:31.661684 ncdv-master-1 cinder-backup[134755]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 26 21:11:31.662888 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[aa7135b6-a8b6-4a56-aa28-796b4986dabd]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:11:31.664673 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:11:31.686597 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.022s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:11:31.687243 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[e1bd1fab-db17-4b11-96e8-368d0ef9ec51]: (4, ('/dev/sda1\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:11:31.691621 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:11:31.708198 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.019s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:11:31.708514 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[70cd8e03-ac30-4fc0-b310-a52c3ce89497]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:11:31.709732 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:11:31.718791 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.009s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:11:31.720307 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[3a94bca7-afab-44b8-ac90-0090d8ef298a]: (4, ('c6a66835-8686-c942-a93c-00f6321af5e6\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:11:31.721314 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] Running cmd (subprocess): nvme version {{(pid=134755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:11:31.766410 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] CMD "nvme version" returned: 0 in 0.044s {{(pid=134755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:11:31.769232 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134755) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 26 21:11:31.770153 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134755) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 26 21:11:31.771175 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:44f9348f-d915-47a9-a95c-57bf517d9528 dsc: {{(pid=134755) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 26 21:11:31.774386 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.utils [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] <== get_connector_properties: return (170ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.16', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:92f8692cf1e', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c6a66835-8686-c942-a93c-00f6321af5e6', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:44f9348f-d915-47a9-a95c-57bf517d9528', 'found_dsc': ''} {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:11:32.806839 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connector [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] Factory for rbd on None {{(pid=134755) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 26 21:11:32.808245 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-f15536f9-5d63-4bc4-8237-4141ad30b570', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'f15536f9-5d63-4bc4-8237-4141ad30b570', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:11:32.809341 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.linuxrbd [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] opening connection to ceph cluster (timeout=-1). {{(pid=134755) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 26 21:11:32.866286 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] <== connect_volume: return (57ms) {'path': } {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:11:32.880485 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_db.sqlalchemy.engines [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134755) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 26 21:11:32.914745 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] Starting backup of volume='f15536f9-5d63-4bc4-8237-4141ad30b570'. {{(pid=134755) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Mar 26 21:11:32.915640 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] Volume file is RBD: attempting optimized backup {{(pid=134755) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Mar 26 21:11:32.966084 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] Creating base image 'volume-f15536f9-5d63-4bc4-8237-4141ad30b570.backup.516180f7-6a66-4457-8a3c-08aefebb3c78' {{(pid=134755) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Mar 26 21:11:32.966586 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134755) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Mar 26 21:11:33.032139 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] Using --from-snap 'None' for incremental backup of volume f15536f9-5d63-4bc4-8237-4141ad30b570. {{(pid=134755) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Mar 26 21:11:33.032583 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] Creating backup snapshot='backup.516180f7-6a66-4457-8a3c-08aefebb3c78.snap.1648329093.0321367' {{(pid=134755) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Mar 26 21:11:33.658356 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] Performing differential transfer from 'volume-f15536f9-5d63-4bc4-8237-4141ad30b570' to 'volume-f15536f9-5d63-4bc4-8237-4141ad30b570.backup.516180f7-6a66-4457-8a3c-08aefebb3c78' {{(pid=134755) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Mar 26 21:11:33.659644 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_h9d5f2e4 --pool volumes volumes/volume-f15536f9-5d63-4bc4-8237-4141ad30b570@backup.516180f7-6a66-4457-8a3c-08aefebb3c78.snap.1648329093.0321367 -' into... {{(pid=134755) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Mar 26 21:11:33.660426 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-f15536f9-5d63-4bc4-8237-4141ad30b570.backup.516180f7-6a66-4457-8a3c-08aefebb3c78' {{(pid=134755) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Mar 26 21:11:34.727749 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] Differential backup transfer completed in 1.0686s {{(pid=134755) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Mar 26 21:11:34.728458 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] Getting metadata type 'volume-base-metadata' {{(pid=134755) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Mar 26 21:11:34.764175 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134755) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Mar 26 21:11:34.764997 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] Getting metadata type 'volume-metadata' {{(pid=134755) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Mar 26 21:11:34.775027 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] No metadata type 'volume-metadata' available {{(pid=134755) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Mar 26 21:11:34.776146 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] Getting metadata type 'volume-glance-metadata' {{(pid=134755) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Mar 26 21:11:34.785594 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] No metadata type 'volume-glance-metadata' available {{(pid=134755) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Mar 26 21:11:34.791927 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] Backing up metadata for volume f15536f9-5d63-4bc4-8237-4141ad30b570. {{(pid=134755) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Mar 26 21:11:34.868299 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] Backup '516180f7-6a66-4457-8a3c-08aefebb3c78' of volume f15536f9-5d63-4bc4-8237-4141ad30b570 finished. {{(pid=134755) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Mar 26 21:11:34.869662 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-f15536f9-5d63-4bc4-8237-4141ad30b570', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'f15536f9-5d63-4bc4-8237-4141ad30b570', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:11:34.894926 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] <== disconnect_volume: return (24ms) None {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:11:35.011977 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-94dfa0c1-a04b-4949-b775-7d58bc80cd55 tempest-VolumesBackupsTest-1678098904 None] Create backup finished. backup: 516180f7-6a66-4457-8a3c-08aefebb3c78. Mar 26 21:11:35.808526 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:11:35.809219 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:11:35.837650 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-5dd638c8-c2ef-49ce-a63e-4c4ca034d202 tempest-VolumesBackupsTest-1678098904 None] Delete backup started, backup: 516180f7-6a66-4457-8a3c-08aefebb3c78. Mar 26 21:11:35.840928 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-5dd638c8-c2ef-49ce-a63e-4c4ca034d202 tempest-VolumesBackupsTest-1678098904 None] Delete started for backup=516180f7-6a66-4457-8a3c-08aefebb3c78 {{(pid=134755) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Mar 26 21:11:35.841543 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-5dd638c8-c2ef-49ce-a63e-4c4ca034d202 tempest-VolumesBackupsTest-1678098904 None] Trying diff format basename='volume-f15536f9-5d63-4bc4-8237-4141ad30b570.backup.516180f7-6a66-4457-8a3c-08aefebb3c78' for backup base image of volume f15536f9-5d63-4bc4-8237-4141ad30b570. {{(pid=134755) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Mar 26 21:11:35.923761 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-5dd638c8-c2ef-49ce-a63e-4c4ca034d202 tempest-VolumesBackupsTest-1678098904 None] Looking for snapshot of backup base 'volume-f15536f9-5d63-4bc4-8237-4141ad30b570.backup.516180f7-6a66-4457-8a3c-08aefebb3c78' {{(pid=134755) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 26 21:11:35.924149 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-5dd638c8-c2ef-49ce-a63e-4c4ca034d202 tempest-VolumesBackupsTest-1678098904 None] Found snapshot 'backup.516180f7-6a66-4457-8a3c-08aefebb3c78.snap.1648329093.0321367' {{(pid=134755) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Mar 26 21:11:35.924520 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-5dd638c8-c2ef-49ce-a63e-4c4ca034d202 tempest-VolumesBackupsTest-1678098904 None] Deleting backup snapshot='backup.516180f7-6a66-4457-8a3c-08aefebb3c78.snap.1648329093.0321367' {{(pid=134755) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Mar 26 21:11:36.759958 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.drivers.ceph [None req-5dd638c8-c2ef-49ce-a63e-4c4ca034d202 tempest-VolumesBackupsTest-1678098904 None] Deleting backup base image='volume-f15536f9-5d63-4bc4-8237-4141ad30b570.backup.516180f7-6a66-4457-8a3c-08aefebb3c78' of volume f15536f9-5d63-4bc4-8237-4141ad30b570. Mar 26 21:11:37.196099 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-5dd638c8-c2ef-49ce-a63e-4c4ca034d202 tempest-VolumesBackupsTest-1678098904 None] Base backup image='volume-f15536f9-5d63-4bc4-8237-4141ad30b570.backup.516180f7-6a66-4457-8a3c-08aefebb3c78' of volume f15536f9-5d63-4bc4-8237-4141ad30b570 deleted. {{(pid=134755) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Mar 26 21:11:37.264724 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-5dd638c8-c2ef-49ce-a63e-4c4ca034d202 tempest-VolumesBackupsTest-1678098904 None] Delete of backup '516180f7-6a66-4457-8a3c-08aefebb3c78' for volume 'f15536f9-5d63-4bc4-8237-4141ad30b570' finished. {{(pid=134755) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Mar 26 21:11:37.274855 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_db.sqlalchemy.engines [None req-5dd638c8-c2ef-49ce-a63e-4c4ca034d202 tempest-VolumesBackupsTest-1678098904 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134755) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 26 21:11:37.324608 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.quota [None req-5dd638c8-c2ef-49ce-a63e-4c4ca034d202 tempest-VolumesBackupsTest-1678098904 None] Created reservations ['febb3dff-c496-4ac4-b584-4b8aaef51bec', '9f32a631-099a-4afe-bf0e-50ed67be1eeb'] {{(pid=134755) reserve /opt/stack/cinder/cinder/quota.py:755}} Mar 26 21:11:37.386661 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-5dd638c8-c2ef-49ce-a63e-4c4ca034d202 tempest-VolumesBackupsTest-1678098904 None] Delete backup finished, backup 516180f7-6a66-4457-8a3c-08aefebb3c78 deleted. Mar 26 21:11:47.439388 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] Create backup started, backup: 302ef0ea-3207-4ba4-a502-e34966251e52 volume: 50b33a17-1e04-41a5-bced-d5d91836b5d6. Mar 26 21:11:47.448329 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-03-26T21:11:47Z,data_timestamp=2022-03-26T21:11:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsTest-Backup-1480593260',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=302ef0ea-3207-4ba4-a502-e34966251e52,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='13edcc6f41ce42ff8b88edb21a3ae4ec',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='579df559018e45808c3886ac22ecc165',volume_id=50b33a17-1e04-41a5-bced-d5d91836b5d6) Mar 26 21:11:47.512271 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.utils [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.16', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:11:47.516362 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:11:47.538253 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.021s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:11:47.538744 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[fda4a675-2ac5-4229-bb4e-af60bea160af]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:92f8692cf1e\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:11:47.544240 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:11:47.562511 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 26 21:11:47.562858 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[a05f8138-d057-478c-ab42-68ea2a3b58ce]: [Errno 2] No such file or directory: 'systool' {{(pid=163609) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 26 21:11:47.562858 ncdv-master-1 cinder-backup[134755]: Traceback (most recent call last): Mar 26 21:11:47.562858 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 26 21:11:47.562858 ncdv-master-1 cinder-backup[134755]: ret = func(*f_args, **f_kwargs) Mar 26 21:11:47.562858 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 26 21:11:47.562858 ncdv-master-1 cinder-backup[134755]: return func(*args, **kwargs) Mar 26 21:11:47.562858 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 26 21:11:47.562858 ncdv-master-1 cinder-backup[134755]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 26 21:11:47.562858 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 26 21:11:47.562858 ncdv-master-1 cinder-backup[134755]: return putils.execute(on_execute=on_execute, Mar 26 21:11:47.562858 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 26 21:11:47.562858 ncdv-master-1 cinder-backup[134755]: obj = subprocess.Popen(cmd, Mar 26 21:11:47.562858 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 26 21:11:47.562858 ncdv-master-1 cinder-backup[134755]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 26 21:11:47.562858 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 26 21:11:47.562858 ncdv-master-1 cinder-backup[134755]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 26 21:11:47.562858 ncdv-master-1 cinder-backup[134755]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 26 21:11:47.564400 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[a05f8138-d057-478c-ab42-68ea2a3b58ce]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:11:47.572091 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:11:47.575757 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 26 21:11:47.577600 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[771a5a48-76f3-48e7-810c-4b5993146ae9]: [Errno 2] No such file or directory: 'systool' {{(pid=163609) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 26 21:11:47.577600 ncdv-master-1 cinder-backup[134755]: Traceback (most recent call last): Mar 26 21:11:47.577600 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 26 21:11:47.577600 ncdv-master-1 cinder-backup[134755]: ret = func(*f_args, **f_kwargs) Mar 26 21:11:47.577600 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 26 21:11:47.577600 ncdv-master-1 cinder-backup[134755]: return func(*args, **kwargs) Mar 26 21:11:47.577600 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 26 21:11:47.577600 ncdv-master-1 cinder-backup[134755]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 26 21:11:47.577600 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 26 21:11:47.577600 ncdv-master-1 cinder-backup[134755]: return putils.execute(on_execute=on_execute, Mar 26 21:11:47.577600 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 26 21:11:47.577600 ncdv-master-1 cinder-backup[134755]: obj = subprocess.Popen(cmd, Mar 26 21:11:47.577600 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 26 21:11:47.577600 ncdv-master-1 cinder-backup[134755]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 26 21:11:47.577600 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 26 21:11:47.577600 ncdv-master-1 cinder-backup[134755]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 26 21:11:47.577600 ncdv-master-1 cinder-backup[134755]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 26 21:11:47.578932 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[771a5a48-76f3-48e7-810c-4b5993146ae9]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:11:47.581402 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:11:47.630020 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.029s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:11:47.630414 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[17ce4d9f-e678-4e86-b79b-94139f52c2cc]: (4, ('/dev/sda1\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:11:47.632470 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:11:47.670448 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.038s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:11:47.671366 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[f12866c5-666d-4819-990a-97f9164d8ff7]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:11:47.672941 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:11:47.684002 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.011s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:11:47.684514 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[3cd75170-da6b-48f1-a129-b08058785e44]: (4, ('c6a66835-8686-c942-a93c-00f6321af5e6\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:11:47.685780 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] Running cmd (subprocess): nvme version {{(pid=134755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:11:47.709849 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] CMD "nvme version" returned: 0 in 0.022s {{(pid=134755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:11:47.712416 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134755) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 26 21:11:47.713068 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134755) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 26 21:11:47.714258 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:44f9348f-d915-47a9-a95c-57bf517d9528 dsc: {{(pid=134755) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 26 21:11:47.715226 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.utils [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] <== get_connector_properties: return (202ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.16', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:92f8692cf1e', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c6a66835-8686-c942-a93c-00f6321af5e6', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:44f9348f-d915-47a9-a95c-57bf517d9528', 'found_dsc': ''} {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:11:47.908722 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_db.sqlalchemy.engines [None req-3e2636c3-f868-49d5-9eb0-1c88dcfc918b None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134755) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 26 21:11:48.547702 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connector [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] Factory for rbd on None {{(pid=134755) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 26 21:11:48.550406 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-50b33a17-1e04-41a5-bced-d5d91836b5d6', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '50b33a17-1e04-41a5-bced-d5d91836b5d6', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:11:48.551979 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.linuxrbd [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] opening connection to ceph cluster (timeout=-1). {{(pid=134755) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 26 21:11:48.675130 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] <== connect_volume: return (123ms) {'path': } {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:11:48.713907 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] Starting backup of volume='50b33a17-1e04-41a5-bced-d5d91836b5d6'. {{(pid=134755) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Mar 26 21:11:48.715106 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] Volume file is RBD: attempting optimized backup {{(pid=134755) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Mar 26 21:11:48.756079 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] Creating base image 'volume-50b33a17-1e04-41a5-bced-d5d91836b5d6.backup.302ef0ea-3207-4ba4-a502-e34966251e52' {{(pid=134755) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Mar 26 21:11:48.756493 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134755) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Mar 26 21:11:48.797425 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] Using --from-snap 'None' for incremental backup of volume 50b33a17-1e04-41a5-bced-d5d91836b5d6. {{(pid=134755) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Mar 26 21:11:48.797826 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] Creating backup snapshot='backup.302ef0ea-3207-4ba4-a502-e34966251e52.snap.1648329108.797369' {{(pid=134755) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Mar 26 21:11:49.604906 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] Performing differential transfer from 'volume-50b33a17-1e04-41a5-bced-d5d91836b5d6' to 'volume-50b33a17-1e04-41a5-bced-d5d91836b5d6.backup.302ef0ea-3207-4ba4-a502-e34966251e52' {{(pid=134755) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Mar 26 21:11:49.607599 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_6jh79__o --pool volumes volumes/volume-50b33a17-1e04-41a5-bced-d5d91836b5d6@backup.302ef0ea-3207-4ba4-a502-e34966251e52.snap.1648329108.797369 -' into... {{(pid=134755) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Mar 26 21:11:49.608025 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-50b33a17-1e04-41a5-bced-d5d91836b5d6.backup.302ef0ea-3207-4ba4-a502-e34966251e52' {{(pid=134755) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Mar 26 21:11:57.169896 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] Differential backup transfer completed in 7.5674s {{(pid=134755) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Mar 26 21:11:57.170637 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] Getting metadata type 'volume-base-metadata' {{(pid=134755) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Mar 26 21:11:57.193823 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134755) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Mar 26 21:11:57.194245 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] Getting metadata type 'volume-metadata' {{(pid=134755) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Mar 26 21:11:57.202256 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] No metadata type 'volume-metadata' available {{(pid=134755) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Mar 26 21:11:57.202561 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] Getting metadata type 'volume-glance-metadata' {{(pid=134755) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Mar 26 21:11:57.211777 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=134755) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}} Mar 26 21:11:57.212892 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] Backing up metadata for volume 50b33a17-1e04-41a5-bced-d5d91836b5d6. {{(pid=134755) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Mar 26 21:11:57.290962 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] Backup '302ef0ea-3207-4ba4-a502-e34966251e52' of volume 50b33a17-1e04-41a5-bced-d5d91836b5d6 finished. {{(pid=134755) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Mar 26 21:11:57.294142 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-50b33a17-1e04-41a5-bced-d5d91836b5d6', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '50b33a17-1e04-41a5-bced-d5d91836b5d6', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:11:57.335998 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] <== disconnect_volume: return (39ms) None {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:11:57.462996 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-b7618356-f592-4a7d-88c8-f5e2a72c4606 tempest-VolumesBackupsTest-1678098904 None] Create backup finished. backup: 302ef0ea-3207-4ba4-a502-e34966251e52. Mar 26 21:11:59.265262 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] Restore backup started, backup: 302ef0ea-3207-4ba4-a502-e34966251e52 volume: 33e5349f-5622-44a1-843c-247b4e4d5d29. Mar 26 21:11:59.305239 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.utils [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.16', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:11:59.312575 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:11:59.331215 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.022s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:11:59.334798 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[b5d2cf57-89d6-4fc2-9660-d1f8a8b2136e]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:92f8692cf1e\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:11:59.338627 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:11:59.363156 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 26 21:11:59.364248 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[06e71536-1d18-486d-9b74-f412a9b1b8c4]: [Errno 2] No such file or directory: 'systool' {{(pid=163609) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 26 21:11:59.364248 ncdv-master-1 cinder-backup[134755]: Traceback (most recent call last): Mar 26 21:11:59.364248 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 26 21:11:59.364248 ncdv-master-1 cinder-backup[134755]: ret = func(*f_args, **f_kwargs) Mar 26 21:11:59.364248 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 26 21:11:59.364248 ncdv-master-1 cinder-backup[134755]: return func(*args, **kwargs) Mar 26 21:11:59.364248 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 26 21:11:59.364248 ncdv-master-1 cinder-backup[134755]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 26 21:11:59.364248 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 26 21:11:59.364248 ncdv-master-1 cinder-backup[134755]: return putils.execute(on_execute=on_execute, Mar 26 21:11:59.364248 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 26 21:11:59.364248 ncdv-master-1 cinder-backup[134755]: obj = subprocess.Popen(cmd, Mar 26 21:11:59.364248 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 26 21:11:59.364248 ncdv-master-1 cinder-backup[134755]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 26 21:11:59.364248 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 26 21:11:59.364248 ncdv-master-1 cinder-backup[134755]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 26 21:11:59.364248 ncdv-master-1 cinder-backup[134755]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 26 21:11:59.365695 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[06e71536-1d18-486d-9b74-f412a9b1b8c4]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:11:59.366978 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:11:59.378745 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 26 21:11:59.382579 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[9808e162-6dbd-4a68-b65c-86471f32d14d]: [Errno 2] No such file or directory: 'systool' {{(pid=163609) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 26 21:11:59.382579 ncdv-master-1 cinder-backup[134755]: Traceback (most recent call last): Mar 26 21:11:59.382579 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 26 21:11:59.382579 ncdv-master-1 cinder-backup[134755]: ret = func(*f_args, **f_kwargs) Mar 26 21:11:59.382579 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 26 21:11:59.382579 ncdv-master-1 cinder-backup[134755]: return func(*args, **kwargs) Mar 26 21:11:59.382579 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 26 21:11:59.382579 ncdv-master-1 cinder-backup[134755]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 26 21:11:59.382579 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 26 21:11:59.382579 ncdv-master-1 cinder-backup[134755]: return putils.execute(on_execute=on_execute, Mar 26 21:11:59.382579 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 26 21:11:59.382579 ncdv-master-1 cinder-backup[134755]: obj = subprocess.Popen(cmd, Mar 26 21:11:59.382579 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 26 21:11:59.382579 ncdv-master-1 cinder-backup[134755]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 26 21:11:59.382579 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 26 21:11:59.382579 ncdv-master-1 cinder-backup[134755]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 26 21:11:59.382579 ncdv-master-1 cinder-backup[134755]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 26 21:11:59.384294 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[9808e162-6dbd-4a68-b65c-86471f32d14d]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:11:59.386997 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:11:59.405784 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.019s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:11:59.406242 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[17298911-93a3-418b-8f30-f51b638df1fc]: (4, ('/dev/sda1\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:11:59.410998 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:11:59.421089 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.012s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:11:59.421456 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[67a3df34-bdad-4888-a21b-127fcf6dcc96]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:11:59.422859 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:11:59.438414 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.015s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:11:59.439338 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[3dc40466-1d51-4969-8e87-42b1013cf879]: (4, ('c6a66835-8686-c942-a93c-00f6321af5e6\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:11:59.441330 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] Running cmd (subprocess): nvme version {{(pid=134755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:11:59.479573 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] CMD "nvme version" returned: 0 in 0.038s {{(pid=134755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:11:59.481587 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134755) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 26 21:11:59.482704 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134755) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 26 21:11:59.483256 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:44f9348f-d915-47a9-a95c-57bf517d9528 dsc: {{(pid=134755) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 26 21:11:59.484518 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.utils [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] <== get_connector_properties: return (177ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.16', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:92f8692cf1e', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c6a66835-8686-c942-a93c-00f6321af5e6', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:44f9348f-d915-47a9-a95c-57bf517d9528', 'found_dsc': ''} {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:12:00.311127 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connector [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] Factory for rbd on None {{(pid=134755) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 26 21:12:00.312589 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-33e5349f-5622-44a1-843c-247b4e4d5d29', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '33e5349f-5622-44a1-843c-247b4e4d5d29', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:12:00.314798 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.linuxrbd [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] opening connection to ceph cluster (timeout=-1). {{(pid=134755) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 26 21:12:00.402980 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] <== connect_volume: return (89ms) {'path': } {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:12:00.417548 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_db.sqlalchemy.engines [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134755) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 26 21:12:00.433894 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] Starting restore from Ceph backup=302ef0ea-3207-4ba4-a502-e34966251e52 to volume=volume-33e5349f-5622-44a1-843c-247b4e4d5d29 {{(pid=134755) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1274}} Mar 26 21:12:00.516676 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] Looking for snapshot of backup base 'volume-50b33a17-1e04-41a5-bced-d5d91836b5d6.backup.302ef0ea-3207-4ba4-a502-e34966251e52' {{(pid=134755) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 26 21:12:00.517256 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] Found snapshot 'backup.302ef0ea-3207-4ba4-a502-e34966251e52.snap.1648329108.797369' {{(pid=134755) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Mar 26 21:12:00.528963 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] Volume file is RBD. {{(pid=134755) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1184}} Mar 26 21:12:00.607294 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] Attempting differential restore. {{(pid=134755) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1234}} Mar 26 21:12:00.608021 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] Attempting incremental restore from base='volume-50b33a17-1e04-41a5-bced-d5d91836b5d6.backup.302ef0ea-3207-4ba4-a502-e34966251e52' snap='backup.302ef0ea-3207-4ba4-a502-e34966251e52.snap.1648329108.797369' {{(pid=134755) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1089}} Mar 26 21:12:00.609346 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] Performing differential transfer from 'volume-50b33a17-1e04-41a5-bced-d5d91836b5d6.backup.302ef0ea-3207-4ba4-a502-e34966251e52' to 'volume-33e5349f-5622-44a1-843c-247b4e4d5d29' {{(pid=134755) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Mar 26 21:12:00.609947 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-50b33a17-1e04-41a5-bced-d5d91836b5d6.backup.302ef0ea-3207-4ba4-a502-e34966251e52@backup.302ef0ea-3207-4ba4-a502-e34966251e52.snap.1648329108.797369 -' into... {{(pid=134755) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Mar 26 21:12:00.610517 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_w3z2nvjw --pool volumes - volumes/volume-33e5349f-5622-44a1-843c-247b4e4d5d29' {{(pid=134755) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Mar 26 21:12:06.709068 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] Restore transfer completed in 6.1001s {{(pid=134755) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1111}} Mar 26 21:12:06.711089 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=134755) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1285}} Mar 26 21:12:06.784073 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] Restoring volume base metadata {{(pid=134755) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} Mar 26 21:12:06.813759 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] No metadata of type 'volume-metadata' to restore {{(pid=134755) put /opt/stack/cinder/cinder/backup/driver.py:345}} Mar 26 21:12:06.814514 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] Restoring volume glance metadata {{(pid=134755) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}} Mar 26 21:12:07.142762 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] Restore to volume 33e5349f-5622-44a1-843c-247b4e4d5d29 finished successfully. {{(pid=134755) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1292}} Mar 26 21:12:07.144614 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-33e5349f-5622-44a1-843c-247b4e4d5d29', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '33e5349f-5622-44a1-843c-247b4e4d5d29', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:12:07.170218 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] <== disconnect_volume: return (24ms) None {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:12:07.249871 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.manager [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] Encryption key ID for volume 33e5349f-5622-44a1-843c-247b4e4d5d29 already matches encryption key ID in backup 302ef0ea-3207-4ba4-a502-e34966251e52. {{(pid=134755) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} Mar 26 21:12:07.306270 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-a666a704-5ade-4d97-805a-4a7affc21275 tempest-VolumesBackupsTest-1678098904 None] Finished restoring backup 302ef0ea-3207-4ba4-a502-e34966251e52 to volume 33e5349f-5622-44a1-843c-247b4e4d5d29. Mar 26 21:12:10.321767 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-6137526d-e2b4-44b5-903d-d87d19c4ede9 tempest-VolumesBackupsTest-1678098904 None] Delete backup started, backup: 302ef0ea-3207-4ba4-a502-e34966251e52. Mar 26 21:12:10.323740 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-6137526d-e2b4-44b5-903d-d87d19c4ede9 tempest-VolumesBackupsTest-1678098904 None] Delete started for backup=302ef0ea-3207-4ba4-a502-e34966251e52 {{(pid=134755) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Mar 26 21:12:10.324435 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-6137526d-e2b4-44b5-903d-d87d19c4ede9 tempest-VolumesBackupsTest-1678098904 None] Trying diff format basename='volume-50b33a17-1e04-41a5-bced-d5d91836b5d6.backup.302ef0ea-3207-4ba4-a502-e34966251e52' for backup base image of volume 50b33a17-1e04-41a5-bced-d5d91836b5d6. {{(pid=134755) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Mar 26 21:12:10.400830 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-6137526d-e2b4-44b5-903d-d87d19c4ede9 tempest-VolumesBackupsTest-1678098904 None] Looking for snapshot of backup base 'volume-50b33a17-1e04-41a5-bced-d5d91836b5d6.backup.302ef0ea-3207-4ba4-a502-e34966251e52' {{(pid=134755) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 26 21:12:10.401396 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-6137526d-e2b4-44b5-903d-d87d19c4ede9 tempest-VolumesBackupsTest-1678098904 None] Found snapshot 'backup.302ef0ea-3207-4ba4-a502-e34966251e52.snap.1648329108.797369' {{(pid=134755) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Mar 26 21:12:10.402305 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-6137526d-e2b4-44b5-903d-d87d19c4ede9 tempest-VolumesBackupsTest-1678098904 None] Deleting backup snapshot='backup.302ef0ea-3207-4ba4-a502-e34966251e52.snap.1648329108.797369' {{(pid=134755) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Mar 26 21:12:11.599461 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.drivers.ceph [None req-6137526d-e2b4-44b5-903d-d87d19c4ede9 tempest-VolumesBackupsTest-1678098904 None] Deleting backup base image='volume-50b33a17-1e04-41a5-bced-d5d91836b5d6.backup.302ef0ea-3207-4ba4-a502-e34966251e52' of volume 50b33a17-1e04-41a5-bced-d5d91836b5d6. Mar 26 21:12:12.338927 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-6137526d-e2b4-44b5-903d-d87d19c4ede9 tempest-VolumesBackupsTest-1678098904 None] Base backup image='volume-50b33a17-1e04-41a5-bced-d5d91836b5d6.backup.302ef0ea-3207-4ba4-a502-e34966251e52' of volume 50b33a17-1e04-41a5-bced-d5d91836b5d6 deleted. {{(pid=134755) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Mar 26 21:12:12.420971 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-6137526d-e2b4-44b5-903d-d87d19c4ede9 tempest-VolumesBackupsTest-1678098904 None] Delete of backup '302ef0ea-3207-4ba4-a502-e34966251e52' for volume '50b33a17-1e04-41a5-bced-d5d91836b5d6' finished. {{(pid=134755) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Mar 26 21:12:12.428381 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_db.sqlalchemy.engines [None req-6137526d-e2b4-44b5-903d-d87d19c4ede9 tempest-VolumesBackupsTest-1678098904 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134755) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 26 21:12:12.460366 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.quota [None req-6137526d-e2b4-44b5-903d-d87d19c4ede9 tempest-VolumesBackupsTest-1678098904 None] Created reservations ['1435611c-db5a-415b-af5d-73b299a2fd60', '21a36e23-ab67-474f-865b-e02a426faa68'] {{(pid=134755) reserve /opt/stack/cinder/cinder/quota.py:755}} Mar 26 21:12:12.503458 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-6137526d-e2b4-44b5-903d-d87d19c4ede9 tempest-VolumesBackupsTest-1678098904 None] Delete backup finished, backup 302ef0ea-3207-4ba4-a502-e34966251e52 deleted. Mar 26 21:12:35.813962 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:12:35.814592 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:13:35.820056 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:13:35.821299 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:14:35.817436 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:14:35.818094 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:15:35.813519 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:15:35.814988 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:16:35.812250 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:16:35.812807 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:17:11.680543 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] Create backup started, backup: e353488c-2a1d-43e9-ab40-d3010695def3 volume: 3b669f91-be81-47a4-af6b-692e4288c80b. Mar 26 21:17:11.694712 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-03-26T21:17:11Z,data_timestamp=2022-03-26T21:17:11Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsAdminTest-Backup-1580242957',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=e353488c-2a1d-43e9-ab40-d3010695def3,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='8186063bcad64e01aaac232da670fcb2',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='4c2951d0babd4eb2a318f9f847477f71',volume_id=3b669f91-be81-47a4-af6b-692e4288c80b) Mar 26 21:17:11.736135 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.utils [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.16', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:17:11.737548 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:17:11.756335 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.019s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:17:11.756740 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[0debc987-13c9-41d5-ab26-576a69f8a71b]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:92f8692cf1e\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:17:11.761355 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:17:11.768501 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 26 21:17:11.769155 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[4239c23e-a297-43a0-9975-f30e9fa2e37f]: [Errno 2] No such file or directory: 'systool' {{(pid=163609) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 26 21:17:11.769155 ncdv-master-1 cinder-backup[134755]: Traceback (most recent call last): Mar 26 21:17:11.769155 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 26 21:17:11.769155 ncdv-master-1 cinder-backup[134755]: ret = func(*f_args, **f_kwargs) Mar 26 21:17:11.769155 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 26 21:17:11.769155 ncdv-master-1 cinder-backup[134755]: return func(*args, **kwargs) Mar 26 21:17:11.769155 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 26 21:17:11.769155 ncdv-master-1 cinder-backup[134755]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 26 21:17:11.769155 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 26 21:17:11.769155 ncdv-master-1 cinder-backup[134755]: return putils.execute(on_execute=on_execute, Mar 26 21:17:11.769155 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 26 21:17:11.769155 ncdv-master-1 cinder-backup[134755]: obj = subprocess.Popen(cmd, Mar 26 21:17:11.769155 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 26 21:17:11.769155 ncdv-master-1 cinder-backup[134755]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 26 21:17:11.769155 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 26 21:17:11.769155 ncdv-master-1 cinder-backup[134755]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 26 21:17:11.769155 ncdv-master-1 cinder-backup[134755]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 26 21:17:11.770541 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[4239c23e-a297-43a0-9975-f30e9fa2e37f]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:17:11.772159 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:17:11.778937 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 26 21:17:11.779668 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[eb206fc6-becc-43f5-bbfd-96b7093d0b8f]: [Errno 2] No such file or directory: 'systool' {{(pid=163609) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 26 21:17:11.779668 ncdv-master-1 cinder-backup[134755]: Traceback (most recent call last): Mar 26 21:17:11.779668 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 26 21:17:11.779668 ncdv-master-1 cinder-backup[134755]: ret = func(*f_args, **f_kwargs) Mar 26 21:17:11.779668 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 26 21:17:11.779668 ncdv-master-1 cinder-backup[134755]: return func(*args, **kwargs) Mar 26 21:17:11.779668 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 26 21:17:11.779668 ncdv-master-1 cinder-backup[134755]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 26 21:17:11.779668 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 26 21:17:11.779668 ncdv-master-1 cinder-backup[134755]: return putils.execute(on_execute=on_execute, Mar 26 21:17:11.779668 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 26 21:17:11.779668 ncdv-master-1 cinder-backup[134755]: obj = subprocess.Popen(cmd, Mar 26 21:17:11.779668 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 26 21:17:11.779668 ncdv-master-1 cinder-backup[134755]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 26 21:17:11.779668 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 26 21:17:11.779668 ncdv-master-1 cinder-backup[134755]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 26 21:17:11.779668 ncdv-master-1 cinder-backup[134755]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 26 21:17:11.781566 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[eb206fc6-becc-43f5-bbfd-96b7093d0b8f]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:17:11.783242 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:17:11.795142 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.011s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:17:11.796491 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[fbd5788e-d0e2-45d7-9825-6f3ce58995d6]: (4, ('/dev/sda1\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:17:11.800153 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:17:11.811178 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.012s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:17:11.811462 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[778a44ae-e90a-414a-a5bd-9608c28178f1]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:17:11.822114 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:17:11.832848 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.012s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:17:11.833432 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[67dd995c-d540-4043-ac7b-f5dc8e82d9a0]: (4, ('c6a66835-8686-c942-a93c-00f6321af5e6\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:17:11.834197 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] Running cmd (subprocess): nvme version {{(pid=134755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:17:11.865872 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] CMD "nvme version" returned: 0 in 0.031s {{(pid=134755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:17:11.867760 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134755) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 26 21:17:11.868045 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134755) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 26 21:17:11.868419 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:44f9348f-d915-47a9-a95c-57bf517d9528 dsc: {{(pid=134755) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 26 21:17:11.869057 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.utils [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] <== get_connector_properties: return (131ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.16', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:92f8692cf1e', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c6a66835-8686-c942-a93c-00f6321af5e6', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:44f9348f-d915-47a9-a95c-57bf517d9528', 'found_dsc': ''} {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:17:12.561255 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connector [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] Factory for rbd on None {{(pid=134755) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 26 21:17:12.562764 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-3b669f91-be81-47a4-af6b-692e4288c80b', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '3b669f91-be81-47a4-af6b-692e4288c80b', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:17:12.563968 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.linuxrbd [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] opening connection to ceph cluster (timeout=-1). {{(pid=134755) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 26 21:17:12.609566 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] <== connect_volume: return (46ms) {'path': } {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:17:12.616730 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_db.sqlalchemy.engines [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134755) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 26 21:17:12.635458 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] Starting backup of volume='3b669f91-be81-47a4-af6b-692e4288c80b'. {{(pid=134755) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Mar 26 21:17:12.638676 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] Volume file is RBD: attempting optimized backup {{(pid=134755) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Mar 26 21:17:12.661651 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] Creating base image 'volume-3b669f91-be81-47a4-af6b-692e4288c80b.backup.e353488c-2a1d-43e9-ab40-d3010695def3' {{(pid=134755) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Mar 26 21:17:12.662059 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134755) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Mar 26 21:17:12.706690 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] Using --from-snap 'None' for incremental backup of volume 3b669f91-be81-47a4-af6b-692e4288c80b. {{(pid=134755) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Mar 26 21:17:12.707265 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] Creating backup snapshot='backup.e353488c-2a1d-43e9-ab40-d3010695def3.snap.1648329432.706716' {{(pid=134755) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Mar 26 21:17:13.138852 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] Performing differential transfer from 'volume-3b669f91-be81-47a4-af6b-692e4288c80b' to 'volume-3b669f91-be81-47a4-af6b-692e4288c80b.backup.e353488c-2a1d-43e9-ab40-d3010695def3' {{(pid=134755) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Mar 26 21:17:13.139300 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_veh4n6hi --pool volumes volumes/volume-3b669f91-be81-47a4-af6b-692e4288c80b@backup.e353488c-2a1d-43e9-ab40-d3010695def3.snap.1648329432.706716 -' into... {{(pid=134755) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Mar 26 21:17:13.139715 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-3b669f91-be81-47a4-af6b-692e4288c80b.backup.e353488c-2a1d-43e9-ab40-d3010695def3' {{(pid=134755) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Mar 26 21:17:14.178152 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] Differential backup transfer completed in 1.0387s {{(pid=134755) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Mar 26 21:17:14.178782 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] Getting metadata type 'volume-base-metadata' {{(pid=134755) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Mar 26 21:17:14.193557 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134755) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Mar 26 21:17:14.193978 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] Getting metadata type 'volume-metadata' {{(pid=134755) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Mar 26 21:17:14.202334 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] No metadata type 'volume-metadata' available {{(pid=134755) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Mar 26 21:17:14.202729 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] Getting metadata type 'volume-glance-metadata' {{(pid=134755) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Mar 26 21:17:14.209012 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] No metadata type 'volume-glance-metadata' available {{(pid=134755) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Mar 26 21:17:14.209566 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] Backing up metadata for volume 3b669f91-be81-47a4-af6b-692e4288c80b. {{(pid=134755) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Mar 26 21:17:14.241827 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] Backup 'e353488c-2a1d-43e9-ab40-d3010695def3' of volume 3b669f91-be81-47a4-af6b-692e4288c80b finished. {{(pid=134755) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Mar 26 21:17:14.243438 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-3b669f91-be81-47a4-af6b-692e4288c80b', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '3b669f91-be81-47a4-af6b-692e4288c80b', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:17:14.263613 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] <== disconnect_volume: return (19ms) None {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:17:14.328417 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-a13d924d-e79e-4930-b905-211bafe94647 tempest-VolumesBackupsAdminTest-228261023 None] Create backup finished. backup: e353488c-2a1d-43e9-ab40-d3010695def3. Mar 26 21:17:14.883494 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-3247fc04-1001-4dce-aae0-b9c5f85cfc5d tempest-VolumesBackupsAdminTest-833499075 None] Export record started, backup: e353488c-2a1d-43e9-ab40-d3010695def3. Mar 26 21:17:14.885498 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-3247fc04-1001-4dce-aae0-b9c5f85cfc5d tempest-VolumesBackupsAdminTest-833499075 None] Export record finished, backup e353488c-2a1d-43e9-ab40-d3010695def3 exported. Mar 26 21:17:15.026350 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-42607b6f-e0d7-4bfd-afc5-9a0730c5914a tempest-VolumesBackupsAdminTest-833499075 None] Import record started, backup_url: eyJkcml2ZXJfaW5mbyI6IHt9LCAiaWQiOiAiMmY4ZDUzZWUtZGQ0OC00MjhkLWI2YmYtYzIzODljMzkzMGIwIiwgInVzZXJfaWQiOiAiNGMyOTUxZDBiYWJkNGViMmEzMThmOWY4NDc0NzdmNzEiLCAicHJvamVjdF9pZCI6ICI4MTg2MDYzYmNhZDY0ZTAxYWFhYzIzMmRhNjcwZmNiMiIsICJ2b2x1bWVfaWQiOiAiM2I2NjlmOTEtYmU4MS00N2E0LWFmNmItNjkyZTQyODhjODBiIiwgImhvc3QiOiAibmNkdi1tYXN0ZXItMSIsICJhdmFpbGFiaWxpdHlfem9uZSI6IG51bGwsICJjb250YWluZXIiOiAiYmFja3VwcyIsICJwYXJlbnRfaWQiOiBudWxsLCAic3RhdHVzIjogImF2YWlsYWJsZSIsICJmYWlsX3JlYXNvbiI6IG51bGwsICJzaXplIjogMSwgImRpc3BsYXlfbmFtZSI6ICJ0ZW1wZXN0LVZvbHVtZXNCYWNrdXBzQWRtaW5UZXN0LUJhY2t1cC0xNTgwMjQyOTU3IiwgImRpc3BsYXlfZGVzY3JpcHRpb24iOiBudWxsLCAic2VydmljZV9tZXRhZGF0YSI6ICJ7XCJiYXNlXCI6IFwidm9sdW1lLTNiNjY5ZjkxLWJlODEtNDdhNC1hZjZiLTY5MmU0Mjg4YzgwYi5iYWNrdXAuZTM1MzQ4OGMtMmExZC00M2U5LWFiNDAtZDMwMTA2OTVkZWYzXCJ9IiwgInNlcnZpY2UiOiAiY2luZGVyLmJhY2t1cC5kcml2ZXJzLmNlcGguQ2VwaEJhY2t1cERyaXZlciIsICJvYmplY3RfY291bnQiOiAwLCAidGVtcF92b2x1bWVfaWQiOiBudWxsLCAidGVtcF9zbmFwc2hvdF9pZCI6IG51bGwsICJudW1fZGVwZW5kZW50X2JhY2t1cHMiOiAwLCAic25hcHNob3RfaWQiOiBudWxsLCAiZGF0YV90aW1lc3RhbXAiOiAiMjAyMi0wMy0yNlQyMToxNzoxMloiLCAicmVzdG9yZV92b2x1bWVfaWQiOiBudWxsLCAibWV0YWRhdGEiOiB7fSwgImVuY3J5cHRpb25fa2V5X2lkIjogbnVsbCwgImNyZWF0ZWRfYXQiOiAiMjAyMi0wMy0yNlQyMToxNzoxMloiLCAidXBkYXRlZF9hdCI6ICIyMDIyLTAzLTI2VDIxOjE3OjE0WiIsICJkZWxldGVkX2F0IjogbnVsbCwgImRlbGV0ZWQiOiBmYWxzZX0=. Mar 26 21:17:15.035583 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_db.sqlalchemy.engines [None req-42607b6f-e0d7-4bfd-afc5-9a0730c5914a tempest-VolumesBackupsAdminTest-833499075 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134755) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 26 21:17:15.080080 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-42607b6f-e0d7-4bfd-afc5-9a0730c5914a tempest-VolumesBackupsAdminTest-833499075 None] Import record id 2f8d53ee-dd48-428d-b6bf-c2389c3930b0 metadata from driver finished. Mar 26 21:17:17.406515 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] Restore backup started, backup: e353488c-2a1d-43e9-ab40-d3010695def3 volume: 070cb53b-b3a8-49f0-bd7e-156bddf48e43. Mar 26 21:17:17.430292 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.utils [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.16', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:17:17.431894 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:17:17.449789 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.017s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:17:17.450108 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[3916f3fe-5f8d-408f-9781-c92571644d2d]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:92f8692cf1e\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:17:17.451395 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:17:17.460054 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 26 21:17:17.460618 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[7e03b069-a57c-4944-bab9-c406cd1adb45]: [Errno 2] No such file or directory: 'systool' {{(pid=163609) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 26 21:17:17.460618 ncdv-master-1 cinder-backup[134755]: Traceback (most recent call last): Mar 26 21:17:17.460618 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 26 21:17:17.460618 ncdv-master-1 cinder-backup[134755]: ret = func(*f_args, **f_kwargs) Mar 26 21:17:17.460618 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 26 21:17:17.460618 ncdv-master-1 cinder-backup[134755]: return func(*args, **kwargs) Mar 26 21:17:17.460618 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 26 21:17:17.460618 ncdv-master-1 cinder-backup[134755]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 26 21:17:17.460618 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 26 21:17:17.460618 ncdv-master-1 cinder-backup[134755]: return putils.execute(on_execute=on_execute, Mar 26 21:17:17.460618 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 26 21:17:17.460618 ncdv-master-1 cinder-backup[134755]: obj = subprocess.Popen(cmd, Mar 26 21:17:17.460618 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 26 21:17:17.460618 ncdv-master-1 cinder-backup[134755]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 26 21:17:17.460618 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 26 21:17:17.460618 ncdv-master-1 cinder-backup[134755]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 26 21:17:17.460618 ncdv-master-1 cinder-backup[134755]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 26 21:17:17.461776 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[7e03b069-a57c-4944-bab9-c406cd1adb45]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:17:17.462870 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:17:17.469164 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 26 21:17:17.469777 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[0af5753f-4ca5-4c95-b0fe-cbf6f7c7aa83]: [Errno 2] No such file or directory: 'systool' {{(pid=163609) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 26 21:17:17.469777 ncdv-master-1 cinder-backup[134755]: Traceback (most recent call last): Mar 26 21:17:17.469777 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 26 21:17:17.469777 ncdv-master-1 cinder-backup[134755]: ret = func(*f_args, **f_kwargs) Mar 26 21:17:17.469777 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 26 21:17:17.469777 ncdv-master-1 cinder-backup[134755]: return func(*args, **kwargs) Mar 26 21:17:17.469777 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 26 21:17:17.469777 ncdv-master-1 cinder-backup[134755]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 26 21:17:17.469777 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 26 21:17:17.469777 ncdv-master-1 cinder-backup[134755]: return putils.execute(on_execute=on_execute, Mar 26 21:17:17.469777 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 26 21:17:17.469777 ncdv-master-1 cinder-backup[134755]: obj = subprocess.Popen(cmd, Mar 26 21:17:17.469777 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 26 21:17:17.469777 ncdv-master-1 cinder-backup[134755]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 26 21:17:17.469777 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 26 21:17:17.469777 ncdv-master-1 cinder-backup[134755]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 26 21:17:17.469777 ncdv-master-1 cinder-backup[134755]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 26 21:17:17.470791 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[0af5753f-4ca5-4c95-b0fe-cbf6f7c7aa83]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:17:17.472027 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:17:17.481617 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.009s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:17:17.481931 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[98bd6970-52fd-4d90-9f49-91fc604f72f7]: (4, ('/dev/sda1\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:17:17.485332 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:17:17.492668 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.009s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:17:17.493132 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[43f3fa3b-de5d-4de7-b3f7-c1c9472756a7]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:17:17.495059 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:17:17.504240 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.009s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:17:17.504595 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[feb71548-142f-42e1-a049-1f2e01f14380]: (4, ('c6a66835-8686-c942-a93c-00f6321af5e6\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:17:17.505432 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] Running cmd (subprocess): nvme version {{(pid=134755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:17:17.524353 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] CMD "nvme version" returned: 0 in 0.019s {{(pid=134755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:17:17.526095 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134755) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 26 21:17:17.526494 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134755) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 26 21:17:17.527154 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:44f9348f-d915-47a9-a95c-57bf517d9528 dsc: {{(pid=134755) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 26 21:17:17.527842 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.utils [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] <== get_connector_properties: return (96ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.16', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:92f8692cf1e', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c6a66835-8686-c942-a93c-00f6321af5e6', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:44f9348f-d915-47a9-a95c-57bf517d9528', 'found_dsc': ''} {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:17:17.958225 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_db.sqlalchemy.engines [None req-3e2636c3-f868-49d5-9eb0-1c88dcfc918b None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134755) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 26 21:17:18.218392 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connector [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] Factory for rbd on None {{(pid=134755) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 26 21:17:18.219960 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-070cb53b-b3a8-49f0-bd7e-156bddf48e43', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '070cb53b-b3a8-49f0-bd7e-156bddf48e43', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:17:18.221115 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.linuxrbd [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] opening connection to ceph cluster (timeout=-1). {{(pid=134755) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 26 21:17:18.264410 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] <== connect_volume: return (43ms) {'path': } {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:17:18.275615 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] Starting restore from Ceph backup=e353488c-2a1d-43e9-ab40-d3010695def3 to volume=volume-070cb53b-b3a8-49f0-bd7e-156bddf48e43 {{(pid=134755) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1274}} Mar 26 21:17:18.339050 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] Looking for snapshot of backup base 'volume-3b669f91-be81-47a4-af6b-692e4288c80b.backup.e353488c-2a1d-43e9-ab40-d3010695def3' {{(pid=134755) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 26 21:17:18.339584 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] Found snapshot 'backup.e353488c-2a1d-43e9-ab40-d3010695def3.snap.1648329432.706716' {{(pid=134755) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Mar 26 21:17:18.344548 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] Volume file is RBD. {{(pid=134755) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1184}} Mar 26 21:17:18.378544 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] Attempting differential restore. {{(pid=134755) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1234}} Mar 26 21:17:18.379048 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] Attempting incremental restore from base='volume-3b669f91-be81-47a4-af6b-692e4288c80b.backup.e353488c-2a1d-43e9-ab40-d3010695def3' snap='backup.e353488c-2a1d-43e9-ab40-d3010695def3.snap.1648329432.706716' {{(pid=134755) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1089}} Mar 26 21:17:18.379455 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] Performing differential transfer from 'volume-3b669f91-be81-47a4-af6b-692e4288c80b.backup.e353488c-2a1d-43e9-ab40-d3010695def3' to 'volume-070cb53b-b3a8-49f0-bd7e-156bddf48e43' {{(pid=134755) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Mar 26 21:17:18.379864 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-3b669f91-be81-47a4-af6b-692e4288c80b.backup.e353488c-2a1d-43e9-ab40-d3010695def3@backup.e353488c-2a1d-43e9-ab40-d3010695def3.snap.1648329432.706716 -' into... {{(pid=134755) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Mar 26 21:17:18.380271 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_lczp5wgi --pool volumes - volumes/volume-070cb53b-b3a8-49f0-bd7e-156bddf48e43' {{(pid=134755) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Mar 26 21:17:19.283143 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] Restore transfer completed in 0.9031s {{(pid=134755) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1111}} Mar 26 21:17:19.284180 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=134755) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1285}} Mar 26 21:17:19.308161 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] Restoring volume base metadata {{(pid=134755) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} Mar 26 21:17:19.329666 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] No metadata of type 'volume-metadata' to restore {{(pid=134755) put /opt/stack/cinder/cinder/backup/driver.py:345}} Mar 26 21:17:19.330271 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] No metadata of type 'volume-glance-metadata' to restore {{(pid=134755) put /opt/stack/cinder/cinder/backup/driver.py:345}} Mar 26 21:17:19.334171 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] Restore to volume 070cb53b-b3a8-49f0-bd7e-156bddf48e43 finished successfully. {{(pid=134755) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1292}} Mar 26 21:17:19.335847 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-070cb53b-b3a8-49f0-bd7e-156bddf48e43', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '070cb53b-b3a8-49f0-bd7e-156bddf48e43', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:17:19.347854 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] <== disconnect_volume: return (11ms) None {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:17:19.399419 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.manager [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] Encryption key ID for volume 070cb53b-b3a8-49f0-bd7e-156bddf48e43 already matches encryption key ID in backup e353488c-2a1d-43e9-ab40-d3010695def3. {{(pid=134755) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} Mar 26 21:17:19.432395 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-9c3bd668-6e3c-41a6-b975-f4aa428c86dd tempest-VolumesBackupsAdminTest-228261023 None] Finished restoring backup e353488c-2a1d-43e9-ab40-d3010695def3 to volume 070cb53b-b3a8-49f0-bd7e-156bddf48e43. Mar 26 21:17:19.736591 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-0e9bd7b2-c90e-4ee0-ac59-8ae66876970f tempest-VolumesBackupsAdminTest-833499075 None] Delete backup started, backup: 2f8d53ee-dd48-428d-b6bf-c2389c3930b0. Mar 26 21:17:19.740951 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-0e9bd7b2-c90e-4ee0-ac59-8ae66876970f tempest-VolumesBackupsAdminTest-833499075 None] Delete started for backup=2f8d53ee-dd48-428d-b6bf-c2389c3930b0 {{(pid=134755) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Mar 26 21:17:19.744189 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-0e9bd7b2-c90e-4ee0-ac59-8ae66876970f tempest-VolumesBackupsAdminTest-833499075 None] Trying diff format basename='volume-3b669f91-be81-47a4-af6b-692e4288c80b.backup.e353488c-2a1d-43e9-ab40-d3010695def3' for backup base image of volume 3b669f91-be81-47a4-af6b-692e4288c80b. {{(pid=134755) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Mar 26 21:17:19.814130 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-0e9bd7b2-c90e-4ee0-ac59-8ae66876970f tempest-VolumesBackupsAdminTest-833499075 None] Looking for snapshot of backup base 'volume-3b669f91-be81-47a4-af6b-692e4288c80b.backup.e353488c-2a1d-43e9-ab40-d3010695def3' {{(pid=134755) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 26 21:17:19.814522 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-0e9bd7b2-c90e-4ee0-ac59-8ae66876970f tempest-VolumesBackupsAdminTest-833499075 None] Backup '2f8d53ee-dd48-428d-b6bf-c2389c3930b0' has no snapshot {{(pid=134755) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:915}} Mar 26 21:17:19.814918 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-0e9bd7b2-c90e-4ee0-ac59-8ae66876970f tempest-VolumesBackupsAdminTest-833499075 None] No backup snapshot to delete {{(pid=134755) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:476}} Mar 26 21:17:19.824449 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.drivers.ceph [None req-0e9bd7b2-c90e-4ee0-ac59-8ae66876970f tempest-VolumesBackupsAdminTest-833499075 None] Backup base image of volume 3b669f91-be81-47a4-af6b-692e4288c80b still has 1 snapshots so skipping base image delete. Mar 26 21:17:19.860794 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-0e9bd7b2-c90e-4ee0-ac59-8ae66876970f tempest-VolumesBackupsAdminTest-833499075 None] Metadata backup object 'backup.2f8d53ee-dd48-428d-b6bf-c2389c3930b0.meta' not found - ignoring {{(pid=134755) remove_if_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:160}} Mar 26 21:17:19.864105 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-0e9bd7b2-c90e-4ee0-ac59-8ae66876970f tempest-VolumesBackupsAdminTest-833499075 None] Delete of backup '2f8d53ee-dd48-428d-b6bf-c2389c3930b0' for volume '3b669f91-be81-47a4-af6b-692e4288c80b' finished. {{(pid=134755) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Mar 26 21:17:19.869717 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_db.sqlalchemy.engines [None req-0e9bd7b2-c90e-4ee0-ac59-8ae66876970f tempest-VolumesBackupsAdminTest-833499075 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134755) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 26 21:17:19.892844 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.quota [None req-0e9bd7b2-c90e-4ee0-ac59-8ae66876970f tempest-VolumesBackupsAdminTest-833499075 None] Created reservations ['47bf004c-45f8-4c9f-98bb-149ce08240dc', 'ed19a780-4233-4de8-b0bc-890584ad88f6'] {{(pid=134755) reserve /opt/stack/cinder/cinder/quota.py:755}} Mar 26 21:17:19.919127 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-0e9bd7b2-c90e-4ee0-ac59-8ae66876970f tempest-VolumesBackupsAdminTest-833499075 None] Delete backup finished, backup 2f8d53ee-dd48-428d-b6bf-c2389c3930b0 deleted. Mar 26 21:17:20.834248 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-f36efb15-51b8-485a-bb0c-ad2d91c8a9b6 tempest-VolumesBackupsAdminTest-228261023 None] Delete backup started, backup: e353488c-2a1d-43e9-ab40-d3010695def3. Mar 26 21:17:20.834889 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-f36efb15-51b8-485a-bb0c-ad2d91c8a9b6 tempest-VolumesBackupsAdminTest-228261023 None] Delete started for backup=e353488c-2a1d-43e9-ab40-d3010695def3 {{(pid=134755) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Mar 26 21:17:20.834889 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-f36efb15-51b8-485a-bb0c-ad2d91c8a9b6 tempest-VolumesBackupsAdminTest-228261023 None] Trying diff format basename='volume-3b669f91-be81-47a4-af6b-692e4288c80b.backup.e353488c-2a1d-43e9-ab40-d3010695def3' for backup base image of volume 3b669f91-be81-47a4-af6b-692e4288c80b. {{(pid=134755) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Mar 26 21:17:20.891096 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-f36efb15-51b8-485a-bb0c-ad2d91c8a9b6 tempest-VolumesBackupsAdminTest-228261023 None] Looking for snapshot of backup base 'volume-3b669f91-be81-47a4-af6b-692e4288c80b.backup.e353488c-2a1d-43e9-ab40-d3010695def3' {{(pid=134755) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 26 21:17:20.891585 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-f36efb15-51b8-485a-bb0c-ad2d91c8a9b6 tempest-VolumesBackupsAdminTest-228261023 None] Found snapshot 'backup.e353488c-2a1d-43e9-ab40-d3010695def3.snap.1648329432.706716' {{(pid=134755) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Mar 26 21:17:20.891987 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-f36efb15-51b8-485a-bb0c-ad2d91c8a9b6 tempest-VolumesBackupsAdminTest-228261023 None] Deleting backup snapshot='backup.e353488c-2a1d-43e9-ab40-d3010695def3.snap.1648329432.706716' {{(pid=134755) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Mar 26 21:17:21.258945 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.drivers.ceph [None req-f36efb15-51b8-485a-bb0c-ad2d91c8a9b6 tempest-VolumesBackupsAdminTest-228261023 None] Deleting backup base image='volume-3b669f91-be81-47a4-af6b-692e4288c80b.backup.e353488c-2a1d-43e9-ab40-d3010695def3' of volume 3b669f91-be81-47a4-af6b-692e4288c80b. Mar 26 21:17:21.448418 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-f36efb15-51b8-485a-bb0c-ad2d91c8a9b6 tempest-VolumesBackupsAdminTest-228261023 None] Base backup image='volume-3b669f91-be81-47a4-af6b-692e4288c80b.backup.e353488c-2a1d-43e9-ab40-d3010695def3' of volume 3b669f91-be81-47a4-af6b-692e4288c80b deleted. {{(pid=134755) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Mar 26 21:17:21.486665 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-f36efb15-51b8-485a-bb0c-ad2d91c8a9b6 tempest-VolumesBackupsAdminTest-228261023 None] Delete of backup 'e353488c-2a1d-43e9-ab40-d3010695def3' for volume '3b669f91-be81-47a4-af6b-692e4288c80b' finished. {{(pid=134755) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Mar 26 21:17:21.492343 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_db.sqlalchemy.engines [None req-f36efb15-51b8-485a-bb0c-ad2d91c8a9b6 tempest-VolumesBackupsAdminTest-228261023 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134755) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 26 21:17:21.514705 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.quota [None req-f36efb15-51b8-485a-bb0c-ad2d91c8a9b6 tempest-VolumesBackupsAdminTest-228261023 None] Created reservations ['18f63d61-c6ae-4922-8b6f-b0796ddc4839', 'e011b965-da7b-4cd8-9c3e-a429e8128a7c'] {{(pid=134755) reserve /opt/stack/cinder/cinder/quota.py:755}} Mar 26 21:17:21.539293 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-f36efb15-51b8-485a-bb0c-ad2d91c8a9b6 tempest-VolumesBackupsAdminTest-228261023 None] Delete backup finished, backup e353488c-2a1d-43e9-ab40-d3010695def3 deleted. Mar 26 21:17:23.368948 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] Create backup started, backup: 60c8343b-0eaa-42bf-878d-d87dcd0f4253 volume: 8a13b1b5-3010-42dd-9829-45b35909b8db. Mar 26 21:17:23.380754 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-03-26T21:17:23Z,data_timestamp=2022-03-26T21:17:23Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsAdminTest-Backup-801704782',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=60c8343b-0eaa-42bf-878d-d87dcd0f4253,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='8186063bcad64e01aaac232da670fcb2',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='4c2951d0babd4eb2a318f9f847477f71',volume_id=8a13b1b5-3010-42dd-9829-45b35909b8db) Mar 26 21:17:23.421192 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.utils [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.16', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:17:23.423033 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:17:23.432272 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.009s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:17:23.432711 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[68f274f1-bcef-42bd-b23d-739f3461110a]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:92f8692cf1e\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:17:23.435681 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:17:23.442274 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 26 21:17:23.443466 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[ef9a5447-0520-43e7-bc7b-74bec43a39a4]: [Errno 2] No such file or directory: 'systool' {{(pid=163609) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 26 21:17:23.443466 ncdv-master-1 cinder-backup[134755]: Traceback (most recent call last): Mar 26 21:17:23.443466 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 26 21:17:23.443466 ncdv-master-1 cinder-backup[134755]: ret = func(*f_args, **f_kwargs) Mar 26 21:17:23.443466 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 26 21:17:23.443466 ncdv-master-1 cinder-backup[134755]: return func(*args, **kwargs) Mar 26 21:17:23.443466 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 26 21:17:23.443466 ncdv-master-1 cinder-backup[134755]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 26 21:17:23.443466 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 26 21:17:23.443466 ncdv-master-1 cinder-backup[134755]: return putils.execute(on_execute=on_execute, Mar 26 21:17:23.443466 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 26 21:17:23.443466 ncdv-master-1 cinder-backup[134755]: obj = subprocess.Popen(cmd, Mar 26 21:17:23.443466 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 26 21:17:23.443466 ncdv-master-1 cinder-backup[134755]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 26 21:17:23.443466 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 26 21:17:23.443466 ncdv-master-1 cinder-backup[134755]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 26 21:17:23.443466 ncdv-master-1 cinder-backup[134755]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 26 21:17:23.444502 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[ef9a5447-0520-43e7-bc7b-74bec43a39a4]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:17:23.445590 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:17:23.452890 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 26 21:17:23.453489 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[ea5d33e1-aa44-46b1-beff-058d6a4ae616]: [Errno 2] No such file or directory: 'systool' {{(pid=163609) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 26 21:17:23.453489 ncdv-master-1 cinder-backup[134755]: Traceback (most recent call last): Mar 26 21:17:23.453489 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 26 21:17:23.453489 ncdv-master-1 cinder-backup[134755]: ret = func(*f_args, **f_kwargs) Mar 26 21:17:23.453489 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 26 21:17:23.453489 ncdv-master-1 cinder-backup[134755]: return func(*args, **kwargs) Mar 26 21:17:23.453489 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 26 21:17:23.453489 ncdv-master-1 cinder-backup[134755]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 26 21:17:23.453489 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 26 21:17:23.453489 ncdv-master-1 cinder-backup[134755]: return putils.execute(on_execute=on_execute, Mar 26 21:17:23.453489 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 26 21:17:23.453489 ncdv-master-1 cinder-backup[134755]: obj = subprocess.Popen(cmd, Mar 26 21:17:23.453489 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 26 21:17:23.453489 ncdv-master-1 cinder-backup[134755]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 26 21:17:23.453489 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 26 21:17:23.453489 ncdv-master-1 cinder-backup[134755]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 26 21:17:23.453489 ncdv-master-1 cinder-backup[134755]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 26 21:17:23.454464 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[ea5d33e1-aa44-46b1-beff-058d6a4ae616]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:17:23.455143 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:17:23.464148 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.009s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:17:23.464935 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[a673d4d5-bf1a-42ea-ad32-e7bb5952a468]: (4, ('/dev/sda1\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:17:23.466489 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:17:23.475290 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.009s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:17:23.475678 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[2898da92-7ba1-4af2-a935-cc4e46fec647]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:17:23.476941 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:17:23.487821 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.011s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:17:23.488184 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[00311068-5226-429e-bed6-9b3e5e649a7c]: (4, ('c6a66835-8686-c942-a93c-00f6321af5e6\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:17:23.488936 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] Running cmd (subprocess): nvme version {{(pid=134755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:17:23.507792 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] CMD "nvme version" returned: 0 in 0.019s {{(pid=134755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:17:23.509819 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134755) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 26 21:17:23.510201 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134755) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 26 21:17:23.510582 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:44f9348f-d915-47a9-a95c-57bf517d9528 dsc: {{(pid=134755) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 26 21:17:23.511343 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.utils [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] <== get_connector_properties: return (89ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.16', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:92f8692cf1e', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c6a66835-8686-c942-a93c-00f6321af5e6', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:44f9348f-d915-47a9-a95c-57bf517d9528', 'found_dsc': ''} {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:17:24.183853 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connector [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] Factory for rbd on None {{(pid=134755) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 26 21:17:24.185372 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-8a13b1b5-3010-42dd-9829-45b35909b8db', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '8a13b1b5-3010-42dd-9829-45b35909b8db', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:17:24.186551 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.linuxrbd [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] opening connection to ceph cluster (timeout=-1). {{(pid=134755) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 26 21:17:24.230729 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] <== connect_volume: return (44ms) {'path': } {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:17:24.238582 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_db.sqlalchemy.engines [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134755) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 26 21:17:24.259420 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] Starting backup of volume='8a13b1b5-3010-42dd-9829-45b35909b8db'. {{(pid=134755) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Mar 26 21:17:24.260424 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] Volume file is RBD: attempting optimized backup {{(pid=134755) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Mar 26 21:17:24.283389 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] Creating base image 'volume-8a13b1b5-3010-42dd-9829-45b35909b8db.backup.60c8343b-0eaa-42bf-878d-d87dcd0f4253' {{(pid=134755) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Mar 26 21:17:24.284170 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134755) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Mar 26 21:17:24.316294 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] Using --from-snap 'None' for incremental backup of volume 8a13b1b5-3010-42dd-9829-45b35909b8db. {{(pid=134755) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Mar 26 21:17:24.316715 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] Creating backup snapshot='backup.60c8343b-0eaa-42bf-878d-d87dcd0f4253.snap.1648329444.3163192' {{(pid=134755) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Mar 26 21:17:25.288520 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] Performing differential transfer from 'volume-8a13b1b5-3010-42dd-9829-45b35909b8db' to 'volume-8a13b1b5-3010-42dd-9829-45b35909b8db.backup.60c8343b-0eaa-42bf-878d-d87dcd0f4253' {{(pid=134755) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Mar 26 21:17:25.289205 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_sqmmrl3l --pool volumes volumes/volume-8a13b1b5-3010-42dd-9829-45b35909b8db@backup.60c8343b-0eaa-42bf-878d-d87dcd0f4253.snap.1648329444.3163192 -' into... {{(pid=134755) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Mar 26 21:17:25.289295 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-8a13b1b5-3010-42dd-9829-45b35909b8db.backup.60c8343b-0eaa-42bf-878d-d87dcd0f4253' {{(pid=134755) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Mar 26 21:17:26.031799 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] Differential backup transfer completed in 0.7429s {{(pid=134755) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Mar 26 21:17:26.032401 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] Getting metadata type 'volume-base-metadata' {{(pid=134755) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Mar 26 21:17:26.046236 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134755) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Mar 26 21:17:26.046653 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] Getting metadata type 'volume-metadata' {{(pid=134755) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Mar 26 21:17:26.053232 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] No metadata type 'volume-metadata' available {{(pid=134755) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Mar 26 21:17:26.053628 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] Getting metadata type 'volume-glance-metadata' {{(pid=134755) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Mar 26 21:17:26.060130 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] No metadata type 'volume-glance-metadata' available {{(pid=134755) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Mar 26 21:17:26.060711 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] Backing up metadata for volume 8a13b1b5-3010-42dd-9829-45b35909b8db. {{(pid=134755) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Mar 26 21:17:26.095182 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] Backup '60c8343b-0eaa-42bf-878d-d87dcd0f4253' of volume 8a13b1b5-3010-42dd-9829-45b35909b8db finished. {{(pid=134755) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Mar 26 21:17:26.096841 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-8a13b1b5-3010-42dd-9829-45b35909b8db', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '8a13b1b5-3010-42dd-9829-45b35909b8db', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:17:26.117540 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] <== disconnect_volume: return (20ms) None {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:17:26.179249 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-9a836ee2-bce6-4670-8cc4-1b863486ef4c tempest-VolumesBackupsAdminTest-228261023 None] Create backup finished. backup: 60c8343b-0eaa-42bf-878d-d87dcd0f4253. Mar 26 21:17:26.547533 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-d8c84ac6-9d02-4aba-a536-f79221424616 tempest-VolumesBackupsAdminTest-833499075 None] Reset backup status started, backup_id: 60c8343b-0eaa-42bf-878d-d87dcd0f4253, status: error. Mar 26 21:17:26.547533 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-d8c84ac6-9d02-4aba-a536-f79221424616 tempest-VolumesBackupsAdminTest-833499075 None] Backup service: cinder.backup.drivers.ceph.CephBackupDriver. Mar 26 21:17:26.632885 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-499497d5-748f-4446-b113-151bfdfd8438 tempest-VolumesBackupsAdminTest-228261023 None] Delete backup started, backup: 60c8343b-0eaa-42bf-878d-d87dcd0f4253. Mar 26 21:17:26.635007 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-499497d5-748f-4446-b113-151bfdfd8438 tempest-VolumesBackupsAdminTest-228261023 None] Delete started for backup=60c8343b-0eaa-42bf-878d-d87dcd0f4253 {{(pid=134755) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Mar 26 21:17:26.635473 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-499497d5-748f-4446-b113-151bfdfd8438 tempest-VolumesBackupsAdminTest-228261023 None] Trying diff format basename='volume-8a13b1b5-3010-42dd-9829-45b35909b8db.backup.60c8343b-0eaa-42bf-878d-d87dcd0f4253' for backup base image of volume 8a13b1b5-3010-42dd-9829-45b35909b8db. {{(pid=134755) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Mar 26 21:17:26.685682 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-499497d5-748f-4446-b113-151bfdfd8438 tempest-VolumesBackupsAdminTest-228261023 None] Looking for snapshot of backup base 'volume-8a13b1b5-3010-42dd-9829-45b35909b8db.backup.60c8343b-0eaa-42bf-878d-d87dcd0f4253' {{(pid=134755) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 26 21:17:26.686318 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-499497d5-748f-4446-b113-151bfdfd8438 tempest-VolumesBackupsAdminTest-228261023 None] Found snapshot 'backup.60c8343b-0eaa-42bf-878d-d87dcd0f4253.snap.1648329444.3163192' {{(pid=134755) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Mar 26 21:17:26.686782 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-499497d5-748f-4446-b113-151bfdfd8438 tempest-VolumesBackupsAdminTest-228261023 None] Deleting backup snapshot='backup.60c8343b-0eaa-42bf-878d-d87dcd0f4253.snap.1648329444.3163192' {{(pid=134755) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Mar 26 21:17:27.348651 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.drivers.ceph [None req-499497d5-748f-4446-b113-151bfdfd8438 tempest-VolumesBackupsAdminTest-228261023 None] Deleting backup base image='volume-8a13b1b5-3010-42dd-9829-45b35909b8db.backup.60c8343b-0eaa-42bf-878d-d87dcd0f4253' of volume 8a13b1b5-3010-42dd-9829-45b35909b8db. Mar 26 21:17:27.523603 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-499497d5-748f-4446-b113-151bfdfd8438 tempest-VolumesBackupsAdminTest-228261023 None] Base backup image='volume-8a13b1b5-3010-42dd-9829-45b35909b8db.backup.60c8343b-0eaa-42bf-878d-d87dcd0f4253' of volume 8a13b1b5-3010-42dd-9829-45b35909b8db deleted. {{(pid=134755) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Mar 26 21:17:27.563639 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-499497d5-748f-4446-b113-151bfdfd8438 tempest-VolumesBackupsAdminTest-228261023 None] Delete of backup '60c8343b-0eaa-42bf-878d-d87dcd0f4253' for volume '8a13b1b5-3010-42dd-9829-45b35909b8db' finished. {{(pid=134755) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Mar 26 21:17:27.569467 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_db.sqlalchemy.engines [None req-499497d5-748f-4446-b113-151bfdfd8438 tempest-VolumesBackupsAdminTest-228261023 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134755) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 26 21:17:27.593128 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.quota [None req-499497d5-748f-4446-b113-151bfdfd8438 tempest-VolumesBackupsAdminTest-228261023 None] Created reservations ['1a9d55ef-732a-4a10-b9dd-f5c16b3456fb', 'e9578403-ed00-4515-aa56-d8d6e1e4a40b'] {{(pid=134755) reserve /opt/stack/cinder/cinder/quota.py:755}} Mar 26 21:17:27.616249 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-499497d5-748f-4446-b113-151bfdfd8438 tempest-VolumesBackupsAdminTest-228261023 None] Delete backup finished, backup 60c8343b-0eaa-42bf-878d-d87dcd0f4253 deleted. Mar 26 21:17:35.809278 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:17:35.809933 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:18:35.805609 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:18:35.806768 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:19:35.802118 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:19:35.802776 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:20:35.807436 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:20:35.808056 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:21:35.813187 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:21:35.813884 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:22:35.818145 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:22:35.818865 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:23:35.823953 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:23:35.824666 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:24:35.829144 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:24:35.829726 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:25:35.834334 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:25:35.835031 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:26:35.838891 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:26:35.839605 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:27:35.844712 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:27:35.845811 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:28:04.821519 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] Create backup started, backup: 5e7cf4fe-3c96-4331-a08b-f822d28b0141 volume: a8b9669a-47aa-4380-95db-03adf51809b5. Mar 26 21:28:04.829496 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-03-26T21:28:04Z,data_timestamp=2022-03-26T21:28:04Z,deleted=False,deleted_at=None,display_description=None,display_name=None,encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=5e7cf4fe-3c96-4331-a08b-f822d28b0141,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='d7d600597d044d62bfa72df67a61c0bb',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='d2954748f375411b87550803a8b2d3ad',volume_id=a8b9669a-47aa-4380-95db-03adf51809b5) Mar 26 21:28:04.873321 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.utils [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.16', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:28:04.874987 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:28:04.883648 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.008s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:28:04.884155 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[e8096de5-574d-48f6-b3b3-d85ef969e754]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:92f8692cf1e\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:28:04.885670 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:28:04.892580 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 26 21:28:04.893369 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[3a71179c-f2f4-4511-8893-34adab75c4ba]: [Errno 2] No such file or directory: 'systool' {{(pid=163609) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 26 21:28:04.893369 ncdv-master-1 cinder-backup[134755]: Traceback (most recent call last): Mar 26 21:28:04.893369 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 26 21:28:04.893369 ncdv-master-1 cinder-backup[134755]: ret = func(*f_args, **f_kwargs) Mar 26 21:28:04.893369 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 26 21:28:04.893369 ncdv-master-1 cinder-backup[134755]: return func(*args, **kwargs) Mar 26 21:28:04.893369 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 26 21:28:04.893369 ncdv-master-1 cinder-backup[134755]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 26 21:28:04.893369 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 26 21:28:04.893369 ncdv-master-1 cinder-backup[134755]: return putils.execute(on_execute=on_execute, Mar 26 21:28:04.893369 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 26 21:28:04.893369 ncdv-master-1 cinder-backup[134755]: obj = subprocess.Popen(cmd, Mar 26 21:28:04.893369 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 26 21:28:04.893369 ncdv-master-1 cinder-backup[134755]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 26 21:28:04.893369 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 26 21:28:04.893369 ncdv-master-1 cinder-backup[134755]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 26 21:28:04.893369 ncdv-master-1 cinder-backup[134755]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 26 21:28:04.894680 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[3a71179c-f2f4-4511-8893-34adab75c4ba]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:28:04.894997 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:28:04.901873 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 26 21:28:04.902278 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[b4146504-4f2c-4b16-ab75-16d1f1864fc3]: [Errno 2] No such file or directory: 'systool' {{(pid=163609) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 26 21:28:04.902278 ncdv-master-1 cinder-backup[134755]: Traceback (most recent call last): Mar 26 21:28:04.902278 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 26 21:28:04.902278 ncdv-master-1 cinder-backup[134755]: ret = func(*f_args, **f_kwargs) Mar 26 21:28:04.902278 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 26 21:28:04.902278 ncdv-master-1 cinder-backup[134755]: return func(*args, **kwargs) Mar 26 21:28:04.902278 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 26 21:28:04.902278 ncdv-master-1 cinder-backup[134755]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 26 21:28:04.902278 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 26 21:28:04.902278 ncdv-master-1 cinder-backup[134755]: return putils.execute(on_execute=on_execute, Mar 26 21:28:04.902278 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 26 21:28:04.902278 ncdv-master-1 cinder-backup[134755]: obj = subprocess.Popen(cmd, Mar 26 21:28:04.902278 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 26 21:28:04.902278 ncdv-master-1 cinder-backup[134755]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 26 21:28:04.902278 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 26 21:28:04.902278 ncdv-master-1 cinder-backup[134755]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 26 21:28:04.902278 ncdv-master-1 cinder-backup[134755]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 26 21:28:04.903364 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[b4146504-4f2c-4b16-ab75-16d1f1864fc3]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:28:04.904512 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:28:04.914316 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.010s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:28:04.914989 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[e7a90997-b6ed-4d90-a52f-9a122b75f2f5]: (4, ('/dev/sda1\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:28:04.916480 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:28:04.927057 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.010s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:28:04.927793 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[01d02fd9-7125-451f-a3a0-b8354c6fc506]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:28:04.929402 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:28:04.938876 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.009s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:28:04.939217 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[8888402d-536f-452b-ab07-6339c83aff59]: (4, ('c6a66835-8686-c942-a93c-00f6321af5e6\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:28:04.940009 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] Running cmd (subprocess): nvme version {{(pid=134755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:28:04.960420 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] CMD "nvme version" returned: 0 in 0.020s {{(pid=134755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:28:04.962208 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134755) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 26 21:28:04.962577 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134755) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 26 21:28:04.963756 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:44f9348f-d915-47a9-a95c-57bf517d9528 dsc: {{(pid=134755) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 26 21:28:04.964321 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.utils [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] <== get_connector_properties: return (89ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.16', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:92f8692cf1e', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c6a66835-8686-c942-a93c-00f6321af5e6', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:44f9348f-d915-47a9-a95c-57bf517d9528', 'found_dsc': ''} {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:28:05.712076 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connector [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] Factory for rbd on None {{(pid=134755) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 26 21:28:05.713637 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-a8b9669a-47aa-4380-95db-03adf51809b5', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'a8b9669a-47aa-4380-95db-03adf51809b5', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:28:05.714829 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.linuxrbd [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] opening connection to ceph cluster (timeout=-1). {{(pid=134755) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 26 21:28:05.786194 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] <== connect_volume: return (72ms) {'path': } {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:28:05.796303 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_db.sqlalchemy.engines [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134755) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 26 21:28:05.816224 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] Starting backup of volume='a8b9669a-47aa-4380-95db-03adf51809b5'. {{(pid=134755) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Mar 26 21:28:05.818371 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] Volume file is RBD: attempting optimized backup {{(pid=134755) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Mar 26 21:28:05.850508 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] Creating base image 'volume-a8b9669a-47aa-4380-95db-03adf51809b5.backup.5e7cf4fe-3c96-4331-a08b-f822d28b0141' {{(pid=134755) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Mar 26 21:28:05.851282 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134755) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Mar 26 21:28:05.892529 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] Using --from-snap 'None' for incremental backup of volume a8b9669a-47aa-4380-95db-03adf51809b5. {{(pid=134755) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Mar 26 21:28:05.892929 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] Creating backup snapshot='backup.5e7cf4fe-3c96-4331-a08b-f822d28b0141.snap.1648330085.892543' {{(pid=134755) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Mar 26 21:28:05.980178 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] Performing differential transfer from 'volume-a8b9669a-47aa-4380-95db-03adf51809b5' to 'volume-a8b9669a-47aa-4380-95db-03adf51809b5.backup.5e7cf4fe-3c96-4331-a08b-f822d28b0141' {{(pid=134755) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Mar 26 21:28:05.980877 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_o3k8font --pool volumes volumes/volume-a8b9669a-47aa-4380-95db-03adf51809b5@backup.5e7cf4fe-3c96-4331-a08b-f822d28b0141.snap.1648330085.892543 -' into... {{(pid=134755) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Mar 26 21:28:05.981240 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-a8b9669a-47aa-4380-95db-03adf51809b5.backup.5e7cf4fe-3c96-4331-a08b-f822d28b0141' {{(pid=134755) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Mar 26 21:28:11.148131 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] Differential backup transfer completed in 5.1674s {{(pid=134755) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Mar 26 21:28:11.148871 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] Getting metadata type 'volume-base-metadata' {{(pid=134755) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Mar 26 21:28:11.162536 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134755) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Mar 26 21:28:11.163377 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] Getting metadata type 'volume-metadata' {{(pid=134755) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Mar 26 21:28:11.171051 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] No metadata type 'volume-metadata' available {{(pid=134755) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Mar 26 21:28:11.171880 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] Getting metadata type 'volume-glance-metadata' {{(pid=134755) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Mar 26 21:28:11.179909 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=134755) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}} Mar 26 21:28:11.180649 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] Backing up metadata for volume a8b9669a-47aa-4380-95db-03adf51809b5. {{(pid=134755) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Mar 26 21:28:11.214947 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] Backup '5e7cf4fe-3c96-4331-a08b-f822d28b0141' of volume a8b9669a-47aa-4380-95db-03adf51809b5 finished. {{(pid=134755) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Mar 26 21:28:11.216836 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-a8b9669a-47aa-4380-95db-03adf51809b5', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'a8b9669a-47aa-4380-95db-03adf51809b5', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:28:11.241809 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] <== disconnect_volume: return (24ms) None {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:28:11.299577 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-630bd3e1-88a9-4ad4-b37e-064c8fcce968 tempest-TestVolumeBackupRestore-1318254806 None] Create backup finished. backup: 5e7cf4fe-3c96-4331-a08b-f822d28b0141. Mar 26 21:28:13.337619 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] Restore backup started, backup: 5e7cf4fe-3c96-4331-a08b-f822d28b0141 volume: bfbd40d8-0fc0-42c7-a34d-717f8a141508. Mar 26 21:28:13.351746 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.utils [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.16', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:28:13.353185 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:28:13.361049 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.008s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:28:13.361565 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[b572990e-ae07-4377-8911-8c64906cc4b2]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:92f8692cf1e\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:28:13.362882 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:28:13.369185 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 26 21:28:13.370220 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[ef928311-c530-4cfb-8e9b-9978efba79b3]: [Errno 2] No such file or directory: 'systool' {{(pid=163609) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 26 21:28:13.370220 ncdv-master-1 cinder-backup[134755]: Traceback (most recent call last): Mar 26 21:28:13.370220 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 26 21:28:13.370220 ncdv-master-1 cinder-backup[134755]: ret = func(*f_args, **f_kwargs) Mar 26 21:28:13.370220 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 26 21:28:13.370220 ncdv-master-1 cinder-backup[134755]: return func(*args, **kwargs) Mar 26 21:28:13.370220 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 26 21:28:13.370220 ncdv-master-1 cinder-backup[134755]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 26 21:28:13.370220 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 26 21:28:13.370220 ncdv-master-1 cinder-backup[134755]: return putils.execute(on_execute=on_execute, Mar 26 21:28:13.370220 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 26 21:28:13.370220 ncdv-master-1 cinder-backup[134755]: obj = subprocess.Popen(cmd, Mar 26 21:28:13.370220 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 26 21:28:13.370220 ncdv-master-1 cinder-backup[134755]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 26 21:28:13.370220 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 26 21:28:13.370220 ncdv-master-1 cinder-backup[134755]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 26 21:28:13.370220 ncdv-master-1 cinder-backup[134755]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 26 21:28:13.371492 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[ef928311-c530-4cfb-8e9b-9978efba79b3]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:28:13.371671 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:28:13.377031 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 26 21:28:13.377637 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[f89db77e-4f99-435f-aa18-b7bebae25e68]: [Errno 2] No such file or directory: 'systool' {{(pid=163609) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 26 21:28:13.377637 ncdv-master-1 cinder-backup[134755]: Traceback (most recent call last): Mar 26 21:28:13.377637 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 26 21:28:13.377637 ncdv-master-1 cinder-backup[134755]: ret = func(*f_args, **f_kwargs) Mar 26 21:28:13.377637 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 26 21:28:13.377637 ncdv-master-1 cinder-backup[134755]: return func(*args, **kwargs) Mar 26 21:28:13.377637 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 26 21:28:13.377637 ncdv-master-1 cinder-backup[134755]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 26 21:28:13.377637 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 26 21:28:13.377637 ncdv-master-1 cinder-backup[134755]: return putils.execute(on_execute=on_execute, Mar 26 21:28:13.377637 ncdv-master-1 cinder-backup[134755]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 26 21:28:13.377637 ncdv-master-1 cinder-backup[134755]: obj = subprocess.Popen(cmd, Mar 26 21:28:13.377637 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 26 21:28:13.377637 ncdv-master-1 cinder-backup[134755]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 26 21:28:13.377637 ncdv-master-1 cinder-backup[134755]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 26 21:28:13.377637 ncdv-master-1 cinder-backup[134755]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 26 21:28:13.377637 ncdv-master-1 cinder-backup[134755]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 26 21:28:13.378958 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[f89db77e-4f99-435f-aa18-b7bebae25e68]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:28:13.379356 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:28:13.387426 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.008s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:28:13.387767 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[981bb4d2-60e7-450e-97c4-29dcacc0cc67]: (4, ('/dev/sda1\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:28:13.389385 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:28:13.397414 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.008s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:28:13.397753 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[293b2c65-800b-4d45-8aeb-913eb782c5f6]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:28:13.398826 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:28:13.405952 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.007s {{(pid=163609) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:28:13.406286 ncdv-master-1 cinder-backup[134755]: DEBUG oslo.privsep.daemon [-] privsep: reply[5f7bf380-6525-4a5d-b5bb-cba450245ec9]: (4, ('c6a66835-8686-c942-a93c-00f6321af5e6\n', '')) {{(pid=163609) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 26 21:28:13.407086 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] Running cmd (subprocess): nvme version {{(pid=134755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 26 21:28:13.426345 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_concurrency.processutils [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] CMD "nvme version" returned: 0 in 0.019s {{(pid=134755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 26 21:28:13.428341 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134755) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 26 21:28:13.428723 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134755) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 26 21:28:13.429108 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.lightos [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:44f9348f-d915-47a9-a95c-57bf517d9528 dsc: {{(pid=134755) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 26 21:28:13.429748 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.utils [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] <== get_connector_properties: return (77ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.16', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:92f8692cf1e', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c6a66835-8686-c942-a93c-00f6321af5e6', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:44f9348f-d915-47a9-a95c-57bf517d9528', 'found_dsc': ''} {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:28:14.137521 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connector [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] Factory for rbd on None {{(pid=134755) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 26 21:28:14.138987 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-bfbd40d8-0fc0-42c7-a34d-717f8a141508', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'bfbd40d8-0fc0-42c7-a34d-717f8a141508', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:28:14.140080 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.linuxrbd [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] opening connection to ceph cluster (timeout=-1). {{(pid=134755) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 26 21:28:14.178818 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] <== connect_volume: return (39ms) {'path': } {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:28:14.187324 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_db.sqlalchemy.engines [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134755) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 26 21:28:14.197863 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] Starting restore from Ceph backup=5e7cf4fe-3c96-4331-a08b-f822d28b0141 to volume=volume-bfbd40d8-0fc0-42c7-a34d-717f8a141508 {{(pid=134755) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1274}} Mar 26 21:28:14.255910 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] Looking for snapshot of backup base 'volume-a8b9669a-47aa-4380-95db-03adf51809b5.backup.5e7cf4fe-3c96-4331-a08b-f822d28b0141' {{(pid=134755) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 26 21:28:14.256315 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] Found snapshot 'backup.5e7cf4fe-3c96-4331-a08b-f822d28b0141.snap.1648330085.892543' {{(pid=134755) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Mar 26 21:28:14.260731 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] Volume file is RBD. {{(pid=134755) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1184}} Mar 26 21:28:14.292141 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] Attempting differential restore. {{(pid=134755) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1234}} Mar 26 21:28:14.292584 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] Attempting incremental restore from base='volume-a8b9669a-47aa-4380-95db-03adf51809b5.backup.5e7cf4fe-3c96-4331-a08b-f822d28b0141' snap='backup.5e7cf4fe-3c96-4331-a08b-f822d28b0141.snap.1648330085.892543' {{(pid=134755) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1089}} Mar 26 21:28:14.292988 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] Performing differential transfer from 'volume-a8b9669a-47aa-4380-95db-03adf51809b5.backup.5e7cf4fe-3c96-4331-a08b-f822d28b0141' to 'volume-bfbd40d8-0fc0-42c7-a34d-717f8a141508' {{(pid=134755) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Mar 26 21:28:14.293444 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-a8b9669a-47aa-4380-95db-03adf51809b5.backup.5e7cf4fe-3c96-4331-a08b-f822d28b0141@backup.5e7cf4fe-3c96-4331-a08b-f822d28b0141.snap.1648330085.892543 -' into... {{(pid=134755) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Mar 26 21:28:14.293818 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_gkh6tq86 --pool volumes - volumes/volume-bfbd40d8-0fc0-42c7-a34d-717f8a141508' {{(pid=134755) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Mar 26 21:28:17.671379 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] Restore transfer completed in 3.3778s {{(pid=134755) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1111}} Mar 26 21:28:17.672423 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=134755) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1285}} Mar 26 21:28:17.694405 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] Restoring volume base metadata {{(pid=134755) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} Mar 26 21:28:17.715844 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] No metadata of type 'volume-metadata' to restore {{(pid=134755) put /opt/stack/cinder/cinder/backup/driver.py:345}} Mar 26 21:28:17.716332 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.driver [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] Restoring volume glance metadata {{(pid=134755) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}} Mar 26 21:28:17.848342 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] Restore to volume bfbd40d8-0fc0-42c7-a34d-717f8a141508 finished successfully. {{(pid=134755) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1292}} Mar 26 21:28:17.849846 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-bfbd40d8-0fc0-42c7-a34d-717f8a141508', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'bfbd40d8-0fc0-42c7-a34d-717f8a141508', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 26 21:28:17.859737 ncdv-master-1 cinder-backup[134755]: DEBUG os_brick.initiator.connectors.rbd [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] <== disconnect_volume: return (9ms) None {{(pid=134755) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 26 21:28:17.916258 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.manager [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] Encryption key ID for volume bfbd40d8-0fc0-42c7-a34d-717f8a141508 already matches encryption key ID in backup 5e7cf4fe-3c96-4331-a08b-f822d28b0141. {{(pid=134755) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} Mar 26 21:28:17.945254 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-16c62633-944d-4a08-887c-7c36d307b410 tempest-TestVolumeBackupRestore-1318254806 None] Finished restoring backup 5e7cf4fe-3c96-4331-a08b-f822d28b0141 to volume bfbd40d8-0fc0-42c7-a34d-717f8a141508. Mar 26 21:28:35.849715 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:28:35.850399 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:29:21.514307 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-5e9dc326-6567-47fb-bc99-5364c5df9036 tempest-TestVolumeBackupRestore-1318254806 None] Delete backup started, backup: 5e7cf4fe-3c96-4331-a08b-f822d28b0141. Mar 26 21:29:21.516459 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-5e9dc326-6567-47fb-bc99-5364c5df9036 tempest-TestVolumeBackupRestore-1318254806 None] Delete started for backup=5e7cf4fe-3c96-4331-a08b-f822d28b0141 {{(pid=134755) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Mar 26 21:29:21.516921 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-5e9dc326-6567-47fb-bc99-5364c5df9036 tempest-TestVolumeBackupRestore-1318254806 None] Trying diff format basename='volume-a8b9669a-47aa-4380-95db-03adf51809b5.backup.5e7cf4fe-3c96-4331-a08b-f822d28b0141' for backup base image of volume a8b9669a-47aa-4380-95db-03adf51809b5. {{(pid=134755) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Mar 26 21:29:21.572832 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-5e9dc326-6567-47fb-bc99-5364c5df9036 tempest-TestVolumeBackupRestore-1318254806 None] Looking for snapshot of backup base 'volume-a8b9669a-47aa-4380-95db-03adf51809b5.backup.5e7cf4fe-3c96-4331-a08b-f822d28b0141' {{(pid=134755) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 26 21:29:21.573312 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-5e9dc326-6567-47fb-bc99-5364c5df9036 tempest-TestVolumeBackupRestore-1318254806 None] Found snapshot 'backup.5e7cf4fe-3c96-4331-a08b-f822d28b0141.snap.1648330085.892543' {{(pid=134755) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Mar 26 21:29:21.573797 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-5e9dc326-6567-47fb-bc99-5364c5df9036 tempest-TestVolumeBackupRestore-1318254806 None] Deleting backup snapshot='backup.5e7cf4fe-3c96-4331-a08b-f822d28b0141.snap.1648330085.892543' {{(pid=134755) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Mar 26 21:29:21.947579 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.drivers.ceph [None req-5e9dc326-6567-47fb-bc99-5364c5df9036 tempest-TestVolumeBackupRestore-1318254806 None] Deleting backup base image='volume-a8b9669a-47aa-4380-95db-03adf51809b5.backup.5e7cf4fe-3c96-4331-a08b-f822d28b0141' of volume a8b9669a-47aa-4380-95db-03adf51809b5. Mar 26 21:29:22.465704 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-5e9dc326-6567-47fb-bc99-5364c5df9036 tempest-TestVolumeBackupRestore-1318254806 None] Base backup image='volume-a8b9669a-47aa-4380-95db-03adf51809b5.backup.5e7cf4fe-3c96-4331-a08b-f822d28b0141' of volume a8b9669a-47aa-4380-95db-03adf51809b5 deleted. {{(pid=134755) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Mar 26 21:29:22.509481 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.backup.drivers.ceph [None req-5e9dc326-6567-47fb-bc99-5364c5df9036 tempest-TestVolumeBackupRestore-1318254806 None] Delete of backup '5e7cf4fe-3c96-4331-a08b-f822d28b0141' for volume 'a8b9669a-47aa-4380-95db-03adf51809b5' finished. {{(pid=134755) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Mar 26 21:29:22.515594 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_db.sqlalchemy.engines [None req-5e9dc326-6567-47fb-bc99-5364c5df9036 tempest-TestVolumeBackupRestore-1318254806 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134755) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 26 21:29:22.538988 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.quota [None req-5e9dc326-6567-47fb-bc99-5364c5df9036 tempest-TestVolumeBackupRestore-1318254806 None] Created reservations ['7fb94e90-5d44-424f-afe6-ced082dca3f3', 'e8666ec7-5119-453a-8dea-e5a52ef7ad3f'] {{(pid=134755) reserve /opt/stack/cinder/cinder/quota.py:755}} Mar 26 21:29:22.564221 ncdv-master-1 cinder-backup[134755]: INFO cinder.backup.manager [None req-5e9dc326-6567-47fb-bc99-5364c5df9036 tempest-TestVolumeBackupRestore-1318254806 None] Delete backup finished, backup 5e7cf4fe-3c96-4331-a08b-f822d28b0141 deleted. Mar 26 21:29:35.845917 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:29:35.846564 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:30:35.850757 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:30:35.851439 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:31:35.856328 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:31:35.857069 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:32:35.861914 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:32:35.862604 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:33:35.866490 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:33:35.867218 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:34:35.871830 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:34:35.872522 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:35:35.876962 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:35:35.877603 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:36:35.886222 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:36:35.886891 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:37:35.881744 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:37:35.883068 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:38:35.881261 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:38:35.881955 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:39:35.886475 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:39:35.887196 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:40:35.891812 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:40:35.892589 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:41:35.896689 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:41:35.897476 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:42:35.894725 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:42:35.895391 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:43:35.899753 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:43:35.900468 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:44:35.904909 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:44:35.905614 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:45:35.910599 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:45:35.911327 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:46:35.916237 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:46:35.916961 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:47:35.921404 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:47:35.922073 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:48:35.916779 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:48:35.917481 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:49:35.921891 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:49:35.922553 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:50:35.926685 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:50:35.928031 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:51:35.932566 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:51:35.933267 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:52:35.938319 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:52:35.939086 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:53:35.943522 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:53:35.944259 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:54:35.948714 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:54:35.949421 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:55:35.953238 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:55:35.953960 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:56:35.958534 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:56:35.959224 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:57:35.963722 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:57:35.964438 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:58:35.968185 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:58:35.968912 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 21:59:35.973811 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 21:59:35.974481 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 22:00:35.979162 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 22:00:35.979800 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 22:01:35.984129 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 22:01:35.984774 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 22:02:35.989615 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 22:02:35.990227 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 22:03:35.994866 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 22:03:35.995490 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 22:04:35.999994 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 22:04:36.000672 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 22:05:36.004954 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 22:05:36.005594 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 22:06:36.011109 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 22:06:36.011753 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 22:07:36.016879 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 22:07:36.017519 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 26 22:08:36.021658 ncdv-master-1 cinder-backup[134755]: DEBUG oslo_service.periodic_task [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134755) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 26 22:08:36.022280 ncdv-master-1 cinder-backup[134755]: DEBUG cinder.manager [None req-917a0951-c617-4a91-a93f-569fd2ec6901 None None] Notifying Schedulers of capabilities ... {{(pid=134755) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}