-- Logs begin at Sat 2022-04-02 20:01:18 UTC, end at Sat 2022-04-02 22:19:51 UTC. -- Apr 02 20:42:32.011870 ncdv-master-1 systemd[1]: Started Devstack devstack@c-bak.service. Apr 02 20:42:35.312527 ncdv-master-1 cinder-backup[129253]: INFO cinder.cmd.backup [-] Backup running with 1 processes. Apr 02 20:42:35.314248 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_concurrency.lockutils [-] Acquired lock "singleton_lock" {{(pid=129253) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:294}} Apr 02 20:42:35.314777 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_concurrency.lockutils [-] Releasing lock "singleton_lock" {{(pid=129253) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:312}} Apr 02 20:42:35.470529 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_db.sqlalchemy.engines [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129253) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 02 20:42:35.688847 ncdv-master-1 cinder-backup[129253]: INFO cinder.rpc [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] Automatically selected cinder-backup objects version 1.39 as minimum service version. Apr 02 20:42:35.692760 ncdv-master-1 cinder-backup[129253]: INFO cinder.rpc [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] Automatically selected cinder-backup RPC version 2.3 as minimum service version. Apr 02 20:42:35.696691 ncdv-master-1 cinder-backup[129253]: INFO cinder.rpc [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] Automatically selected cinder-volume objects version 1.39 as minimum service version. Apr 02 20:42:35.700244 ncdv-master-1 cinder-backup[129253]: INFO cinder.rpc [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] Automatically selected cinder-volume RPC version 3.18 as minimum service version. Apr 02 20:42:35.710150 ncdv-master-1 cinder-backup[129253]: INFO cinder.rpc [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] Automatically selected cinder-scheduler objects version 1.39 as minimum service version. Apr 02 20:42:35.710150 ncdv-master-1 cinder-backup[129253]: INFO cinder.rpc [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] Automatically selected cinder-scheduler RPC version 3.12 as minimum service version. Apr 02 20:42:35.762914 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_db.sqlalchemy.engines [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129253) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 02 20:42:35.786563 ncdv-master-1 cinder-backup[129253]: INFO oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] Starting 1 workers Apr 02 20:42:35.791789 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] Started child 129697 {{(pid=129253) _start_child /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:575}} Apr 02 20:42:35.793124 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] Full set of CONF: {{(pid=129253) wait /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:649}} Apr 02 20:42:35.793262 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] ******************************************************************************** {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Apr 02 20:42:35.793545 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] Configuration options gathered from: {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Apr 02 20:42:35.794472 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] command line args: ['--config-file', '/etc/cinder/cinder.conf'] {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Apr 02 20:42:35.795181 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] config files: ['/etc/cinder/cinder.conf'] {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Apr 02 20:42:35.795575 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] ================================================================================ {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Apr 02 20:42:35.796255 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] allow_availability_zone_fallback = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.797308 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] allow_compression_on_image_upload = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.797902 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] allowed_direct_url_schemes = [] {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.798307 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] api_rate_limit = True {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.798808 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] auth_strategy = keystone {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.799207 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_availability_zone = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.799596 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backup_api_class = cinder.backup.api.API {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.799997 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backup_ceph_chunk_size = 134217728 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.804497 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backup_ceph_conf = /etc/ceph/ceph.conf {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.804966 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backup_ceph_image_journals = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.806075 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backup_ceph_pool = backups {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.806731 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backup_ceph_stripe_count = 0 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.807183 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backup_ceph_stripe_unit = 0 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.807501 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backup_ceph_user = cinder-bak {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.807901 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backup_driver = cinder.backup.drivers.ceph.CephBackupDriver {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.808503 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backup_driver_init_check_interval = 60 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.814590 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backup_driver_stats_polling_interval = 60 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.815042 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backup_manager = cinder.backup.manager.BackupManager {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.815762 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backup_max_operations = 15 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.824118 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backup_metadata_version = 2 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.824118 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backup_name_template = backup-%s {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.824118 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backup_native_threads_pool_size = 60 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.824118 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backup_object_number_per_notification = 10 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.824741 ncdv-master-1 cinder-backup[129697]: INFO cinder.service [-] Starting cinder-backup node (version 20.1.0) Apr 02 20:42:35.825223 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backup_service_inithost_offload = True {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.825223 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backup_timer_interval = 120 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.825223 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backup_use_same_host = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.825223 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backup_use_temp_snapshot = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.825223 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backup_workers = 1 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.825223 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] chap_password = **** {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.831443 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] chap_username = {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.831839 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.832345 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cinder_internal_tenant_project_id = dc74fe8a08da402d9cd82fd42d3f8f20 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.832775 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cinder_internal_tenant_user_id = 64e876b0e0844090aa652c98af67ab8b {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.833268 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] compression_format = gzip {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.833961 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] config_dir = [] {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.834299 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] config_file = ['/etc/cinder/cinder.conf'] {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.834957 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] config_source = [] {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.835321 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] consistencygroup_api_class = cinder.consistencygroup.api.API {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.835723 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] control_exchange = openstack {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.836151 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] debug = True {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.837113 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] default_availability_zone = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.838687 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] default_group_type = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.839290 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'swiftclient=WARN', 'botocore=WARN'] {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.841318 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] default_volume_type = ceph {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.842964 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] driver_client_cert = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.843328 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] driver_client_cert_key = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.843681 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] driver_data_namespace = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.847156 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] driver_ssl_cert_path = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.847562 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] driver_ssl_cert_verify = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.847924 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] driver_use_ssl = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.848245 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] enable_new_services = True {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.848632 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] enable_unsupported_driver = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.849039 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] enabled_backends = ['ceph'] {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.851565 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] enforce_multipath_for_image_xfer = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.851918 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] filter_function = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.852402 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] glance_api_insecure = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.852957 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] glance_api_servers = ['http://10.222.0.18/image'] {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.853161 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] glance_api_ssl_compression = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.853558 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] glance_ca_certificates_file = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.853936 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] glance_catalog_info = image:glance:publicURL {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.854291 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] glance_certfile = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.861440 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] glance_core_properties = ['checksum', 'container_format', 'disk_format', 'image_name', 'image_id', 'min_disk', 'min_ram', 'name', 'size'] {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.861440 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] glance_keyfile = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.861440 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] glance_num_retries = 3 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.861440 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] glance_request_timeout = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.861440 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] goodness_function = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.861440 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] graceful_shutdown_timeout = 5 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.862039 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] group_api_class = cinder.group.api.API {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.862039 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] host = ncdv-master-1 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.862391 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] iet_conf = /etc/iet/ietd.conf {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.862860 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] image_compress_on_upload = True {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.863638 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] image_conversion_address_space_limit = 1 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.864496 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] image_conversion_cpu_limit = 60 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.868615 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] image_conversion_dir = /opt/stack/data/cinder/conversion {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.868986 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] image_upload_use_cinder_backend = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.869364 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] image_upload_use_internal_tenant = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.872661 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] image_volume_cache_enabled = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.873050 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] image_volume_cache_max_count = 0 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.873405 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] image_volume_cache_max_size_gb = 0 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.873792 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] instance_format = [instance: %(uuid)s] {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.874157 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] instance_uuid_format = [instance: %(uuid)s] {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.874542 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] iscsi_iotype = fileio {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.875124 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] iscsi_secondary_ip_addresses = [] {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.875486 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] iscsi_target_flags = {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.875843 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] iscsi_write_cache = on {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.876193 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] iser_helper = tgtadm {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.876833 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] iser_ip_address = 10.222.0.18 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.877237 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] iser_port = 3260 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.878931 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.lockutils [None req-d7f3f213-319b-482b-a14a-8652a5ed5609 None None] Lock "cleanup_incomplete_backups_129253" acquired by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: waited 0.006s {{(pid=129697) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:386}} Apr 02 20:42:35.879279 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-d7f3f213-319b-482b-a14a-8652a5ed5609 None None] Cleaning up incomplete backup operations. Apr 02 20:42:35.884348 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.884873 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] log_config_append = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.885477 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.885944 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] log_dir = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.886395 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] log_file = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.886694 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] log_options = True {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.887052 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] log_rotate_interval = 1 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.887492 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] log_rotate_interval_type = days {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.888034 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] log_rotation_type = none {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.888264 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.888627 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.889005 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.889463 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.891385 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] logging_user_identity_format = %(user)s %(project)s %(domain)s %(user_domain)s %(project_domain)s {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.891823 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] max_age = 0 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.892181 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] max_logfile_count = 30 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.892576 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] max_logfile_size_mb = 200 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.894187 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] max_over_subscription_ratio = 20.0 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.894571 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] message_reap_interval = 86400 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.897039 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] message_ttl = 2592000 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.897416 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] monkey_patch = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.897839 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] monkey_patch_modules = [] {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.898139 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] my_ip = 10.222.0.18 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.898495 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] no_snapshot_gb_quota = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.898842 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] num_iser_scan_tries = 3 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.901903 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] num_shell_tries = 3 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.902272 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] num_volume_device_scan_tries = 3 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.902636 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] nvmet_ns_id = 10 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.903087 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] nvmet_port_id = 1 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.903400 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] osapi_max_limit = 1000 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.912591 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] osapi_volume_ext_list = [] {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.912591 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] osapi_volume_extension = ['cinder.api.contrib.standard_extensions'] {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.912591 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] osapi_volume_listen = 0.0.0.0 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.912591 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] osapi_volume_listen_port = 8776 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.912591 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] osapi_volume_use_ssl = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.912591 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] osapi_volume_workers = 2 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.913044 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] per_volume_size_limit = -1 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.913044 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] periodic_fuzzy_delay = 60 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.913438 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] periodic_interval = 60 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.913881 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] project_id_regex = [0-9a-f\-]+ {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.914376 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] public_endpoint = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.914664 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] publish_errors = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.918382 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] quota_backup_gigabytes = 1000 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.918745 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] quota_backups = 10 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.919115 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] quota_consistencygroups = 10 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.919501 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] quota_driver = cinder.quota.DbQuotaDriver {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.919855 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] quota_gigabytes = 1000 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.921971 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] quota_groups = 10 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.922355 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] quota_snapshots = 10 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.922724 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] quota_volumes = 10 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.923083 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] rate_limit_burst = 0 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.923454 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] rate_limit_except_level = CRITICAL {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.923807 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] rate_limit_interval = 0 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.924419 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] replication_device = **** {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.924564 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] report_discard_supported = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.924903 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] report_interval = 10 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.925357 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] reservation_clean_interval = 86400 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.926672 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] reservation_expire = 86400 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.927003 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] reserved_percentage = 0 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.927375 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] resource_query_filters_file = /etc/cinder/resource_filters.json {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.927900 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] restore_discard_excess_bytes = True {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.928310 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] rootwrap_config = /etc/cinder/rootwrap.conf {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.928727 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] rpc_response_timeout = 120 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.929235 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] run_external_periodic_tasks = True {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.929627 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] scheduler_manager = cinder.scheduler.manager.SchedulerManager {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.930000 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] scst_target_driver = iscsi {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.930371 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] scst_target_iqn_name = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.930665 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] service_down_time = 60 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.931042 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] snapshot_name_template = snapshot-%s {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.931392 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] split_loggers = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.931685 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] state_path = /opt/stack/data/cinder {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.932045 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] storage_availability_zone = nova {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.932405 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] storage_protocol = iscsi {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.932784 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] syslog_log_facility = LOG_USER {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.933139 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] target_helper = lioadm {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.934168 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] target_ip_address = 10.222.0.18 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.934544 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] target_port = 3260 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.934897 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] target_prefix = iqn.2010-10.org.openstack: {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.935285 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] target_protocol = iscsi {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.935630 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] trace_flags = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.936524 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] transfer_api_class = cinder.transfer.api.API {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.936883 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] transport_url = **** {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.937233 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] until_refresh = 0 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.937604 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] use_chap_auth = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.937950 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] use_default_quota_class = True {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.938290 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] use_eventlog = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.938638 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] use_journal = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.938981 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] use_json = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.939320 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] use_multipath_for_image_xfer = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.939664 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] use_stderr = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.940252 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] use_syslog = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.940669 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] verify_glance_signatures = enabled {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.941040 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] volume_api_class = cinder.volume.api.API {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.941378 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] volume_backend_name = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.941762 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] volume_clear = zero {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.942094 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] volume_clear_ionice = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.942440 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] volume_clear_size = 0 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.942822 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.943129 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] volume_copy_bps_limit = 0 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.943483 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] volume_dd_blocksize = 1M {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.943834 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] volume_manager = cinder.volume.manager.VolumeManager {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.944293 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] volume_name_template = volume-%s {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.945249 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] volume_usage_audit_period = month {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.945249 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] volumes_dir = /opt/stack/data/cinder/volumes {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.945564 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] watch_log_file = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 02 20:42:35.947414 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.947786 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.948196 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_policy.enforce_new_defaults = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.948599 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_policy.enforce_scope = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.949006 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_policy.policy_default_rule = default {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.949410 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_policy.policy_dirs = ['policy.d'] {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.949856 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_policy.policy_file = policy.yaml {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.950260 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.950604 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.950946 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.951290 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.951688 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.952069 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_concurrency.disable_process_locking = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.952479 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_concurrency.lock_path = /opt/stack/data/cinder {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.952887 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.admin_password = **** {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.953251 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.admin_tenant_name = admin {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.953589 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.admin_token = **** {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.953973 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.admin_user = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.954351 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.auth_admin_prefix = {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.954704 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.956197 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.auth_port = 35357 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.956197 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.auth_protocol = https {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.956197 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.auth_section = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.956197 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.auth_type = password {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.957340 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.auth_uri = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.957690 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.auth_version = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.958036 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.cache = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.958410 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.cafile = /opt/stack/data/ca-bundle.pem {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.959037 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.certfile = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.959384 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.delay_auth_decision = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.959766 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.enforce_token_bind = permissive {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.960346 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.http_connect_timeout = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.960520 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.http_request_max_retries = 3 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.960839 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.identity_uri = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.961202 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.include_service_catalog = True {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.962105 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.insecure = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.962465 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.interface = public {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.962818 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.keyfile = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.963293 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.963666 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.964365 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.964741 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.965145 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.965424 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.memcache_secret_key = **** {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.965783 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.memcache_security_strategy = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.966130 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.966530 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.memcached_servers = ['localhost:11211'] {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.967405 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.region_name = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.967780 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.service_token_roles = ['service'] {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.968175 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.service_token_roles_required = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.968564 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.service_type = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.968915 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.token_cache_time = 300 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.969308 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] keystone_authtoken.www_authenticate_uri = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.969747 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.backend = dogpile.cache.null {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.970128 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.backend_argument = **** {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.970707 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.config_prefix = cache.oslo {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.971120 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.dead_timeout = 60.0 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.971488 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.debug_cache_backend = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.971836 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.enable_retry_client = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.972229 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.enable_socket_keepalive = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.972694 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.enabled = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.973070 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.expiration_time = 600 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.973409 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.hashclient_retry_attempts = 2 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.973759 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.hashclient_retry_delay = 1.0 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.974107 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.memcache_dead_retry = 300 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.974470 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.memcache_pool_connection_get_timeout = 10 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.974869 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.memcache_pool_flush_on_reconnect = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.975347 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.memcache_pool_maxsize = 10 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.975622 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.memcache_pool_unused_timeout = 60 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.975996 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.memcache_servers = ['localhost:11211'] {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.976365 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.memcache_socket_timeout = 1.0 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.976769 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.proxies = [] {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.977974 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.retry_attempts = 2 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.978337 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.retry_delay = 0.0 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.978853 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.socket_keepalive_count = 1 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.979457 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.socket_keepalive_idle = 1 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.979457 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.socket_keepalive_interval = 1 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.979738 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.tls_allowed_ciphers = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.984476 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.tls_cafile = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.984476 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.tls_certfile = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.984476 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.tls_enabled = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.984476 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cache.tls_keyfile = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.984476 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] coordination.backend_url = etcd3+http://10.222.0.18:2379 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.984476 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] database.backend = sqlalchemy {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.984476 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] database.connection = **** {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.985366 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] database.connection_debug = 0 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.985366 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] database.connection_parameters = {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.985366 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] database.connection_recycle_time = 3600 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.985366 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] database.connection_trace = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.985366 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] database.db_inc_retry_interval = True {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.985366 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] database.db_max_retries = 20 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.985366 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] database.db_max_retry_interval = 10 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.986236 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] database.db_retry_interval = 1 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.986236 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] database.max_overflow = 50 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.986236 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] database.max_pool_size = 5 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.986586 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] database.max_retries = 10 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.987783 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] database.mysql_enable_ndb = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.988198 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] database.mysql_sql_mode = TRADITIONAL {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.988572 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] database.pool_timeout = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.988928 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] database.retry_interval = 10 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.989615 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] database.slave_connection = **** {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.990097 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] database.sqlite_synchronous = True {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.990448 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] database.use_db_reconnect = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.990842 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.991211 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.991592 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_metrics.metrics_process_name = {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.992166 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.993130 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.993669 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] profiler.connection_string = messaging:// {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.993969 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] profiler.enabled = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.994384 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] profiler.es_doc_type = notification {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.994732 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] profiler.es_scroll_size = 10000 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.995243 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] profiler.es_scroll_time = 2m {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.995625 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] profiler.filter_error_trace = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.995999 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] profiler.hmac_keys = SECRET_KEY {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.996419 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] profiler.sentinel_service_name = mymaster {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.996805 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] profiler.socket_timeout = 0.1 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.997287 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] profiler.trace_sqlalchemy = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.998076 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] privsep_osbrick.capabilities = [21] {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.998523 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] privsep_osbrick.group = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.998864 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] privsep_osbrick.helper_command = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.999237 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] privsep_osbrick.logger_name = oslo_privsep.daemon {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.999584 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] privsep_osbrick.thread_pool_size = 4 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:35.999961 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] privsep_osbrick.user = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.000352 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cinder_sys_admin.capabilities = [0, 1, 2, 3, 12, 21] {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.000952 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cinder_sys_admin.group = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.001299 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cinder_sys_admin.helper_command = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.001682 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cinder_sys_admin.logger_name = oslo_privsep.daemon {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.001997 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cinder_sys_admin.thread_pool_size = 4 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.002336 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] cinder_sys_admin.user = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.002703 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] service_user.auth_section = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.003053 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] service_user.auth_type = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.003387 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] service_user.cafile = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.003723 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] service_user.certfile = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.004092 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] service_user.collect_timing = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.004504 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] service_user.insecure = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.004844 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] service_user.keyfile = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.005186 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] service_user.send_service_user_token = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.005526 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] service_user.split_loggers = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.005867 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] service_user.timeout = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.006417 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.backend_availability_zone = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.006788 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.chap_password = **** {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.007235 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.chap_username = {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.007648 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.009145 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.deferred_deletion_delay = 0 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.009509 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.deferred_deletion_purge_interval = 60 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.009871 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.driver_client_cert = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.010214 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.driver_client_cert_key = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.010547 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.driver_data_namespace = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.010887 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.driver_ssl_cert_path = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.011227 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.driver_ssl_cert_verify = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.011573 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.driver_use_ssl = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.011915 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.enable_deferred_deletion = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.012256 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.enable_unsupported_driver = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.012639 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.enforce_multipath_for_image_xfer = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.012976 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.filter_function = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.013321 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.goodness_function = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.013703 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.iet_conf = /etc/iet/ietd.conf {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.014046 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.image_upload_use_cinder_backend = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.014388 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.image_upload_use_internal_tenant = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.014729 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.image_volume_cache_enabled = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.015078 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.image_volume_cache_max_count = 0 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.015417 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.image_volume_cache_max_size_gb = 0 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.015769 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.iscsi_iotype = fileio {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.016120 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.iscsi_secondary_ip_addresses = [] {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.016487 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.iscsi_target_flags = {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.016843 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.iscsi_write_cache = on {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.017193 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.iser_helper = tgtadm {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.018576 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.iser_ip_address = 10.222.0.18 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.018957 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.iser_port = 3260 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.019317 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.019807 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.max_over_subscription_ratio = 20.0 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.020041 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.num_iser_scan_tries = 3 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.020512 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.num_shell_tries = 3 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.020859 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.num_volume_device_scan_tries = 3 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.021200 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.nvmet_ns_id = 10 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.021567 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.nvmet_port_id = 1 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.022047 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.rados_connect_timeout = -1 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.022419 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.rados_connection_interval = 5 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.022761 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.rados_connection_retries = 3 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.023113 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.rbd_ceph_conf = {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.023498 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.rbd_cluster_name = ceph {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.023845 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.rbd_exclusive_cinder_pool = True {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.024262 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.rbd_flatten_volume_from_snapshot = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.024629 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.rbd_max_clone_depth = 5 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.025370 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.rbd_pool = rbd {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.025722 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.rbd_secret_uuid = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.026062 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.rbd_store_chunk_size = 4 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.026404 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.rbd_user = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.026753 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.replication_connect_timeout = 5 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.027089 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.replication_device = **** {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.027548 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.report_discard_supported = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.027937 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.report_dynamic_total_capacity = True {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.028729 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.reserved_percentage = 0 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.029096 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.scst_target_driver = iscsi {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.029441 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.scst_target_iqn_name = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.029796 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.storage_protocol = iscsi {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.030148 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.target_helper = tgtadm {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.030626 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.target_ip_address = 10.222.0.18 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.030973 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.target_port = 3260 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.031323 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.target_prefix = iqn.2010-10.org.openstack: {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.031676 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.target_protocol = iscsi {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.032013 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.trace_flags = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.032356 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.unique_fqdn_network = True {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.032735 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.use_chap_auth = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.033080 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.use_multipath_for_image_xfer = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.033415 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.volume_backend_name = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.034151 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.volume_clear = zero {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.034613 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.volume_clear_ionice = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.034962 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.volume_clear_size = 0 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.035315 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.035675 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.volume_copy_bps_limit = 0 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.036026 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.volume_dd_blocksize = 1M {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.036538 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] backend_defaults.volumes_dir = /opt/stack/data/cinder/volumes {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.036919 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_reports.file_event_handler = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.037303 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_reports.file_event_handler_interval = 1 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.037755 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_reports.log_dir = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.038218 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.038932 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.039306 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.040206 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.040600 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.040947 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.041323 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_rabbit.heartbeat_in_pthread = True {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.041919 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_rabbit.heartbeat_rate = 2 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.042389 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.042732 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.043111 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.043456 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.043830 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.044183 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.044552 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.044910 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.045246 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.045586 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.045956 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.046293 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.046654 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.047008 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.047397 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_rabbit.ssl = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.047753 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_rabbit.ssl_ca_file = {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.048258 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_rabbit.ssl_cert_file = {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.049331 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_rabbit.ssl_key_file = {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.049711 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_rabbit.ssl_version = {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.050145 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_notifications.driver = [] {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.050500 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_notifications.retry = -1 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.050910 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.051252 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] oslo_messaging_notifications.transport_url = **** {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.051694 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] key_manager.backend = cinder.keymgr.conf_key_mgr.ConfKeyManager {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.052123 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] barbican.auth_endpoint = http://localhost/identity/v3 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.052480 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] barbican.barbican_api_version = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.052884 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] barbican.barbican_endpoint = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.053262 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] barbican.barbican_endpoint_type = public {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.053603 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] barbican.barbican_region_name = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.053938 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] barbican.cafile = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.054277 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] barbican.certfile = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.054701 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] barbican.collect_timing = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.055047 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] barbican.insecure = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.055967 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] barbican.keyfile = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.056327 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] barbican.number_of_retries = 60 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.056736 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] barbican.retry_delay = 1 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.057090 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] barbican.send_service_user_token = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.057439 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] barbican.split_loggers = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.057783 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] barbican.timeout = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.058262 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] barbican.verify_ssl = True {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.058604 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] barbican.verify_ssl_path = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.059354 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] barbican_service_user.auth_section = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.059724 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] barbican_service_user.auth_type = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.060060 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] barbican_service_user.cafile = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.060395 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] barbican_service_user.certfile = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.060986 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] barbican_service_user.collect_timing = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.061337 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] barbican_service_user.insecure = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.061672 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] barbican_service_user.keyfile = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.062013 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] barbican_service_user.split_loggers = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.062348 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] barbican_service_user.timeout = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.062721 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] vault.approle_role_id = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.063058 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] vault.approle_secret_id = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.063536 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] vault.cafile = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.063875 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] vault.certfile = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.064217 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] vault.collect_timing = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.064580 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] vault.insecure = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.064915 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] vault.keyfile = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.065270 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] vault.kv_mountpoint = secret {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.065615 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] vault.kv_version = 2 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.065955 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] vault.namespace = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.066285 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] vault.root_token_id = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.066630 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] vault.split_loggers = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.066965 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] vault.ssl_ca_crt_file = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.067298 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] vault.timeout = None {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.067639 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] vault.use_ssl = False {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.067993 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] vault.vault_url = http://127.0.0.1:8200 {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 02 20:42:36.068756 ncdv-master-1 cinder-backup[129253]: DEBUG oslo_service.service [None req-c6a557ad-8273-4e63-a0f7-dfdbaa4bd809 None None] ******************************************************************************** {{(pid=129253) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Apr 02 20:42:36.089468 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_db.sqlalchemy.engines [None req-d7f3f213-319b-482b-a14a-8652a5ed5609 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129697) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 02 20:42:36.129237 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.lockutils [None req-d7f3f213-319b-482b-a14a-8652a5ed5609 None None] Lock "cleanup_incomplete_backups_129253" "released" by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: held 0.250s {{(pid=129697) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:400}} Apr 02 20:42:36.137787 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_db.sqlalchemy.engines [None req-d7f3f213-319b-482b-a14a-8652a5ed5609 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129697) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 02 20:42:36.143816 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-d7f3f213-319b-482b-a14a-8652a5ed5609 None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 20:42:36.172299 ncdv-master-1 cinder-backup[129697]: INFO cinder.keymgr.migration [None req-14e7b4e0-6afa-4da9-a6a2-dca2e056df78 None None] Not migrating encryption keys because the ConfKeyManager is still in use. Apr 02 20:42:36.201626 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.service [None req-d7f3f213-319b-482b-a14a-8652a5ed5609 None None] Creating RPC server for service cinder-backup {{(pid=129697) start /opt/stack/cinder/cinder/service.py:231}} Apr 02 20:42:36.218013 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.service [None req-d7f3f213-319b-482b-a14a-8652a5ed5609 None None] Pinning object versions for RPC server serializer to 1.39 {{(pid=129697) start /opt/stack/cinder/cinder/service.py:237}} Apr 02 20:43:36.248317 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 20:43:36.249116 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 20:44:36.244043 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 20:44:36.244650 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 20:46:36.250005 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 20:46:36.250531 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 20:47:36.246002 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 20:47:36.246698 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 20:48:36.242036 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 20:48:36.242762 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 20:49:36.247146 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 20:49:36.247873 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 20:50:36.252198 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 20:50:36.252931 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 20:51:36.257399 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 20:51:36.257983 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 20:52:36.256082 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 20:52:36.256642 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 20:53:36.258532 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 20:53:36.259197 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 20:54:36.254191 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 20:54:36.254750 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 20:55:36.251958 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 20:55:36.253132 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 20:56:36.251259 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 20:56:36.251902 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 20:57:36.247393 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 20:57:36.248274 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 20:58:36.253078 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 20:58:36.255450 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 20:59:36.254288 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 20:59:36.254866 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:00:36.256057 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:00:36.256733 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:01:36.260718 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:01:36.261387 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:02:36.266135 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:02:36.266773 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:03:36.267680 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:03:36.268870 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:04:36.264995 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:04:36.266213 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:05:36.266372 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:05:36.266924 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:06:36.266778 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:06:36.268829 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:07:36.271077 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:07:36.272230 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:08:36.274401 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:08:36.275186 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:09:36.279804 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:09:36.280528 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:10:36.281315 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:10:36.281945 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:11:36.287303 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:11:36.287948 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:12:36.288272 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:12:36.289139 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:12:36.837104 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] Create backup started, backup: eebc0cd6-3a4c-4d2b-881f-2a7fc5841351 volume: 598ae540-738c-4ad1-aa4f-c1e5f13e1481. Apr 02 21:12:36.851469 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-04-02T21:12:36Z,data_timestamp=2022-04-02T21:12:36Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsAdminTest-Backup-713776170',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=eebc0cd6-3a4c-4d2b-881f-2a7fc5841351,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='733befed769a4ec29b3f10b0efd4504f',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='9c65eea593394a6483c4b998a2ef1340',volume_id=598ae540-738c-4ad1-aa4f-c1e5f13e1481) Apr 02 21:12:37.016502 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.utils [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.18', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:12:37.040806 ncdv-master-1 cinder-backup[129697]: INFO oslo.privsep.daemon [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] Running privsep helper: ['sudo', 'cinder-rootwrap', '/etc/cinder/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/cinder/cinder.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmppf0iyau0/privsep.sock'] Apr 02 21:12:37.190075 ncdv-master-1 sudo[157520]: ubuntu : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf privsep-helper --config-file /etc/cinder/cinder.conf --privsep_context os_brick.privileged.default --privsep_sock_path /tmp/tmppf0iyau0/privsep.sock Apr 02 21:12:37.193133 ncdv-master-1 sudo[157520]: pam_unix(sudo:session): session opened for user root by (uid=0) Apr 02 21:12:42.969682 ncdv-master-1 sudo[157520]: pam_unix(sudo:session): session closed for user root Apr 02 21:12:42.975470 ncdv-master-1 cinder-backup[129697]: INFO oslo.privsep.daemon [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] Spawned new privsep daemon via rootwrap Apr 02 21:12:42.976313 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] Accepted privsep connection to /tmp/tmppf0iyau0/privsep.sock {{(pid=129697) __init__ /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:362}} Apr 02 21:12:42.978128 ncdv-master-1 cinder-backup[129697]: INFO oslo.privsep.daemon [-] privsep daemon starting Apr 02 21:12:42.978820 ncdv-master-1 cinder-backup[129697]: INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 Apr 02 21:12:42.979330 ncdv-master-1 cinder-backup[129697]: INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none Apr 02 21:12:42.979803 ncdv-master-1 cinder-backup[129697]: INFO oslo.privsep.daemon [-] privsep daemon running as pid 157607 Apr 02 21:12:42.987810 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[31522a6d-5028-40eb-9ed2-71ad8d9d895a]: (2,) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:12:43.445637 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:12:43.481434 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.043s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:12:43.481776 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[ee63c611-51a3-4a5a-8acb-a0dde406c43a]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:2a5dbfa3acf\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:12:43.485401 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:12:43.492966 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 02 21:12:43.515870 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[cee1c7a0-9583-4006-95fa-072e556994c3]: [Errno 2] No such file or directory: 'systool' {{(pid=157607) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 02 21:12:43.515870 ncdv-master-1 cinder-backup[129697]: Traceback (most recent call last): Apr 02 21:12:43.515870 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 02 21:12:43.515870 ncdv-master-1 cinder-backup[129697]: ret = func(*f_args, **f_kwargs) Apr 02 21:12:43.515870 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 02 21:12:43.515870 ncdv-master-1 cinder-backup[129697]: return func(*args, **kwargs) Apr 02 21:12:43.515870 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 02 21:12:43.515870 ncdv-master-1 cinder-backup[129697]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 02 21:12:43.515870 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 02 21:12:43.515870 ncdv-master-1 cinder-backup[129697]: return putils.execute(on_execute=on_execute, Apr 02 21:12:43.515870 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 02 21:12:43.515870 ncdv-master-1 cinder-backup[129697]: obj = subprocess.Popen(cmd, Apr 02 21:12:43.515870 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 02 21:12:43.515870 ncdv-master-1 cinder-backup[129697]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 02 21:12:43.515870 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 02 21:12:43.515870 ncdv-master-1 cinder-backup[129697]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 02 21:12:43.515870 ncdv-master-1 cinder-backup[129697]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 02 21:12:43.517295 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[cee1c7a0-9583-4006-95fa-072e556994c3]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:12:43.520994 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:12:43.535495 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 02 21:12:43.538416 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[9b5eb2ff-36fd-40a5-ae3f-4f3cd6485d99]: [Errno 2] No such file or directory: 'systool' {{(pid=157607) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 02 21:12:43.538416 ncdv-master-1 cinder-backup[129697]: Traceback (most recent call last): Apr 02 21:12:43.538416 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 02 21:12:43.538416 ncdv-master-1 cinder-backup[129697]: ret = func(*f_args, **f_kwargs) Apr 02 21:12:43.538416 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 02 21:12:43.538416 ncdv-master-1 cinder-backup[129697]: return func(*args, **kwargs) Apr 02 21:12:43.538416 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 02 21:12:43.538416 ncdv-master-1 cinder-backup[129697]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 02 21:12:43.538416 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 02 21:12:43.538416 ncdv-master-1 cinder-backup[129697]: return putils.execute(on_execute=on_execute, Apr 02 21:12:43.538416 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 02 21:12:43.538416 ncdv-master-1 cinder-backup[129697]: obj = subprocess.Popen(cmd, Apr 02 21:12:43.538416 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 02 21:12:43.538416 ncdv-master-1 cinder-backup[129697]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 02 21:12:43.538416 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 02 21:12:43.538416 ncdv-master-1 cinder-backup[129697]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 02 21:12:43.538416 ncdv-master-1 cinder-backup[129697]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 02 21:12:43.539603 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[9b5eb2ff-36fd-40a5-ae3f-4f3cd6485d99]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:12:43.542629 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:12:43.599069 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.055s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:12:43.599924 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[4e4a7bbf-0689-4b69-8b99-d056329b7a64]: (4, ('/dev/sda1\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:12:43.601489 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:12:43.659361 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.057s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:12:43.659691 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[75c0d566-7162-4735-a2b7-94d68222ddf7]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:12:43.661524 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:12:43.689488 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.028s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:12:43.689958 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[73bbc7cc-4cba-4b65-a21f-c566b350fb10]: (4, ('c36434ea-c598-ca44-bb67-56d5c2bbf98e\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:12:43.691345 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] Running cmd (subprocess): nvme version {{(pid=129697) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:12:43.743510 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] CMD "nvme version" returned: 0 in 0.052s {{(pid=129697) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:12:43.777010 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129697) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Apr 02 21:12:43.777683 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129697) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Apr 02 21:12:43.778160 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:25346655-e237-4641-aa32-2fe34121db47 dsc: {{(pid=129697) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Apr 02 21:12:43.780193 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.utils [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] <== get_connector_properties: return (6761ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.18', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:2a5dbfa3acf', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c36434ea-c598-ca44-bb67-56d5c2bbf98e', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:25346655-e237-4641-aa32-2fe34121db47', 'found_dsc': ''} {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:12:44.603006 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connector [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] Factory for rbd on None {{(pid=129697) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Apr 02 21:12:44.605882 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-598ae540-738c-4ad1-aa4f-c1e5f13e1481', 'hosts': ['10.222.0.18'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '598ae540-738c-4ad1-aa4f-c1e5f13e1481', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:12:44.607028 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.linuxrbd [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] opening connection to ceph cluster (timeout=-1). {{(pid=129697) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Apr 02 21:12:44.686187 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] <== connect_volume: return (79ms) {'path': } {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:12:44.696435 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_db.sqlalchemy.engines [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129697) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 02 21:12:44.720492 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] Starting backup of volume='598ae540-738c-4ad1-aa4f-c1e5f13e1481'. {{(pid=129697) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Apr 02 21:12:44.722334 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] Volume file is RBD: attempting optimized backup {{(pid=129697) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Apr 02 21:12:44.779242 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] Creating base image 'volume-598ae540-738c-4ad1-aa4f-c1e5f13e1481.backup.eebc0cd6-3a4c-4d2b-881f-2a7fc5841351' {{(pid=129697) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Apr 02 21:12:44.779759 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129697) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Apr 02 21:12:46.683528 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] Using --from-snap 'None' for incremental backup of volume 598ae540-738c-4ad1-aa4f-c1e5f13e1481. {{(pid=129697) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Apr 02 21:12:46.684721 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] Creating backup snapshot='backup.eebc0cd6-3a4c-4d2b-881f-2a7fc5841351.snap.1648933966.6842763' {{(pid=129697) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Apr 02 21:12:47.644781 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] Performing differential transfer from 'volume-598ae540-738c-4ad1-aa4f-c1e5f13e1481' to 'volume-598ae540-738c-4ad1-aa4f-c1e5f13e1481.backup.eebc0cd6-3a4c-4d2b-881f-2a7fc5841351' {{(pid=129697) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Apr 02 21:12:47.645424 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_p3snb0ea --pool volumes volumes/volume-598ae540-738c-4ad1-aa4f-c1e5f13e1481@backup.eebc0cd6-3a4c-4d2b-881f-2a7fc5841351.snap.1648933966.6842763 -' into... {{(pid=129697) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Apr 02 21:12:47.645859 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-598ae540-738c-4ad1-aa4f-c1e5f13e1481.backup.eebc0cd6-3a4c-4d2b-881f-2a7fc5841351' {{(pid=129697) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Apr 02 21:12:48.673568 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] Differential backup transfer completed in 1.0277s {{(pid=129697) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Apr 02 21:12:48.674188 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] Getting metadata type 'volume-base-metadata' {{(pid=129697) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Apr 02 21:12:48.688085 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129697) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Apr 02 21:12:48.688518 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] Getting metadata type 'volume-metadata' {{(pid=129697) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Apr 02 21:12:48.699868 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] No metadata type 'volume-metadata' available {{(pid=129697) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Apr 02 21:12:48.700250 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] Getting metadata type 'volume-glance-metadata' {{(pid=129697) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Apr 02 21:12:48.709410 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] No metadata type 'volume-glance-metadata' available {{(pid=129697) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Apr 02 21:12:48.709970 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] Backing up metadata for volume 598ae540-738c-4ad1-aa4f-c1e5f13e1481. {{(pid=129697) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Apr 02 21:12:48.755456 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] Backup 'eebc0cd6-3a4c-4d2b-881f-2a7fc5841351' of volume 598ae540-738c-4ad1-aa4f-c1e5f13e1481 finished. {{(pid=129697) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Apr 02 21:12:48.757129 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-598ae540-738c-4ad1-aa4f-c1e5f13e1481', 'hosts': ['10.222.0.18'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '598ae540-738c-4ad1-aa4f-c1e5f13e1481', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:12:48.777335 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] <== disconnect_volume: return (19ms) None {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:12:48.862985 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-fcda4aee-adb9-4f6b-9ef2-92231ffbbaa3 tempest-VolumesBackupsAdminTest-119965354 None] Create backup finished. backup: eebc0cd6-3a4c-4d2b-881f-2a7fc5841351. Apr 02 21:12:49.563672 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-ed8588d7-21c8-4076-9f2f-a4742dd72243 tempest-VolumesBackupsAdminTest-599724220 None] Export record started, backup: eebc0cd6-3a4c-4d2b-881f-2a7fc5841351. Apr 02 21:12:49.565868 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-ed8588d7-21c8-4076-9f2f-a4742dd72243 tempest-VolumesBackupsAdminTest-599724220 None] Export record finished, backup eebc0cd6-3a4c-4d2b-881f-2a7fc5841351 exported. Apr 02 21:12:49.757129 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-f36c4f6a-2843-477c-88e1-a96009a2238c tempest-VolumesBackupsAdminTest-599724220 None] Import record started, backup_url: eyJkcml2ZXJfaW5mbyI6IHt9LCAiaWQiOiAiNDE0YmYwOTQtZmZkYy00Yjk5LTk4YWYtZTQwMzM1MjkyOTIwIiwgInVzZXJfaWQiOiAiOWM2NWVlYTU5MzM5NGE2NDgzYzRiOTk4YTJlZjEzNDAiLCAicHJvamVjdF9pZCI6ICI3MzNiZWZlZDc2OWE0ZWMyOWIzZjEwYjBlZmQ0NTA0ZiIsICJ2b2x1bWVfaWQiOiAiNTk4YWU1NDAtNzM4Yy00YWQxLWFhNGYtYzFlNWYxM2UxNDgxIiwgImhvc3QiOiAibmNkdi1tYXN0ZXItMSIsICJhdmFpbGFiaWxpdHlfem9uZSI6IG51bGwsICJjb250YWluZXIiOiAiYmFja3VwcyIsICJwYXJlbnRfaWQiOiBudWxsLCAic3RhdHVzIjogImF2YWlsYWJsZSIsICJmYWlsX3JlYXNvbiI6IG51bGwsICJzaXplIjogMSwgImRpc3BsYXlfbmFtZSI6ICJ0ZW1wZXN0LVZvbHVtZXNCYWNrdXBzQWRtaW5UZXN0LUJhY2t1cC03MTM3NzYxNzAiLCAiZGlzcGxheV9kZXNjcmlwdGlvbiI6IG51bGwsICJzZXJ2aWNlX21ldGFkYXRhIjogIntcImJhc2VcIjogXCJ2b2x1bWUtNTk4YWU1NDAtNzM4Yy00YWQxLWFhNGYtYzFlNWYxM2UxNDgxLmJhY2t1cC5lZWJjMGNkNi0zYTRjLTRkMmItODgxZi0yYTdmYzU4NDEzNTFcIn0iLCAic2VydmljZSI6ICJjaW5kZXIuYmFja3VwLmRyaXZlcnMuY2VwaC5DZXBoQmFja3VwRHJpdmVyIiwgIm9iamVjdF9jb3VudCI6IDAsICJ0ZW1wX3ZvbHVtZV9pZCI6IG51bGwsICJ0ZW1wX3NuYXBzaG90X2lkIjogbnVsbCwgIm51bV9kZXBlbmRlbnRfYmFja3VwcyI6IDAsICJzbmFwc2hvdF9pZCI6IG51bGwsICJkYXRhX3RpbWVzdGFtcCI6ICIyMDIyLTA0LTAyVDIxOjEyOjM3WiIsICJyZXN0b3JlX3ZvbHVtZV9pZCI6IG51bGwsICJtZXRhZGF0YSI6IHt9LCAiZW5jcnlwdGlvbl9rZXlfaWQiOiBudWxsLCAiY3JlYXRlZF9hdCI6ICIyMDIyLTA0LTAyVDIxOjEyOjM3WiIsICJ1cGRhdGVkX2F0IjogIjIwMjItMDQtMDJUMjE6MTI6NDlaIiwgImRlbGV0ZWRfYXQiOiBudWxsLCAiZGVsZXRlZCI6IGZhbHNlfQ==. Apr 02 21:12:49.765923 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_db.sqlalchemy.engines [None req-f36c4f6a-2843-477c-88e1-a96009a2238c tempest-VolumesBackupsAdminTest-599724220 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129697) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 02 21:12:49.852561 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-f36c4f6a-2843-477c-88e1-a96009a2238c tempest-VolumesBackupsAdminTest-599724220 None] Import record id 414bf094-ffdc-4b99-98af-e40335292920 metadata from driver finished. Apr 02 21:12:52.463969 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] Restore backup started, backup: eebc0cd6-3a4c-4d2b-881f-2a7fc5841351 volume: 71da2da2-6544-44ed-b674-74a5996b026c. Apr 02 21:12:52.498554 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.utils [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.18', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:12:52.503333 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:12:52.522701 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.022s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:12:52.522996 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[0d4c5472-f6d9-447b-bbbc-00e8ee0f1d2b]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:2a5dbfa3acf\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:12:52.525275 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:12:52.555467 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 02 21:12:52.556069 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[dd8516b7-66d4-499e-843b-d76d024bcc42]: [Errno 2] No such file or directory: 'systool' {{(pid=157607) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 02 21:12:52.556069 ncdv-master-1 cinder-backup[129697]: Traceback (most recent call last): Apr 02 21:12:52.556069 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 02 21:12:52.556069 ncdv-master-1 cinder-backup[129697]: ret = func(*f_args, **f_kwargs) Apr 02 21:12:52.556069 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 02 21:12:52.556069 ncdv-master-1 cinder-backup[129697]: return func(*args, **kwargs) Apr 02 21:12:52.556069 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 02 21:12:52.556069 ncdv-master-1 cinder-backup[129697]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 02 21:12:52.556069 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 02 21:12:52.556069 ncdv-master-1 cinder-backup[129697]: return putils.execute(on_execute=on_execute, Apr 02 21:12:52.556069 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 02 21:12:52.556069 ncdv-master-1 cinder-backup[129697]: obj = subprocess.Popen(cmd, Apr 02 21:12:52.556069 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 02 21:12:52.556069 ncdv-master-1 cinder-backup[129697]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 02 21:12:52.556069 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 02 21:12:52.556069 ncdv-master-1 cinder-backup[129697]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 02 21:12:52.556069 ncdv-master-1 cinder-backup[129697]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 02 21:12:52.559414 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[dd8516b7-66d4-499e-843b-d76d024bcc42]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:12:52.560949 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:12:52.581609 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 02 21:12:52.582189 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[2643c191-8a90-4123-911f-1ea56f5d2802]: [Errno 2] No such file or directory: 'systool' {{(pid=157607) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 02 21:12:52.582189 ncdv-master-1 cinder-backup[129697]: Traceback (most recent call last): Apr 02 21:12:52.582189 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 02 21:12:52.582189 ncdv-master-1 cinder-backup[129697]: ret = func(*f_args, **f_kwargs) Apr 02 21:12:52.582189 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 02 21:12:52.582189 ncdv-master-1 cinder-backup[129697]: return func(*args, **kwargs) Apr 02 21:12:52.582189 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 02 21:12:52.582189 ncdv-master-1 cinder-backup[129697]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 02 21:12:52.582189 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 02 21:12:52.582189 ncdv-master-1 cinder-backup[129697]: return putils.execute(on_execute=on_execute, Apr 02 21:12:52.582189 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 02 21:12:52.582189 ncdv-master-1 cinder-backup[129697]: obj = subprocess.Popen(cmd, Apr 02 21:12:52.582189 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 02 21:12:52.582189 ncdv-master-1 cinder-backup[129697]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 02 21:12:52.582189 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 02 21:12:52.582189 ncdv-master-1 cinder-backup[129697]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 02 21:12:52.582189 ncdv-master-1 cinder-backup[129697]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 02 21:12:52.584151 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[2643c191-8a90-4123-911f-1ea56f5d2802]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:12:52.585932 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:12:52.615249 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.028s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:12:52.615670 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[a0f3cb3d-97d0-4d52-9731-d4b7b4517485]: (4, ('/dev/sda1\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:12:52.619002 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:12:52.645210 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.028s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:12:52.646178 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[543618a2-28b8-4e53-8b64-dd9ab9e5d9ac]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:12:52.647688 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:12:52.703373 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.055s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:12:52.704443 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[b381d31c-36e2-475d-9551-27ee6fa207f3]: (4, ('c36434ea-c598-ca44-bb67-56d5c2bbf98e\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:12:52.705524 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] Running cmd (subprocess): nvme version {{(pid=129697) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:12:52.747102 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] CMD "nvme version" returned: 0 in 0.041s {{(pid=129697) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:12:52.752541 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129697) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Apr 02 21:12:52.752924 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129697) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Apr 02 21:12:52.757754 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:25346655-e237-4641-aa32-2fe34121db47 dsc: {{(pid=129697) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Apr 02 21:12:52.758492 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.utils [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] <== get_connector_properties: return (259ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.18', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:2a5dbfa3acf', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c36434ea-c598-ca44-bb67-56d5c2bbf98e', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:25346655-e237-4641-aa32-2fe34121db47', 'found_dsc': ''} {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:12:53.955569 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connector [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] Factory for rbd on None {{(pid=129697) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Apr 02 21:12:53.957913 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-71da2da2-6544-44ed-b674-74a5996b026c', 'hosts': ['10.222.0.18'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '71da2da2-6544-44ed-b674-74a5996b026c', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:12:53.960303 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.linuxrbd [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] opening connection to ceph cluster (timeout=-1). {{(pid=129697) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Apr 02 21:12:54.055626 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] <== connect_volume: return (96ms) {'path': } {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:12:54.066173 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_db.sqlalchemy.engines [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129697) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 02 21:12:54.086776 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] Starting restore from Ceph backup=eebc0cd6-3a4c-4d2b-881f-2a7fc5841351 to volume=volume-71da2da2-6544-44ed-b674-74a5996b026c {{(pid=129697) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1274}} Apr 02 21:12:54.274706 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] Looking for snapshot of backup base 'volume-598ae540-738c-4ad1-aa4f-c1e5f13e1481.backup.eebc0cd6-3a4c-4d2b-881f-2a7fc5841351' {{(pid=129697) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Apr 02 21:12:54.274706 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] Found snapshot 'backup.eebc0cd6-3a4c-4d2b-881f-2a7fc5841351.snap.1648933966.6842763' {{(pid=129697) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Apr 02 21:12:54.293363 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] Volume file is RBD. {{(pid=129697) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1184}} Apr 02 21:12:54.361144 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] Attempting differential restore. {{(pid=129697) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1234}} Apr 02 21:12:54.361661 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] Attempting incremental restore from base='volume-598ae540-738c-4ad1-aa4f-c1e5f13e1481.backup.eebc0cd6-3a4c-4d2b-881f-2a7fc5841351' snap='backup.eebc0cd6-3a4c-4d2b-881f-2a7fc5841351.snap.1648933966.6842763' {{(pid=129697) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1089}} Apr 02 21:12:54.362062 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] Performing differential transfer from 'volume-598ae540-738c-4ad1-aa4f-c1e5f13e1481.backup.eebc0cd6-3a4c-4d2b-881f-2a7fc5841351' to 'volume-71da2da2-6544-44ed-b674-74a5996b026c' {{(pid=129697) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Apr 02 21:12:54.362465 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-598ae540-738c-4ad1-aa4f-c1e5f13e1481.backup.eebc0cd6-3a4c-4d2b-881f-2a7fc5841351@backup.eebc0cd6-3a4c-4d2b-881f-2a7fc5841351.snap.1648933966.6842763 -' into... {{(pid=129697) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Apr 02 21:12:54.362832 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_7cyifvxy --pool volumes - volumes/volume-71da2da2-6544-44ed-b674-74a5996b026c' {{(pid=129697) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Apr 02 21:12:55.797683 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] Restore transfer completed in 1.4350s {{(pid=129697) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1111}} Apr 02 21:12:55.798688 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=129697) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1285}} Apr 02 21:12:55.854941 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] Restoring volume base metadata {{(pid=129697) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} Apr 02 21:12:55.879106 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] No metadata of type 'volume-metadata' to restore {{(pid=129697) put /opt/stack/cinder/cinder/backup/driver.py:345}} Apr 02 21:12:55.879802 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] No metadata of type 'volume-glance-metadata' to restore {{(pid=129697) put /opt/stack/cinder/cinder/backup/driver.py:345}} Apr 02 21:12:55.884547 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] Restore to volume 71da2da2-6544-44ed-b674-74a5996b026c finished successfully. {{(pid=129697) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1292}} Apr 02 21:12:55.886756 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-71da2da2-6544-44ed-b674-74a5996b026c', 'hosts': ['10.222.0.18'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '71da2da2-6544-44ed-b674-74a5996b026c', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:12:55.908367 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] <== disconnect_volume: return (20ms) None {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:12:55.969343 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.manager [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] Encryption key ID for volume 71da2da2-6544-44ed-b674-74a5996b026c already matches encryption key ID in backup eebc0cd6-3a4c-4d2b-881f-2a7fc5841351. {{(pid=129697) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} Apr 02 21:12:56.021179 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-ac4c3124-1fbb-4f41-9938-8e0ebcc65132 tempest-VolumesBackupsAdminTest-119965354 None] Finished restoring backup eebc0cd6-3a4c-4d2b-881f-2a7fc5841351 to volume 71da2da2-6544-44ed-b674-74a5996b026c. Apr 02 21:12:57.046199 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-d4dbdd88-6d85-4165-932d-a4a7f66adfb2 tempest-VolumesBackupsAdminTest-599724220 None] Delete backup started, backup: 414bf094-ffdc-4b99-98af-e40335292920. Apr 02 21:12:57.047970 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-d4dbdd88-6d85-4165-932d-a4a7f66adfb2 tempest-VolumesBackupsAdminTest-599724220 None] Delete started for backup=414bf094-ffdc-4b99-98af-e40335292920 {{(pid=129697) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Apr 02 21:12:57.048390 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-d4dbdd88-6d85-4165-932d-a4a7f66adfb2 tempest-VolumesBackupsAdminTest-599724220 None] Trying diff format basename='volume-598ae540-738c-4ad1-aa4f-c1e5f13e1481.backup.eebc0cd6-3a4c-4d2b-881f-2a7fc5841351' for backup base image of volume 598ae540-738c-4ad1-aa4f-c1e5f13e1481. {{(pid=129697) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Apr 02 21:12:57.125012 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-d4dbdd88-6d85-4165-932d-a4a7f66adfb2 tempest-VolumesBackupsAdminTest-599724220 None] Looking for snapshot of backup base 'volume-598ae540-738c-4ad1-aa4f-c1e5f13e1481.backup.eebc0cd6-3a4c-4d2b-881f-2a7fc5841351' {{(pid=129697) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Apr 02 21:12:57.125946 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-d4dbdd88-6d85-4165-932d-a4a7f66adfb2 tempest-VolumesBackupsAdminTest-599724220 None] Backup '414bf094-ffdc-4b99-98af-e40335292920' has no snapshot {{(pid=129697) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:915}} Apr 02 21:12:57.126296 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-d4dbdd88-6d85-4165-932d-a4a7f66adfb2 tempest-VolumesBackupsAdminTest-599724220 None] No backup snapshot to delete {{(pid=129697) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:476}} Apr 02 21:12:57.135854 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.drivers.ceph [None req-d4dbdd88-6d85-4165-932d-a4a7f66adfb2 tempest-VolumesBackupsAdminTest-599724220 None] Backup base image of volume 598ae540-738c-4ad1-aa4f-c1e5f13e1481 still has 1 snapshots so skipping base image delete. Apr 02 21:12:57.205002 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-d4dbdd88-6d85-4165-932d-a4a7f66adfb2 tempest-VolumesBackupsAdminTest-599724220 None] Metadata backup object 'backup.414bf094-ffdc-4b99-98af-e40335292920.meta' not found - ignoring {{(pid=129697) remove_if_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:160}} Apr 02 21:12:57.209948 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-d4dbdd88-6d85-4165-932d-a4a7f66adfb2 tempest-VolumesBackupsAdminTest-599724220 None] Delete of backup '414bf094-ffdc-4b99-98af-e40335292920' for volume '598ae540-738c-4ad1-aa4f-c1e5f13e1481' finished. {{(pid=129697) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Apr 02 21:12:57.217659 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_db.sqlalchemy.engines [None req-d4dbdd88-6d85-4165-932d-a4a7f66adfb2 tempest-VolumesBackupsAdminTest-599724220 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129697) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 02 21:12:57.245597 ncdv-master-1 cinder-backup[129697]: WARNING cinder.quota [None req-d4dbdd88-6d85-4165-932d-a4a7f66adfb2 tempest-VolumesBackupsAdminTest-599724220 None] Deprecated: Default quota for resource: backups is set by the default quota flag: quota_backups, it is now deprecated. Please use the default quota class for default quota. Apr 02 21:12:57.247343 ncdv-master-1 cinder-backup[129697]: WARNING cinder.quota [None req-d4dbdd88-6d85-4165-932d-a4a7f66adfb2 tempest-VolumesBackupsAdminTest-599724220 None] Deprecated: Default quota for resource: backup_gigabytes is set by the default quota flag: quota_backup_gigabytes, it is now deprecated. Please use the default quota class for default quota. Apr 02 21:12:57.272600 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.quota [None req-d4dbdd88-6d85-4165-932d-a4a7f66adfb2 tempest-VolumesBackupsAdminTest-599724220 None] Created reservations ['6be38b69-d908-4fb1-b053-e704b5f577b8', '71c41407-95d7-41ec-b926-b5ef196adb1e'] {{(pid=129697) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 02 21:12:57.325742 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-d4dbdd88-6d85-4165-932d-a4a7f66adfb2 tempest-VolumesBackupsAdminTest-599724220 None] Delete backup finished, backup 414bf094-ffdc-4b99-98af-e40335292920 deleted. Apr 02 21:12:58.168278 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-8d3218ad-a52f-487e-9afd-eba1a5511d2f tempest-VolumesBackupsAdminTest-119965354 None] Delete backup started, backup: eebc0cd6-3a4c-4d2b-881f-2a7fc5841351. Apr 02 21:12:58.170773 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-8d3218ad-a52f-487e-9afd-eba1a5511d2f tempest-VolumesBackupsAdminTest-119965354 None] Delete started for backup=eebc0cd6-3a4c-4d2b-881f-2a7fc5841351 {{(pid=129697) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Apr 02 21:12:58.170893 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-8d3218ad-a52f-487e-9afd-eba1a5511d2f tempest-VolumesBackupsAdminTest-119965354 None] Trying diff format basename='volume-598ae540-738c-4ad1-aa4f-c1e5f13e1481.backup.eebc0cd6-3a4c-4d2b-881f-2a7fc5841351' for backup base image of volume 598ae540-738c-4ad1-aa4f-c1e5f13e1481. {{(pid=129697) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Apr 02 21:12:58.264280 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-8d3218ad-a52f-487e-9afd-eba1a5511d2f tempest-VolumesBackupsAdminTest-119965354 None] Looking for snapshot of backup base 'volume-598ae540-738c-4ad1-aa4f-c1e5f13e1481.backup.eebc0cd6-3a4c-4d2b-881f-2a7fc5841351' {{(pid=129697) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Apr 02 21:12:58.264748 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-8d3218ad-a52f-487e-9afd-eba1a5511d2f tempest-VolumesBackupsAdminTest-119965354 None] Found snapshot 'backup.eebc0cd6-3a4c-4d2b-881f-2a7fc5841351.snap.1648933966.6842763' {{(pid=129697) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Apr 02 21:12:58.265341 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-8d3218ad-a52f-487e-9afd-eba1a5511d2f tempest-VolumesBackupsAdminTest-119965354 None] Deleting backup snapshot='backup.eebc0cd6-3a4c-4d2b-881f-2a7fc5841351.snap.1648933966.6842763' {{(pid=129697) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Apr 02 21:12:58.696669 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.drivers.ceph [None req-8d3218ad-a52f-487e-9afd-eba1a5511d2f tempest-VolumesBackupsAdminTest-119965354 None] Deleting backup base image='volume-598ae540-738c-4ad1-aa4f-c1e5f13e1481.backup.eebc0cd6-3a4c-4d2b-881f-2a7fc5841351' of volume 598ae540-738c-4ad1-aa4f-c1e5f13e1481. Apr 02 21:12:58.986649 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-8d3218ad-a52f-487e-9afd-eba1a5511d2f tempest-VolumesBackupsAdminTest-119965354 None] Base backup image='volume-598ae540-738c-4ad1-aa4f-c1e5f13e1481.backup.eebc0cd6-3a4c-4d2b-881f-2a7fc5841351' of volume 598ae540-738c-4ad1-aa4f-c1e5f13e1481 deleted. {{(pid=129697) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Apr 02 21:12:59.054492 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-8d3218ad-a52f-487e-9afd-eba1a5511d2f tempest-VolumesBackupsAdminTest-119965354 None] Delete of backup 'eebc0cd6-3a4c-4d2b-881f-2a7fc5841351' for volume '598ae540-738c-4ad1-aa4f-c1e5f13e1481' finished. {{(pid=129697) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Apr 02 21:12:59.062220 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_db.sqlalchemy.engines [None req-8d3218ad-a52f-487e-9afd-eba1a5511d2f tempest-VolumesBackupsAdminTest-119965354 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129697) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 02 21:12:59.093012 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.quota [None req-8d3218ad-a52f-487e-9afd-eba1a5511d2f tempest-VolumesBackupsAdminTest-119965354 None] Created reservations ['ef4ea0a2-b74a-455f-9ea5-13ae557a58a4', 'e9bdf14c-d9e7-4f3a-a051-3b8d0be861de'] {{(pid=129697) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 02 21:12:59.137696 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-8d3218ad-a52f-487e-9afd-eba1a5511d2f tempest-VolumesBackupsAdminTest-119965354 None] Delete backup finished, backup eebc0cd6-3a4c-4d2b-881f-2a7fc5841351 deleted. Apr 02 21:13:01.040322 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] Create backup started, backup: b95dff28-554e-45c2-a740-5f66c9c2f3d4 volume: 4be85628-1d3b-468d-96cb-ed8984971395. Apr 02 21:13:01.047789 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-04-02T21:13:00Z,data_timestamp=2022-04-02T21:13:00Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsAdminTest-Backup-1178197629',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=b95dff28-554e-45c2-a740-5f66c9c2f3d4,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='733befed769a4ec29b3f10b0efd4504f',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='9c65eea593394a6483c4b998a2ef1340',volume_id=4be85628-1d3b-468d-96cb-ed8984971395) Apr 02 21:13:01.089533 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.utils [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.18', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:13:01.091577 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:13:01.098948 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.007s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:13:01.099751 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[95061b27-d149-4817-8d40-0e08ca5a4658]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:2a5dbfa3acf\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:13:01.106192 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:13:01.112383 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 02 21:13:01.112711 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[f78f7d43-0695-43c3-b7d6-482065ab9616]: [Errno 2] No such file or directory: 'systool' {{(pid=157607) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 02 21:13:01.112711 ncdv-master-1 cinder-backup[129697]: Traceback (most recent call last): Apr 02 21:13:01.112711 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 02 21:13:01.112711 ncdv-master-1 cinder-backup[129697]: ret = func(*f_args, **f_kwargs) Apr 02 21:13:01.112711 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 02 21:13:01.112711 ncdv-master-1 cinder-backup[129697]: return func(*args, **kwargs) Apr 02 21:13:01.112711 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 02 21:13:01.112711 ncdv-master-1 cinder-backup[129697]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 02 21:13:01.112711 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 02 21:13:01.112711 ncdv-master-1 cinder-backup[129697]: return putils.execute(on_execute=on_execute, Apr 02 21:13:01.112711 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 02 21:13:01.112711 ncdv-master-1 cinder-backup[129697]: obj = subprocess.Popen(cmd, Apr 02 21:13:01.112711 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 02 21:13:01.112711 ncdv-master-1 cinder-backup[129697]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 02 21:13:01.112711 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 02 21:13:01.112711 ncdv-master-1 cinder-backup[129697]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 02 21:13:01.112711 ncdv-master-1 cinder-backup[129697]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 02 21:13:01.113987 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[f78f7d43-0695-43c3-b7d6-482065ab9616]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:13:01.115364 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:13:01.124159 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 02 21:13:01.125502 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[44aba2d8-5ec2-4279-bf1f-afa276f4a415]: [Errno 2] No such file or directory: 'systool' {{(pid=157607) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 02 21:13:01.125502 ncdv-master-1 cinder-backup[129697]: Traceback (most recent call last): Apr 02 21:13:01.125502 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 02 21:13:01.125502 ncdv-master-1 cinder-backup[129697]: ret = func(*f_args, **f_kwargs) Apr 02 21:13:01.125502 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 02 21:13:01.125502 ncdv-master-1 cinder-backup[129697]: return func(*args, **kwargs) Apr 02 21:13:01.125502 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 02 21:13:01.125502 ncdv-master-1 cinder-backup[129697]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 02 21:13:01.125502 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 02 21:13:01.125502 ncdv-master-1 cinder-backup[129697]: return putils.execute(on_execute=on_execute, Apr 02 21:13:01.125502 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 02 21:13:01.125502 ncdv-master-1 cinder-backup[129697]: obj = subprocess.Popen(cmd, Apr 02 21:13:01.125502 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 02 21:13:01.125502 ncdv-master-1 cinder-backup[129697]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 02 21:13:01.125502 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 02 21:13:01.125502 ncdv-master-1 cinder-backup[129697]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 02 21:13:01.125502 ncdv-master-1 cinder-backup[129697]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 02 21:13:01.126494 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[44aba2d8-5ec2-4279-bf1f-afa276f4a415]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:13:01.128637 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:13:01.145623 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.017s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:13:01.147838 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[9936f4ca-5b76-4f65-9f80-8ba66faf9f58]: (4, ('/dev/sda1\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:13:01.149353 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:13:01.168488 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.018s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:13:01.169354 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[d4899b4b-1390-44a0-ba89-58cd8993cc44]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:13:01.171028 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:13:01.179592 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.009s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:13:01.180130 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[4ba89996-8c72-4d79-b25d-45aa4cd53c96]: (4, ('c36434ea-c598-ca44-bb67-56d5c2bbf98e\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:13:01.180913 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] Running cmd (subprocess): nvme version {{(pid=129697) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:13:01.201636 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] CMD "nvme version" returned: 0 in 0.020s {{(pid=129697) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:13:01.204382 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129697) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Apr 02 21:13:01.204925 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129697) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Apr 02 21:13:01.205306 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:25346655-e237-4641-aa32-2fe34121db47 dsc: {{(pid=129697) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Apr 02 21:13:01.205952 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.utils [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] <== get_connector_properties: return (115ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.18', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:2a5dbfa3acf', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c36434ea-c598-ca44-bb67-56d5c2bbf98e', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:25346655-e237-4641-aa32-2fe34121db47', 'found_dsc': ''} {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:13:02.088168 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connector [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] Factory for rbd on None {{(pid=129697) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Apr 02 21:13:02.090309 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-4be85628-1d3b-468d-96cb-ed8984971395', 'hosts': ['10.222.0.18'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '4be85628-1d3b-468d-96cb-ed8984971395', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:13:02.091413 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.linuxrbd [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] opening connection to ceph cluster (timeout=-1). {{(pid=129697) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Apr 02 21:13:02.155964 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] <== connect_volume: return (64ms) {'path': } {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:13:02.163676 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_db.sqlalchemy.engines [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129697) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 02 21:13:02.186205 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] Starting backup of volume='4be85628-1d3b-468d-96cb-ed8984971395'. {{(pid=129697) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Apr 02 21:13:02.187074 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] Volume file is RBD: attempting optimized backup {{(pid=129697) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Apr 02 21:13:02.229302 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] Creating base image 'volume-4be85628-1d3b-468d-96cb-ed8984971395.backup.b95dff28-554e-45c2-a740-5f66c9c2f3d4' {{(pid=129697) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Apr 02 21:13:02.229881 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129697) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Apr 02 21:13:02.278108 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] Using --from-snap 'None' for incremental backup of volume 4be85628-1d3b-468d-96cb-ed8984971395. {{(pid=129697) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Apr 02 21:13:02.278998 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] Creating backup snapshot='backup.b95dff28-554e-45c2-a740-5f66c9c2f3d4.snap.1648933982.2781463' {{(pid=129697) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Apr 02 21:13:02.379604 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] Performing differential transfer from 'volume-4be85628-1d3b-468d-96cb-ed8984971395' to 'volume-4be85628-1d3b-468d-96cb-ed8984971395.backup.b95dff28-554e-45c2-a740-5f66c9c2f3d4' {{(pid=129697) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Apr 02 21:13:02.380363 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_zmjtakem --pool volumes volumes/volume-4be85628-1d3b-468d-96cb-ed8984971395@backup.b95dff28-554e-45c2-a740-5f66c9c2f3d4.snap.1648933982.2781463 -' into... {{(pid=129697) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Apr 02 21:13:02.380913 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-4be85628-1d3b-468d-96cb-ed8984971395.backup.b95dff28-554e-45c2-a740-5f66c9c2f3d4' {{(pid=129697) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Apr 02 21:13:03.472791 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] Differential backup transfer completed in 1.0929s {{(pid=129697) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Apr 02 21:13:03.473995 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] Getting metadata type 'volume-base-metadata' {{(pid=129697) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Apr 02 21:13:03.490285 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129697) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Apr 02 21:13:03.490901 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] Getting metadata type 'volume-metadata' {{(pid=129697) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Apr 02 21:13:03.502358 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] No metadata type 'volume-metadata' available {{(pid=129697) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Apr 02 21:13:03.502969 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] Getting metadata type 'volume-glance-metadata' {{(pid=129697) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Apr 02 21:13:03.510382 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] No metadata type 'volume-glance-metadata' available {{(pid=129697) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Apr 02 21:13:03.511182 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] Backing up metadata for volume 4be85628-1d3b-468d-96cb-ed8984971395. {{(pid=129697) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Apr 02 21:13:03.585089 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] Backup 'b95dff28-554e-45c2-a740-5f66c9c2f3d4' of volume 4be85628-1d3b-468d-96cb-ed8984971395 finished. {{(pid=129697) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Apr 02 21:13:03.587411 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-4be85628-1d3b-468d-96cb-ed8984971395', 'hosts': ['10.222.0.18'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '4be85628-1d3b-468d-96cb-ed8984971395', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:13:03.645959 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] <== disconnect_volume: return (50ms) None {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:13:03.728030 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-cf8fc03e-7dd9-4280-bf27-68a8b2934a83 tempest-VolumesBackupsAdminTest-119965354 None] Create backup finished. backup: b95dff28-554e-45c2-a740-5f66c9c2f3d4. Apr 02 21:13:04.252782 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-2bb1ecee-1cd8-48a0-987d-c41a8c1bc61a tempest-VolumesBackupsAdminTest-599724220 None] Reset backup status started, backup_id: b95dff28-554e-45c2-a740-5f66c9c2f3d4, status: error. Apr 02 21:13:04.253441 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-2bb1ecee-1cd8-48a0-987d-c41a8c1bc61a tempest-VolumesBackupsAdminTest-599724220 None] Backup service: cinder.backup.drivers.ceph.CephBackupDriver. Apr 02 21:13:04.329239 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-e3217863-2fa1-46fc-973c-106d43e1026f tempest-VolumesBackupsAdminTest-119965354 None] Delete backup started, backup: b95dff28-554e-45c2-a740-5f66c9c2f3d4. Apr 02 21:13:04.332899 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-e3217863-2fa1-46fc-973c-106d43e1026f tempest-VolumesBackupsAdminTest-119965354 None] Delete started for backup=b95dff28-554e-45c2-a740-5f66c9c2f3d4 {{(pid=129697) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Apr 02 21:13:04.333518 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-e3217863-2fa1-46fc-973c-106d43e1026f tempest-VolumesBackupsAdminTest-119965354 None] Trying diff format basename='volume-4be85628-1d3b-468d-96cb-ed8984971395.backup.b95dff28-554e-45c2-a740-5f66c9c2f3d4' for backup base image of volume 4be85628-1d3b-468d-96cb-ed8984971395. {{(pid=129697) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Apr 02 21:13:04.410379 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-e3217863-2fa1-46fc-973c-106d43e1026f tempest-VolumesBackupsAdminTest-119965354 None] Looking for snapshot of backup base 'volume-4be85628-1d3b-468d-96cb-ed8984971395.backup.b95dff28-554e-45c2-a740-5f66c9c2f3d4' {{(pid=129697) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Apr 02 21:13:04.410763 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-e3217863-2fa1-46fc-973c-106d43e1026f tempest-VolumesBackupsAdminTest-119965354 None] Found snapshot 'backup.b95dff28-554e-45c2-a740-5f66c9c2f3d4.snap.1648933982.2781463' {{(pid=129697) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Apr 02 21:13:04.411684 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-e3217863-2fa1-46fc-973c-106d43e1026f tempest-VolumesBackupsAdminTest-119965354 None] Deleting backup snapshot='backup.b95dff28-554e-45c2-a740-5f66c9c2f3d4.snap.1648933982.2781463' {{(pid=129697) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Apr 02 21:13:05.483939 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.drivers.ceph [None req-e3217863-2fa1-46fc-973c-106d43e1026f tempest-VolumesBackupsAdminTest-119965354 None] Deleting backup base image='volume-4be85628-1d3b-468d-96cb-ed8984971395.backup.b95dff28-554e-45c2-a740-5f66c9c2f3d4' of volume 4be85628-1d3b-468d-96cb-ed8984971395. Apr 02 21:13:05.817783 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-e3217863-2fa1-46fc-973c-106d43e1026f tempest-VolumesBackupsAdminTest-119965354 None] Base backup image='volume-4be85628-1d3b-468d-96cb-ed8984971395.backup.b95dff28-554e-45c2-a740-5f66c9c2f3d4' of volume 4be85628-1d3b-468d-96cb-ed8984971395 deleted. {{(pid=129697) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Apr 02 21:13:05.881763 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-e3217863-2fa1-46fc-973c-106d43e1026f tempest-VolumesBackupsAdminTest-119965354 None] Delete of backup 'b95dff28-554e-45c2-a740-5f66c9c2f3d4' for volume '4be85628-1d3b-468d-96cb-ed8984971395' finished. {{(pid=129697) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Apr 02 21:13:05.889790 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_db.sqlalchemy.engines [None req-e3217863-2fa1-46fc-973c-106d43e1026f tempest-VolumesBackupsAdminTest-119965354 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129697) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 02 21:13:05.938283 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.quota [None req-e3217863-2fa1-46fc-973c-106d43e1026f tempest-VolumesBackupsAdminTest-119965354 None] Created reservations ['87c6e667-c2bc-4fc3-b37a-236ca8a13928', '24fe05bc-5a44-42c4-9d62-99f1af6de930'] {{(pid=129697) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 02 21:13:05.996783 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-e3217863-2fa1-46fc-973c-106d43e1026f tempest-VolumesBackupsAdminTest-119965354 None] Delete backup finished, backup b95dff28-554e-45c2-a740-5f66c9c2f3d4 deleted. Apr 02 21:13:36.288222 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:13:36.290008 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:13:59.029417 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] Create backup started, backup: 571723a0-981b-48cd-b10b-5b7e41cd9630 volume: 9269a2b6-cc9d-40b2-b86f-09d04ce54736. Apr 02 21:13:59.040434 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-04-02T21:13:58Z,data_timestamp=2022-04-02T21:13:58Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsTest-Backup-1066010717',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=571723a0-981b-48cd-b10b-5b7e41cd9630,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='3265dbc2dbab442dbb562ef4a3652650',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='b80790d4e9484dfd92f9ad7e840637c3',volume_id=9269a2b6-cc9d-40b2-b86f-09d04ce54736) Apr 02 21:13:59.111852 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.utils [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.18', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:13:59.119080 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:13:59.146786 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.033s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:13:59.147732 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[3cda0ddb-4d2c-4f8f-bb4d-7e271ee41433]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:2a5dbfa3acf\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:13:59.163432 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:13:59.194100 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 02 21:13:59.194592 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[a2b7890c-3bdf-4eda-a872-53988a62b940]: [Errno 2] No such file or directory: 'systool' {{(pid=157607) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 02 21:13:59.194592 ncdv-master-1 cinder-backup[129697]: Traceback (most recent call last): Apr 02 21:13:59.194592 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 02 21:13:59.194592 ncdv-master-1 cinder-backup[129697]: ret = func(*f_args, **f_kwargs) Apr 02 21:13:59.194592 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 02 21:13:59.194592 ncdv-master-1 cinder-backup[129697]: return func(*args, **kwargs) Apr 02 21:13:59.194592 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 02 21:13:59.194592 ncdv-master-1 cinder-backup[129697]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 02 21:13:59.194592 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 02 21:13:59.194592 ncdv-master-1 cinder-backup[129697]: return putils.execute(on_execute=on_execute, Apr 02 21:13:59.194592 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 02 21:13:59.194592 ncdv-master-1 cinder-backup[129697]: obj = subprocess.Popen(cmd, Apr 02 21:13:59.194592 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 02 21:13:59.194592 ncdv-master-1 cinder-backup[129697]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 02 21:13:59.194592 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 02 21:13:59.194592 ncdv-master-1 cinder-backup[129697]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 02 21:13:59.194592 ncdv-master-1 cinder-backup[129697]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 02 21:13:59.196287 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[a2b7890c-3bdf-4eda-a872-53988a62b940]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:13:59.197017 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:13:59.227053 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 02 21:13:59.227759 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[4a677685-1a88-430b-9807-7ae728444461]: [Errno 2] No such file or directory: 'systool' {{(pid=157607) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 02 21:13:59.227759 ncdv-master-1 cinder-backup[129697]: Traceback (most recent call last): Apr 02 21:13:59.227759 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 02 21:13:59.227759 ncdv-master-1 cinder-backup[129697]: ret = func(*f_args, **f_kwargs) Apr 02 21:13:59.227759 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 02 21:13:59.227759 ncdv-master-1 cinder-backup[129697]: return func(*args, **kwargs) Apr 02 21:13:59.227759 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 02 21:13:59.227759 ncdv-master-1 cinder-backup[129697]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 02 21:13:59.227759 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 02 21:13:59.227759 ncdv-master-1 cinder-backup[129697]: return putils.execute(on_execute=on_execute, Apr 02 21:13:59.227759 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 02 21:13:59.227759 ncdv-master-1 cinder-backup[129697]: obj = subprocess.Popen(cmd, Apr 02 21:13:59.227759 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 02 21:13:59.227759 ncdv-master-1 cinder-backup[129697]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 02 21:13:59.227759 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 02 21:13:59.227759 ncdv-master-1 cinder-backup[129697]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 02 21:13:59.227759 ncdv-master-1 cinder-backup[129697]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 02 21:13:59.229241 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[4a677685-1a88-430b-9807-7ae728444461]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:13:59.229928 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:13:59.261772 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.031s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:13:59.262439 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[53c80cb4-30df-4969-b707-ef317b632b40]: (4, ('/dev/sda1\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:13:59.265841 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:13:59.283059 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.017s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:13:59.283344 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[266e4555-a08b-48a6-a03e-a7e00f5f55aa]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:13:59.285448 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:13:59.302510 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.016s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:13:59.302803 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[176ab043-06d3-4f4e-94c6-d1cf18c24a53]: (4, ('c36434ea-c598-ca44-bb67-56d5c2bbf98e\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:13:59.303967 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] Running cmd (subprocess): nvme version {{(pid=129697) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:13:59.331402 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] CMD "nvme version" returned: 0 in 0.027s {{(pid=129697) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:13:59.334194 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129697) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Apr 02 21:13:59.334860 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129697) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Apr 02 21:13:59.336295 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:25346655-e237-4641-aa32-2fe34121db47 dsc: {{(pid=129697) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Apr 02 21:13:59.337547 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.utils [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] <== get_connector_properties: return (225ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.18', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:2a5dbfa3acf', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c36434ea-c598-ca44-bb67-56d5c2bbf98e', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:25346655-e237-4641-aa32-2fe34121db47', 'found_dsc': ''} {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:14:00.541955 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connector [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] Factory for rbd on None {{(pid=129697) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Apr 02 21:14:00.544911 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-9269a2b6-cc9d-40b2-b86f-09d04ce54736', 'hosts': ['10.222.0.18'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '9269a2b6-cc9d-40b2-b86f-09d04ce54736', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:14:00.546232 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.linuxrbd [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] opening connection to ceph cluster (timeout=-1). {{(pid=129697) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Apr 02 21:14:00.599096 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] <== connect_volume: return (52ms) {'path': } {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:14:00.607271 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_db.sqlalchemy.engines [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129697) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 02 21:14:00.637003 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] Starting backup of volume='9269a2b6-cc9d-40b2-b86f-09d04ce54736'. {{(pid=129697) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Apr 02 21:14:00.637928 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] Volume file is RBD: attempting optimized backup {{(pid=129697) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Apr 02 21:14:00.685221 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] Creating base image 'volume-9269a2b6-cc9d-40b2-b86f-09d04ce54736.backup.571723a0-981b-48cd-b10b-5b7e41cd9630' {{(pid=129697) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Apr 02 21:14:00.685727 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129697) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Apr 02 21:14:00.767649 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] Using --from-snap 'None' for incremental backup of volume 9269a2b6-cc9d-40b2-b86f-09d04ce54736. {{(pid=129697) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Apr 02 21:14:00.768504 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] Creating backup snapshot='backup.571723a0-981b-48cd-b10b-5b7e41cd9630.snap.1648934040.7676485' {{(pid=129697) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Apr 02 21:14:01.792675 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] Performing differential transfer from 'volume-9269a2b6-cc9d-40b2-b86f-09d04ce54736' to 'volume-9269a2b6-cc9d-40b2-b86f-09d04ce54736.backup.571723a0-981b-48cd-b10b-5b7e41cd9630' {{(pid=129697) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Apr 02 21:14:01.793729 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_kgzfh3v6 --pool volumes volumes/volume-9269a2b6-cc9d-40b2-b86f-09d04ce54736@backup.571723a0-981b-48cd-b10b-5b7e41cd9630.snap.1648934040.7676485 -' into... {{(pid=129697) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Apr 02 21:14:01.793729 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-9269a2b6-cc9d-40b2-b86f-09d04ce54736.backup.571723a0-981b-48cd-b10b-5b7e41cd9630' {{(pid=129697) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Apr 02 21:14:02.823106 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] Differential backup transfer completed in 1.0299s {{(pid=129697) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Apr 02 21:14:02.824777 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] Getting metadata type 'volume-base-metadata' {{(pid=129697) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Apr 02 21:14:02.841529 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129697) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Apr 02 21:14:02.842152 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] Getting metadata type 'volume-metadata' {{(pid=129697) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Apr 02 21:14:02.851406 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] No metadata type 'volume-metadata' available {{(pid=129697) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Apr 02 21:14:02.852623 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] Getting metadata type 'volume-glance-metadata' {{(pid=129697) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Apr 02 21:14:02.868273 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] No metadata type 'volume-glance-metadata' available {{(pid=129697) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Apr 02 21:14:02.870119 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] Backing up metadata for volume 9269a2b6-cc9d-40b2-b86f-09d04ce54736. {{(pid=129697) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Apr 02 21:14:02.959451 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] Backup '571723a0-981b-48cd-b10b-5b7e41cd9630' of volume 9269a2b6-cc9d-40b2-b86f-09d04ce54736 finished. {{(pid=129697) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Apr 02 21:14:02.961751 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-9269a2b6-cc9d-40b2-b86f-09d04ce54736', 'hosts': ['10.222.0.18'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '9269a2b6-cc9d-40b2-b86f-09d04ce54736', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:14:02.985207 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] <== disconnect_volume: return (22ms) None {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:14:03.117693 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-73941536-a506-473d-a267-e03bdef2c5ee tempest-VolumesBackupsTest-990025423 None] Create backup finished. backup: 571723a0-981b-48cd-b10b-5b7e41cd9630. Apr 02 21:14:03.401511 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-953c3a24-58b6-4c00-9ec5-3531edcac43e tempest-VolumesBackupsTest-990025423 None] Delete backup started, backup: 571723a0-981b-48cd-b10b-5b7e41cd9630. Apr 02 21:14:03.403395 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-953c3a24-58b6-4c00-9ec5-3531edcac43e tempest-VolumesBackupsTest-990025423 None] Delete started for backup=571723a0-981b-48cd-b10b-5b7e41cd9630 {{(pid=129697) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Apr 02 21:14:03.404215 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-953c3a24-58b6-4c00-9ec5-3531edcac43e tempest-VolumesBackupsTest-990025423 None] Trying diff format basename='volume-9269a2b6-cc9d-40b2-b86f-09d04ce54736.backup.571723a0-981b-48cd-b10b-5b7e41cd9630' for backup base image of volume 9269a2b6-cc9d-40b2-b86f-09d04ce54736. {{(pid=129697) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Apr 02 21:14:03.479573 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-953c3a24-58b6-4c00-9ec5-3531edcac43e tempest-VolumesBackupsTest-990025423 None] Looking for snapshot of backup base 'volume-9269a2b6-cc9d-40b2-b86f-09d04ce54736.backup.571723a0-981b-48cd-b10b-5b7e41cd9630' {{(pid=129697) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Apr 02 21:14:03.479956 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-953c3a24-58b6-4c00-9ec5-3531edcac43e tempest-VolumesBackupsTest-990025423 None] Found snapshot 'backup.571723a0-981b-48cd-b10b-5b7e41cd9630.snap.1648934040.7676485' {{(pid=129697) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Apr 02 21:14:03.480336 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-953c3a24-58b6-4c00-9ec5-3531edcac43e tempest-VolumesBackupsTest-990025423 None] Deleting backup snapshot='backup.571723a0-981b-48cd-b10b-5b7e41cd9630.snap.1648934040.7676485' {{(pid=129697) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Apr 02 21:14:04.603020 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.drivers.ceph [None req-953c3a24-58b6-4c00-9ec5-3531edcac43e tempest-VolumesBackupsTest-990025423 None] Deleting backup base image='volume-9269a2b6-cc9d-40b2-b86f-09d04ce54736.backup.571723a0-981b-48cd-b10b-5b7e41cd9630' of volume 9269a2b6-cc9d-40b2-b86f-09d04ce54736. Apr 02 21:14:04.917579 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-953c3a24-58b6-4c00-9ec5-3531edcac43e tempest-VolumesBackupsTest-990025423 None] Base backup image='volume-9269a2b6-cc9d-40b2-b86f-09d04ce54736.backup.571723a0-981b-48cd-b10b-5b7e41cd9630' of volume 9269a2b6-cc9d-40b2-b86f-09d04ce54736 deleted. {{(pid=129697) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Apr 02 21:14:04.968075 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-953c3a24-58b6-4c00-9ec5-3531edcac43e tempest-VolumesBackupsTest-990025423 None] Delete of backup '571723a0-981b-48cd-b10b-5b7e41cd9630' for volume '9269a2b6-cc9d-40b2-b86f-09d04ce54736' finished. {{(pid=129697) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Apr 02 21:14:04.976949 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_db.sqlalchemy.engines [None req-953c3a24-58b6-4c00-9ec5-3531edcac43e tempest-VolumesBackupsTest-990025423 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129697) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 02 21:14:05.009150 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.quota [None req-953c3a24-58b6-4c00-9ec5-3531edcac43e tempest-VolumesBackupsTest-990025423 None] Created reservations ['75cb9e39-942b-4156-b96a-a7bd55558388', 'c80ef9b1-33bd-4c75-8d85-50f65aa978a7'] {{(pid=129697) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 02 21:14:05.085180 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-953c3a24-58b6-4c00-9ec5-3531edcac43e tempest-VolumesBackupsTest-990025423 None] Delete backup finished, backup 571723a0-981b-48cd-b10b-5b7e41cd9630 deleted. Apr 02 21:14:16.179705 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] Create backup started, backup: c0fa9c74-c64b-42ce-84a0-fb850cace5c9 volume: 0719ac9f-ffc0-414d-823e-794906e1ac85. Apr 02 21:14:16.192210 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-04-02T21:14:16Z,data_timestamp=2022-04-02T21:14:16Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsTest-Backup-1999085688',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=c0fa9c74-c64b-42ce-84a0-fb850cace5c9,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='3265dbc2dbab442dbb562ef4a3652650',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='b80790d4e9484dfd92f9ad7e840637c3',volume_id=0719ac9f-ffc0-414d-823e-794906e1ac85) Apr 02 21:14:16.257705 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.utils [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.18', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:14:16.261608 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:16.283669 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.024s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:14:16.284051 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[6577feda-da25-426e-84d3-912128398543]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:2a5dbfa3acf\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:14:16.286103 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:16.309265 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 02 21:14:16.309906 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[f96f7122-28e6-46bb-9a95-c2d72e726a76]: [Errno 2] No such file or directory: 'systool' {{(pid=157607) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 02 21:14:16.309906 ncdv-master-1 cinder-backup[129697]: Traceback (most recent call last): Apr 02 21:14:16.309906 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 02 21:14:16.309906 ncdv-master-1 cinder-backup[129697]: ret = func(*f_args, **f_kwargs) Apr 02 21:14:16.309906 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 02 21:14:16.309906 ncdv-master-1 cinder-backup[129697]: return func(*args, **kwargs) Apr 02 21:14:16.309906 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 02 21:14:16.309906 ncdv-master-1 cinder-backup[129697]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 02 21:14:16.309906 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 02 21:14:16.309906 ncdv-master-1 cinder-backup[129697]: return putils.execute(on_execute=on_execute, Apr 02 21:14:16.309906 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 02 21:14:16.309906 ncdv-master-1 cinder-backup[129697]: obj = subprocess.Popen(cmd, Apr 02 21:14:16.309906 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 02 21:14:16.309906 ncdv-master-1 cinder-backup[129697]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 02 21:14:16.309906 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 02 21:14:16.309906 ncdv-master-1 cinder-backup[129697]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 02 21:14:16.309906 ncdv-master-1 cinder-backup[129697]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 02 21:14:16.311542 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[f96f7122-28e6-46bb-9a95-c2d72e726a76]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:14:16.313185 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:16.340960 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 02 21:14:16.341443 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[5cb63679-89bd-4c8d-944f-83cc36d05629]: [Errno 2] No such file or directory: 'systool' {{(pid=157607) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 02 21:14:16.341443 ncdv-master-1 cinder-backup[129697]: Traceback (most recent call last): Apr 02 21:14:16.341443 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 02 21:14:16.341443 ncdv-master-1 cinder-backup[129697]: ret = func(*f_args, **f_kwargs) Apr 02 21:14:16.341443 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 02 21:14:16.341443 ncdv-master-1 cinder-backup[129697]: return func(*args, **kwargs) Apr 02 21:14:16.341443 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 02 21:14:16.341443 ncdv-master-1 cinder-backup[129697]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 02 21:14:16.341443 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 02 21:14:16.341443 ncdv-master-1 cinder-backup[129697]: return putils.execute(on_execute=on_execute, Apr 02 21:14:16.341443 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 02 21:14:16.341443 ncdv-master-1 cinder-backup[129697]: obj = subprocess.Popen(cmd, Apr 02 21:14:16.341443 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 02 21:14:16.341443 ncdv-master-1 cinder-backup[129697]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 02 21:14:16.341443 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 02 21:14:16.341443 ncdv-master-1 cinder-backup[129697]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 02 21:14:16.341443 ncdv-master-1 cinder-backup[129697]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 02 21:14:16.343332 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[5cb63679-89bd-4c8d-944f-83cc36d05629]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:14:16.347878 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:16.374781 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.029s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:14:16.376106 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[15a2ecfb-1c17-4dcb-b77b-198d1b6d337f]: (4, ('/dev/sda1\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:14:16.378892 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:16.406726 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.028s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:14:16.407545 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[e069fb31-f998-44f0-be18-3b2f825ba258]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:14:16.410797 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:16.435241 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.023s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:14:16.435738 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[51cf9301-c375-4e8b-a3c1-6ca05637bfcf]: (4, ('c36434ea-c598-ca44-bb67-56d5c2bbf98e\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:14:16.437436 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] Running cmd (subprocess): nvme version {{(pid=129697) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:16.495512 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] CMD "nvme version" returned: 0 in 0.057s {{(pid=129697) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:14:16.510028 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_db.sqlalchemy.engines [None req-93bcaa7a-b4bf-4a4e-ad50-4a6b44355801 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129697) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 02 21:14:16.519008 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129697) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Apr 02 21:14:16.519928 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129697) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Apr 02 21:14:16.520511 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:25346655-e237-4641-aa32-2fe34121db47 dsc: {{(pid=129697) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Apr 02 21:14:16.521584 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.utils [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] <== get_connector_properties: return (263ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.18', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:2a5dbfa3acf', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c36434ea-c598-ca44-bb67-56d5c2bbf98e', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:25346655-e237-4641-aa32-2fe34121db47', 'found_dsc': ''} {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:14:17.582483 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connector [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] Factory for rbd on None {{(pid=129697) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Apr 02 21:14:17.584592 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-0719ac9f-ffc0-414d-823e-794906e1ac85', 'hosts': ['10.222.0.18'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '0719ac9f-ffc0-414d-823e-794906e1ac85', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:14:17.586447 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.linuxrbd [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] opening connection to ceph cluster (timeout=-1). {{(pid=129697) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Apr 02 21:14:17.708203 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] <== connect_volume: return (122ms) {'path': } {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:14:17.751050 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] Starting backup of volume='0719ac9f-ffc0-414d-823e-794906e1ac85'. {{(pid=129697) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Apr 02 21:14:17.752331 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] Volume file is RBD: attempting optimized backup {{(pid=129697) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Apr 02 21:14:17.807856 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] Creating base image 'volume-0719ac9f-ffc0-414d-823e-794906e1ac85.backup.c0fa9c74-c64b-42ce-84a0-fb850cace5c9' {{(pid=129697) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Apr 02 21:14:17.809018 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129697) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Apr 02 21:14:17.887149 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] Using --from-snap 'None' for incremental backup of volume 0719ac9f-ffc0-414d-823e-794906e1ac85. {{(pid=129697) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Apr 02 21:14:17.887526 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] Creating backup snapshot='backup.c0fa9c74-c64b-42ce-84a0-fb850cace5c9.snap.1648934057.8870456' {{(pid=129697) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Apr 02 21:14:18.782319 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] Performing differential transfer from 'volume-0719ac9f-ffc0-414d-823e-794906e1ac85' to 'volume-0719ac9f-ffc0-414d-823e-794906e1ac85.backup.c0fa9c74-c64b-42ce-84a0-fb850cace5c9' {{(pid=129697) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Apr 02 21:14:18.782979 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_j8flju0j --pool volumes volumes/volume-0719ac9f-ffc0-414d-823e-794906e1ac85@backup.c0fa9c74-c64b-42ce-84a0-fb850cace5c9.snap.1648934057.8870456 -' into... {{(pid=129697) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Apr 02 21:14:18.783564 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-0719ac9f-ffc0-414d-823e-794906e1ac85.backup.c0fa9c74-c64b-42ce-84a0-fb850cace5c9' {{(pid=129697) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Apr 02 21:14:27.713324 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] Differential backup transfer completed in 8.9309s {{(pid=129697) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Apr 02 21:14:27.714088 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] Getting metadata type 'volume-base-metadata' {{(pid=129697) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Apr 02 21:14:27.726914 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129697) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Apr 02 21:14:27.727141 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] Getting metadata type 'volume-metadata' {{(pid=129697) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Apr 02 21:14:27.737005 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] No metadata type 'volume-metadata' available {{(pid=129697) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Apr 02 21:14:27.737403 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] Getting metadata type 'volume-glance-metadata' {{(pid=129697) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Apr 02 21:14:27.753783 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=129697) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}} Apr 02 21:14:27.755500 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] Backing up metadata for volume 0719ac9f-ffc0-414d-823e-794906e1ac85. {{(pid=129697) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Apr 02 21:14:27.821391 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] Backup 'c0fa9c74-c64b-42ce-84a0-fb850cace5c9' of volume 0719ac9f-ffc0-414d-823e-794906e1ac85 finished. {{(pid=129697) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Apr 02 21:14:27.823051 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-0719ac9f-ffc0-414d-823e-794906e1ac85', 'hosts': ['10.222.0.18'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '0719ac9f-ffc0-414d-823e-794906e1ac85', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:14:27.872245 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] <== disconnect_volume: return (47ms) None {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:14:27.995877 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-9f207119-fb13-48a3-8e6d-1314ade522a7 tempest-VolumesBackupsTest-990025423 None] Create backup finished. backup: c0fa9c74-c64b-42ce-84a0-fb850cace5c9. Apr 02 21:14:29.188224 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] Create backup started, backup: d77139f9-4985-42bf-8c9f-04be2be11db4 volume: c87f93d6-de80-42a3-9d17-3629759f6c8b snapshot: 4c2685a2-a52a-4e3c-8955-33dada66c326. Apr 02 21:14:29.211200 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-04-02T21:14:28Z,data_timestamp=2022-04-02T21:14:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Backup-1688307190',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=d77139f9-4985-42bf-8c9f-04be2be11db4,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='b5fe471f4a924fd98ef24aa24abb736c',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id='4c2685a2-a52a-4e3c-8955-33dada66c326',status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='08aca3700d594a349789d5f3847894dc',volume_id=c87f93d6-de80-42a3-9d17-3629759f6c8b) Apr 02 21:14:29.882665 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.utils [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.18', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:14:29.892538 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:29.926782 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.040s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:14:29.928825 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[95a8dc45-eb10-4d8d-bdd2-c46882cdd3c6]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:2a5dbfa3acf\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:14:29.931093 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:29.941669 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 02 21:14:29.942250 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[aadf4a10-b18d-42b6-95b4-33a75c5008f5]: [Errno 2] No such file or directory: 'systool' {{(pid=157607) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 02 21:14:29.942250 ncdv-master-1 cinder-backup[129697]: Traceback (most recent call last): Apr 02 21:14:29.942250 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 02 21:14:29.942250 ncdv-master-1 cinder-backup[129697]: ret = func(*f_args, **f_kwargs) Apr 02 21:14:29.942250 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 02 21:14:29.942250 ncdv-master-1 cinder-backup[129697]: return func(*args, **kwargs) Apr 02 21:14:29.942250 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 02 21:14:29.942250 ncdv-master-1 cinder-backup[129697]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 02 21:14:29.942250 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 02 21:14:29.942250 ncdv-master-1 cinder-backup[129697]: return putils.execute(on_execute=on_execute, Apr 02 21:14:29.942250 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 02 21:14:29.942250 ncdv-master-1 cinder-backup[129697]: obj = subprocess.Popen(cmd, Apr 02 21:14:29.942250 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 02 21:14:29.942250 ncdv-master-1 cinder-backup[129697]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 02 21:14:29.942250 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 02 21:14:29.942250 ncdv-master-1 cinder-backup[129697]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 02 21:14:29.942250 ncdv-master-1 cinder-backup[129697]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 02 21:14:29.944030 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[aadf4a10-b18d-42b6-95b4-33a75c5008f5]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:14:29.947629 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:29.967277 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 02 21:14:29.967559 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[85ae4d19-c69b-42da-81fc-2947dee98c59]: [Errno 2] No such file or directory: 'systool' {{(pid=157607) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 02 21:14:29.967559 ncdv-master-1 cinder-backup[129697]: Traceback (most recent call last): Apr 02 21:14:29.967559 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 02 21:14:29.967559 ncdv-master-1 cinder-backup[129697]: ret = func(*f_args, **f_kwargs) Apr 02 21:14:29.967559 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 02 21:14:29.967559 ncdv-master-1 cinder-backup[129697]: return func(*args, **kwargs) Apr 02 21:14:29.967559 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 02 21:14:29.967559 ncdv-master-1 cinder-backup[129697]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 02 21:14:29.967559 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 02 21:14:29.967559 ncdv-master-1 cinder-backup[129697]: return putils.execute(on_execute=on_execute, Apr 02 21:14:29.967559 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 02 21:14:29.967559 ncdv-master-1 cinder-backup[129697]: obj = subprocess.Popen(cmd, Apr 02 21:14:29.967559 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 02 21:14:29.967559 ncdv-master-1 cinder-backup[129697]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 02 21:14:29.967559 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 02 21:14:29.967559 ncdv-master-1 cinder-backup[129697]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 02 21:14:29.967559 ncdv-master-1 cinder-backup[129697]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 02 21:14:29.969067 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[85ae4d19-c69b-42da-81fc-2947dee98c59]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:14:29.973191 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:29.994829 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.022s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:14:29.995490 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[9169a2d5-bb66-4eb6-9f7b-c98f0261f580]: (4, ('/dev/sda1\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:14:29.998902 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:30.022417 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.023s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:14:30.022705 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[34a07b66-03ec-45d2-8559-c92d70bee133]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:14:30.028181 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:30.070503 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.045s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:14:30.071505 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[c5d63453-1d19-4a60-9230-5d78c556e74b]: (4, ('c36434ea-c598-ca44-bb67-56d5c2bbf98e\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:14:30.073746 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] Running cmd (subprocess): nvme version {{(pid=129697) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:30.143467 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] CMD "nvme version" returned: 0 in 0.070s {{(pid=129697) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:14:30.145589 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129697) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Apr 02 21:14:30.146072 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129697) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Apr 02 21:14:30.146569 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:25346655-e237-4641-aa32-2fe34121db47 dsc: {{(pid=129697) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Apr 02 21:14:30.147358 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.utils [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] <== get_connector_properties: return (264ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.18', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:2a5dbfa3acf', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c36434ea-c598-ca44-bb67-56d5c2bbf98e', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:25346655-e237-4641-aa32-2fe34121db47', 'found_dsc': ''} {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:14:30.522936 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] Restore backup started, backup: c0fa9c74-c64b-42ce-84a0-fb850cace5c9 volume: 53aa51fa-494f-4726-ba3d-6510fd7e0a33. Apr 02 21:14:30.545917 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_db.sqlalchemy.engines [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129697) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 02 21:14:30.591330 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.utils [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.18', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:14:30.605494 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:30.644849 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.039s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:14:30.645270 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[4e218ea3-2952-4b7c-9b1d-ff67bf7d357e]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:2a5dbfa3acf\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:14:30.646775 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:30.675383 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 02 21:14:30.676045 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[104e82c4-cee7-4edf-88e7-04943ec46ad2]: [Errno 2] No such file or directory: 'systool' {{(pid=157607) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 02 21:14:30.676045 ncdv-master-1 cinder-backup[129697]: Traceback (most recent call last): Apr 02 21:14:30.676045 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 02 21:14:30.676045 ncdv-master-1 cinder-backup[129697]: ret = func(*f_args, **f_kwargs) Apr 02 21:14:30.676045 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 02 21:14:30.676045 ncdv-master-1 cinder-backup[129697]: return func(*args, **kwargs) Apr 02 21:14:30.676045 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 02 21:14:30.676045 ncdv-master-1 cinder-backup[129697]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 02 21:14:30.676045 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 02 21:14:30.676045 ncdv-master-1 cinder-backup[129697]: return putils.execute(on_execute=on_execute, Apr 02 21:14:30.676045 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 02 21:14:30.676045 ncdv-master-1 cinder-backup[129697]: obj = subprocess.Popen(cmd, Apr 02 21:14:30.676045 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 02 21:14:30.676045 ncdv-master-1 cinder-backup[129697]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 02 21:14:30.676045 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 02 21:14:30.676045 ncdv-master-1 cinder-backup[129697]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 02 21:14:30.676045 ncdv-master-1 cinder-backup[129697]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 02 21:14:30.677408 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[104e82c4-cee7-4edf-88e7-04943ec46ad2]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:14:30.683155 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:30.711250 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 02 21:14:30.711825 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[326d3dc6-b3ec-482e-9175-1bce30ad9b8e]: [Errno 2] No such file or directory: 'systool' {{(pid=157607) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 02 21:14:30.711825 ncdv-master-1 cinder-backup[129697]: Traceback (most recent call last): Apr 02 21:14:30.711825 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 02 21:14:30.711825 ncdv-master-1 cinder-backup[129697]: ret = func(*f_args, **f_kwargs) Apr 02 21:14:30.711825 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 02 21:14:30.711825 ncdv-master-1 cinder-backup[129697]: return func(*args, **kwargs) Apr 02 21:14:30.711825 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 02 21:14:30.711825 ncdv-master-1 cinder-backup[129697]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 02 21:14:30.711825 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 02 21:14:30.711825 ncdv-master-1 cinder-backup[129697]: return putils.execute(on_execute=on_execute, Apr 02 21:14:30.711825 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 02 21:14:30.711825 ncdv-master-1 cinder-backup[129697]: obj = subprocess.Popen(cmd, Apr 02 21:14:30.711825 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 02 21:14:30.711825 ncdv-master-1 cinder-backup[129697]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 02 21:14:30.711825 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 02 21:14:30.711825 ncdv-master-1 cinder-backup[129697]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 02 21:14:30.711825 ncdv-master-1 cinder-backup[129697]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 02 21:14:30.713702 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[326d3dc6-b3ec-482e-9175-1bce30ad9b8e]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:14:30.716857 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:30.748967 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.033s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:14:30.749586 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[6805ea98-bb9c-43f2-991a-bb8fad85c8dc]: (4, ('/dev/sda1\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:14:30.753768 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:30.789916 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.038s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:14:30.790639 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[5ce0413c-f46c-48f3-a313-dc6670f19dd6]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:14:30.794242 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:30.838164 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.045s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:14:30.838873 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[495e4d20-1bd3-4aae-b565-54a741b7321c]: (4, ('c36434ea-c598-ca44-bb67-56d5c2bbf98e\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:14:30.839892 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] Running cmd (subprocess): nvme version {{(pid=129697) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:30.886304 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] CMD "nvme version" returned: 0 in 0.046s {{(pid=129697) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:14:30.889681 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129697) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Apr 02 21:14:30.890063 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129697) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Apr 02 21:14:30.890674 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:25346655-e237-4641-aa32-2fe34121db47 dsc: {{(pid=129697) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Apr 02 21:14:30.891080 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.utils [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] <== get_connector_properties: return (286ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.18', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:2a5dbfa3acf', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c36434ea-c598-ca44-bb67-56d5c2bbf98e', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:25346655-e237-4641-aa32-2fe34121db47', 'found_dsc': ''} {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:14:31.931716 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connector [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] Factory for rbd on None {{(pid=129697) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Apr 02 21:14:31.933922 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-e1774229-73a8-44bb-b628-dc596f4722d2', 'hosts': ['10.222.0.18'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'e1774229-73a8-44bb-b628-dc596f4722d2', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:14:31.935148 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.linuxrbd [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] opening connection to ceph cluster (timeout=-1). {{(pid=129697) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Apr 02 21:14:32.044632 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] <== connect_volume: return (103ms) {'path': } {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:14:32.046648 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_db.sqlalchemy.engines [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129697) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 02 21:14:32.078538 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] Starting backup of volume='c87f93d6-de80-42a3-9d17-3629759f6c8b'. {{(pid=129697) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Apr 02 21:14:32.144559 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129697) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Apr 02 21:14:32.145386 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] Creating backup base image='volume-c87f93d6-de80-42a3-9d17-3629759f6c8b.backup.base' for volume c87f93d6-de80-42a3-9d17-3629759f6c8b. {{(pid=129697) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:830}} Apr 02 21:14:32.209816 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] Copying data from volume c87f93d6-de80-42a3-9d17-3629759f6c8b. {{(pid=129697) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:842}} Apr 02 21:14:32.234444 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] Transferring data between 'volume-c87f93d6-de80-42a3-9d17-3629759f6c8b' and 'volume-c87f93d6-de80-42a3-9d17-3629759f6c8b.backup.base' {{(pid=129697) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:399}} Apr 02 21:14:32.235039 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] 8 chunks of 134217728 bytes to be transferred {{(pid=129697) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:403}} Apr 02 21:14:32.336390 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connector [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] Factory for rbd on None {{(pid=129697) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Apr 02 21:14:32.337676 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-53aa51fa-494f-4726-ba3d-6510fd7e0a33', 'hosts': ['10.222.0.18'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '53aa51fa-494f-4726-ba3d-6510fd7e0a33', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:14:32.338613 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.linuxrbd [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] opening connection to ceph cluster (timeout=-1). {{(pid=129697) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Apr 02 21:14:32.489753 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] <== connect_volume: return (151ms) {'path': } {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:14:32.505709 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] Starting restore from Ceph backup=c0fa9c74-c64b-42ce-84a0-fb850cace5c9 to volume=volume-53aa51fa-494f-4726-ba3d-6510fd7e0a33 {{(pid=129697) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1274}} Apr 02 21:14:32.680183 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] Looking for snapshot of backup base 'volume-0719ac9f-ffc0-414d-823e-794906e1ac85.backup.c0fa9c74-c64b-42ce-84a0-fb850cace5c9' {{(pid=129697) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Apr 02 21:14:32.680869 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] Found snapshot 'backup.c0fa9c74-c64b-42ce-84a0-fb850cace5c9.snap.1648934057.8870456' {{(pid=129697) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Apr 02 21:14:32.703620 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] Volume file is RBD. {{(pid=129697) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1184}} Apr 02 21:14:32.808131 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] Attempting differential restore. {{(pid=129697) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1234}} Apr 02 21:14:32.808906 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] Attempting incremental restore from base='volume-0719ac9f-ffc0-414d-823e-794906e1ac85.backup.c0fa9c74-c64b-42ce-84a0-fb850cace5c9' snap='backup.c0fa9c74-c64b-42ce-84a0-fb850cace5c9.snap.1648934057.8870456' {{(pid=129697) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1089}} Apr 02 21:14:32.809466 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] Performing differential transfer from 'volume-0719ac9f-ffc0-414d-823e-794906e1ac85.backup.c0fa9c74-c64b-42ce-84a0-fb850cace5c9' to 'volume-53aa51fa-494f-4726-ba3d-6510fd7e0a33' {{(pid=129697) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Apr 02 21:14:32.810032 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-0719ac9f-ffc0-414d-823e-794906e1ac85.backup.c0fa9c74-c64b-42ce-84a0-fb850cace5c9@backup.c0fa9c74-c64b-42ce-84a0-fb850cace5c9.snap.1648934057.8870456 -' into... {{(pid=129697) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Apr 02 21:14:32.810571 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_j9uu7fx7 --pool volumes - volumes/volume-53aa51fa-494f-4726-ba3d-6510fd7e0a33' {{(pid=129697) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Apr 02 21:14:35.600004 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] Transferred chunk 1 of 8 (38967K/s) {{(pid=129697) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Apr 02 21:14:36.288178 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:14:36.288656 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:14:38.587904 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] Transferred chunk 2 of 8 (43877K/s) {{(pid=129697) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Apr 02 21:14:41.187222 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] Transferred chunk 3 of 8 (50467K/s) {{(pid=129697) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Apr 02 21:14:42.878160 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] Restore transfer completed in 10.0684s {{(pid=129697) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1111}} Apr 02 21:14:42.879487 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=129697) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1285}} Apr 02 21:14:42.940392 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] Restoring volume base metadata {{(pid=129697) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} Apr 02 21:14:42.966437 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] No metadata of type 'volume-metadata' to restore {{(pid=129697) put /opt/stack/cinder/cinder/backup/driver.py:345}} Apr 02 21:14:42.967014 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] Restoring volume glance metadata {{(pid=129697) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}} Apr 02 21:14:43.110950 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] Transferred chunk 4 of 8 (68154K/s) {{(pid=129697) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Apr 02 21:14:43.271867 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] Restore to volume 53aa51fa-494f-4726-ba3d-6510fd7e0a33 finished successfully. {{(pid=129697) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1292}} Apr 02 21:14:43.275552 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-53aa51fa-494f-4726-ba3d-6510fd7e0a33', 'hosts': ['10.222.0.18'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '53aa51fa-494f-4726-ba3d-6510fd7e0a33', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:14:43.300144 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] <== disconnect_volume: return (23ms) None {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:14:43.417795 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.manager [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] Encryption key ID for volume 53aa51fa-494f-4726-ba3d-6510fd7e0a33 already matches encryption key ID in backup c0fa9c74-c64b-42ce-84a0-fb850cace5c9. {{(pid=129697) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} Apr 02 21:14:43.471499 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-3462c51e-100e-4636-914f-3cb2044ba5c3 tempest-VolumesBackupsTest-990025423 None] Finished restoring backup c0fa9c74-c64b-42ce-84a0-fb850cace5c9 to volume 53aa51fa-494f-4726-ba3d-6510fd7e0a33. Apr 02 21:14:44.688855 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] Transferred chunk 5 of 8 (83161K/s) {{(pid=129697) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Apr 02 21:14:46.276231 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] Transferred chunk 6 of 8 (82946K/s) {{(pid=129697) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Apr 02 21:14:48.114208 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-f2eabf03-64d1-4c50-97ba-0402dafc3477 tempest-VolumesBackupsTest-990025423 None] Delete backup started, backup: c0fa9c74-c64b-42ce-84a0-fb850cace5c9. Apr 02 21:14:48.117129 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-f2eabf03-64d1-4c50-97ba-0402dafc3477 tempest-VolumesBackupsTest-990025423 None] Delete started for backup=c0fa9c74-c64b-42ce-84a0-fb850cace5c9 {{(pid=129697) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Apr 02 21:14:48.117669 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-f2eabf03-64d1-4c50-97ba-0402dafc3477 tempest-VolumesBackupsTest-990025423 None] Trying diff format basename='volume-0719ac9f-ffc0-414d-823e-794906e1ac85.backup.c0fa9c74-c64b-42ce-84a0-fb850cace5c9' for backup base image of volume 0719ac9f-ffc0-414d-823e-794906e1ac85. {{(pid=129697) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Apr 02 21:14:48.133563 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] Transferred chunk 7 of 8 (70639K/s) {{(pid=129697) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Apr 02 21:14:48.222241 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-f2eabf03-64d1-4c50-97ba-0402dafc3477 tempest-VolumesBackupsTest-990025423 None] Looking for snapshot of backup base 'volume-0719ac9f-ffc0-414d-823e-794906e1ac85.backup.c0fa9c74-c64b-42ce-84a0-fb850cace5c9' {{(pid=129697) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Apr 02 21:14:48.222600 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-f2eabf03-64d1-4c50-97ba-0402dafc3477 tempest-VolumesBackupsTest-990025423 None] Found snapshot 'backup.c0fa9c74-c64b-42ce-84a0-fb850cace5c9.snap.1648934057.8870456' {{(pid=129697) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Apr 02 21:14:48.222978 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-f2eabf03-64d1-4c50-97ba-0402dafc3477 tempest-VolumesBackupsTest-990025423 None] Deleting backup snapshot='backup.c0fa9c74-c64b-42ce-84a0-fb850cace5c9.snap.1648934057.8870456' {{(pid=129697) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Apr 02 21:14:48.801836 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.drivers.ceph [None req-f2eabf03-64d1-4c50-97ba-0402dafc3477 tempest-VolumesBackupsTest-990025423 None] Deleting backup base image='volume-0719ac9f-ffc0-414d-823e-794906e1ac85.backup.c0fa9c74-c64b-42ce-84a0-fb850cace5c9' of volume 0719ac9f-ffc0-414d-823e-794906e1ac85. Apr 02 21:14:49.854804 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] Transferred chunk 8 of 8 (76178K/s) {{(pid=129697) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Apr 02 21:14:49.993546 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] Getting metadata type 'volume-base-metadata' {{(pid=129697) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Apr 02 21:14:50.005426 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_db.sqlalchemy.engines [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129697) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 02 21:14:50.022979 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129697) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Apr 02 21:14:50.023834 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] Getting metadata type 'volume-metadata' {{(pid=129697) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Apr 02 21:14:50.043477 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] No metadata type 'volume-metadata' available {{(pid=129697) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Apr 02 21:14:50.045750 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] Getting metadata type 'volume-glance-metadata' {{(pid=129697) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Apr 02 21:14:50.066955 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] No metadata type 'volume-glance-metadata' available {{(pid=129697) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Apr 02 21:14:50.068820 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] Backing up metadata for volume c87f93d6-de80-42a3-9d17-3629759f6c8b. {{(pid=129697) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Apr 02 21:14:50.171700 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] Backup 'd77139f9-4985-42bf-8c9f-04be2be11db4' of volume c87f93d6-de80-42a3-9d17-3629759f6c8b finished. {{(pid=129697) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Apr 02 21:14:50.173827 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-e1774229-73a8-44bb-b628-dc596f4722d2', 'hosts': ['10.222.0.18'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'e1774229-73a8-44bb-b628-dc596f4722d2', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:14:50.201951 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] <== disconnect_volume: return (26ms) None {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:14:50.346028 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-eabbc36e-3e70-44f3-8612-060b1b9d17b0 tempest-VolumesSnapshotTestJSON-244026219 None] Create backup finished. backup: d77139f9-4985-42bf-8c9f-04be2be11db4. Apr 02 21:14:50.393818 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-f2eabf03-64d1-4c50-97ba-0402dafc3477 tempest-VolumesBackupsTest-990025423 None] Base backup image='volume-0719ac9f-ffc0-414d-823e-794906e1ac85.backup.c0fa9c74-c64b-42ce-84a0-fb850cace5c9' of volume 0719ac9f-ffc0-414d-823e-794906e1ac85 deleted. {{(pid=129697) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Apr 02 21:14:50.467186 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-f2eabf03-64d1-4c50-97ba-0402dafc3477 tempest-VolumesBackupsTest-990025423 None] Delete of backup 'c0fa9c74-c64b-42ce-84a0-fb850cace5c9' for volume '0719ac9f-ffc0-414d-823e-794906e1ac85' finished. {{(pid=129697) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Apr 02 21:14:50.503911 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.quota [None req-f2eabf03-64d1-4c50-97ba-0402dafc3477 tempest-VolumesBackupsTest-990025423 None] Created reservations ['c4f66d0c-fdc9-4a98-afd9-ea0e7627df17', '3018221e-1c12-437e-9526-7296d8e4886d'] {{(pid=129697) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 02 21:14:50.554405 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-f2eabf03-64d1-4c50-97ba-0402dafc3477 tempest-VolumesBackupsTest-990025423 None] Delete backup finished, backup c0fa9c74-c64b-42ce-84a0-fb850cace5c9 deleted. Apr 02 21:14:51.194169 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-c0bea7d6-d56c-4334-b196-7952c78f7fd0 tempest-VolumesSnapshotTestJSON-244026219 None] Delete backup started, backup: d77139f9-4985-42bf-8c9f-04be2be11db4. Apr 02 21:14:51.195997 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-c0bea7d6-d56c-4334-b196-7952c78f7fd0 tempest-VolumesSnapshotTestJSON-244026219 None] Delete started for backup=d77139f9-4985-42bf-8c9f-04be2be11db4 {{(pid=129697) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Apr 02 21:14:51.196426 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-c0bea7d6-d56c-4334-b196-7952c78f7fd0 tempest-VolumesSnapshotTestJSON-244026219 None] Trying diff format basename='volume-c87f93d6-de80-42a3-9d17-3629759f6c8b.backup.d77139f9-4985-42bf-8c9f-04be2be11db4' for backup base image of volume c87f93d6-de80-42a3-9d17-3629759f6c8b. {{(pid=129697) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Apr 02 21:14:51.318997 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-c0bea7d6-d56c-4334-b196-7952c78f7fd0 tempest-VolumesSnapshotTestJSON-244026219 None] Image 'volume-c87f93d6-de80-42a3-9d17-3629759f6c8b.backup.d77139f9-4985-42bf-8c9f-04be2be11db4' not found - trying diff format name {{(pid=129697) _rbd_image_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:661}} Apr 02 21:14:51.356399 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-c0bea7d6-d56c-4334-b196-7952c78f7fd0 tempest-VolumesSnapshotTestJSON-244026219 None] Looking for snapshot of backup base 'volume-c87f93d6-de80-42a3-9d17-3629759f6c8b.backup.base' {{(pid=129697) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Apr 02 21:14:51.357008 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-c0bea7d6-d56c-4334-b196-7952c78f7fd0 tempest-VolumesSnapshotTestJSON-244026219 None] Backup base 'volume-c87f93d6-de80-42a3-9d17-3629759f6c8b.backup.base' has no snapshots {{(pid=129697) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:908}} Apr 02 21:14:51.357559 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-c0bea7d6-d56c-4334-b196-7952c78f7fd0 tempest-VolumesSnapshotTestJSON-244026219 None] No backup snapshot to delete {{(pid=129697) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:476}} Apr 02 21:14:51.369798 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.drivers.ceph [None req-c0bea7d6-d56c-4334-b196-7952c78f7fd0 tempest-VolumesSnapshotTestJSON-244026219 None] Deleting backup base image='volume-c87f93d6-de80-42a3-9d17-3629759f6c8b.backup.base' of volume c87f93d6-de80-42a3-9d17-3629759f6c8b. Apr 02 21:14:52.508490 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-c0bea7d6-d56c-4334-b196-7952c78f7fd0 tempest-VolumesSnapshotTestJSON-244026219 None] Base backup image='volume-c87f93d6-de80-42a3-9d17-3629759f6c8b.backup.base' of volume c87f93d6-de80-42a3-9d17-3629759f6c8b deleted. {{(pid=129697) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Apr 02 21:14:52.608733 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-c0bea7d6-d56c-4334-b196-7952c78f7fd0 tempest-VolumesSnapshotTestJSON-244026219 None] Delete of backup 'd77139f9-4985-42bf-8c9f-04be2be11db4' for volume 'c87f93d6-de80-42a3-9d17-3629759f6c8b' finished. {{(pid=129697) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Apr 02 21:14:52.615629 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_db.sqlalchemy.engines [None req-c0bea7d6-d56c-4334-b196-7952c78f7fd0 tempest-VolumesSnapshotTestJSON-244026219 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129697) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 02 21:14:52.649348 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.quota [None req-c0bea7d6-d56c-4334-b196-7952c78f7fd0 tempest-VolumesSnapshotTestJSON-244026219 None] Created reservations ['9368f6d2-4f36-4c43-b738-e335e2ca7af1', 'd3e66a90-f585-4a3f-ae97-40672b0fd48a'] {{(pid=129697) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 02 21:14:52.701742 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-c0bea7d6-d56c-4334-b196-7952c78f7fd0 tempest-VolumesSnapshotTestJSON-244026219 None] Delete backup finished, backup d77139f9-4985-42bf-8c9f-04be2be11db4 deleted. Apr 02 21:14:58.825313 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] Create backup started, backup: e8186e17-2214-483f-8aed-135f277aecd4 volume: a18a946b-4c4d-4ffd-8ca7-d1ccf4d98560. Apr 02 21:14:58.834192 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-04-02T21:14:58Z,data_timestamp=2022-04-02T21:14:58Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsV39Test-Backup-1290784737',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=e8186e17-2214-483f-8aed-135f277aecd4,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='f1f1bda071d042bfaa92c36f88a0582f',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='3ada81b2a79d458c807b333caddde24d',volume_id=a18a946b-4c4d-4ffd-8ca7-d1ccf4d98560) Apr 02 21:14:58.901093 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.utils [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.18', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:14:58.903432 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:58.924447 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.020s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:14:58.925150 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[f7028fa3-6c8d-4f63-857c-023018899362]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:2a5dbfa3acf\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:14:58.929917 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:58.938375 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 02 21:14:58.940336 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[802e1d3b-4a17-46e0-a6e1-a35200ead9d0]: [Errno 2] No such file or directory: 'systool' {{(pid=157607) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 02 21:14:58.940336 ncdv-master-1 cinder-backup[129697]: Traceback (most recent call last): Apr 02 21:14:58.940336 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 02 21:14:58.940336 ncdv-master-1 cinder-backup[129697]: ret = func(*f_args, **f_kwargs) Apr 02 21:14:58.940336 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 02 21:14:58.940336 ncdv-master-1 cinder-backup[129697]: return func(*args, **kwargs) Apr 02 21:14:58.940336 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 02 21:14:58.940336 ncdv-master-1 cinder-backup[129697]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 02 21:14:58.940336 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 02 21:14:58.940336 ncdv-master-1 cinder-backup[129697]: return putils.execute(on_execute=on_execute, Apr 02 21:14:58.940336 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 02 21:14:58.940336 ncdv-master-1 cinder-backup[129697]: obj = subprocess.Popen(cmd, Apr 02 21:14:58.940336 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 02 21:14:58.940336 ncdv-master-1 cinder-backup[129697]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 02 21:14:58.940336 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 02 21:14:58.940336 ncdv-master-1 cinder-backup[129697]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 02 21:14:58.940336 ncdv-master-1 cinder-backup[129697]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 02 21:14:58.941843 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[802e1d3b-4a17-46e0-a6e1-a35200ead9d0]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:14:58.943245 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:58.951110 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 02 21:14:58.951772 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[9c3331e9-0cb4-4e45-8039-d17ce8c1630e]: [Errno 2] No such file or directory: 'systool' {{(pid=157607) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 02 21:14:58.951772 ncdv-master-1 cinder-backup[129697]: Traceback (most recent call last): Apr 02 21:14:58.951772 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 02 21:14:58.951772 ncdv-master-1 cinder-backup[129697]: ret = func(*f_args, **f_kwargs) Apr 02 21:14:58.951772 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 02 21:14:58.951772 ncdv-master-1 cinder-backup[129697]: return func(*args, **kwargs) Apr 02 21:14:58.951772 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 02 21:14:58.951772 ncdv-master-1 cinder-backup[129697]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 02 21:14:58.951772 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 02 21:14:58.951772 ncdv-master-1 cinder-backup[129697]: return putils.execute(on_execute=on_execute, Apr 02 21:14:58.951772 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 02 21:14:58.951772 ncdv-master-1 cinder-backup[129697]: obj = subprocess.Popen(cmd, Apr 02 21:14:58.951772 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 02 21:14:58.951772 ncdv-master-1 cinder-backup[129697]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 02 21:14:58.951772 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 02 21:14:58.951772 ncdv-master-1 cinder-backup[129697]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 02 21:14:58.951772 ncdv-master-1 cinder-backup[129697]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 02 21:14:58.953616 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[9c3331e9-0cb4-4e45-8039-d17ce8c1630e]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:14:58.955478 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:58.982649 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.026s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:14:58.983154 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[2183b652-2dc7-4f96-823c-029c2eaadca8]: (4, ('/dev/sda1\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:14:58.987197 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:59.004185 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.019s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:14:59.005413 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[fca1dbcd-547e-4493-9157-a09b2d047bc1]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:14:59.007565 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:59.018736 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.011s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:14:59.019263 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[97ba456f-c21b-4615-ad39-78d22ec3b34a]: (4, ('c36434ea-c598-ca44-bb67-56d5c2bbf98e\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:14:59.020844 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] Running cmd (subprocess): nvme version {{(pid=129697) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:14:59.109614 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] CMD "nvme version" returned: 0 in 0.087s {{(pid=129697) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:14:59.114562 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129697) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Apr 02 21:14:59.115583 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129697) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Apr 02 21:14:59.116556 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:25346655-e237-4641-aa32-2fe34121db47 dsc: {{(pid=129697) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Apr 02 21:14:59.118177 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.utils [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] <== get_connector_properties: return (215ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.18', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:2a5dbfa3acf', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c36434ea-c598-ca44-bb67-56d5c2bbf98e', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:25346655-e237-4641-aa32-2fe34121db47', 'found_dsc': ''} {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:15:00.181058 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connector [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] Factory for rbd on None {{(pid=129697) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Apr 02 21:15:00.182522 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-a18a946b-4c4d-4ffd-8ca7-d1ccf4d98560', 'hosts': ['10.222.0.18'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'a18a946b-4c4d-4ffd-8ca7-d1ccf4d98560', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:15:00.183616 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.linuxrbd [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] opening connection to ceph cluster (timeout=-1). {{(pid=129697) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Apr 02 21:15:00.239623 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] <== connect_volume: return (56ms) {'path': } {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:15:00.248507 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_db.sqlalchemy.engines [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129697) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 02 21:15:00.271141 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] Starting backup of volume='a18a946b-4c4d-4ffd-8ca7-d1ccf4d98560'. {{(pid=129697) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Apr 02 21:15:00.272396 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] Volume file is RBD: attempting optimized backup {{(pid=129697) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Apr 02 21:15:00.310343 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] Creating base image 'volume-a18a946b-4c4d-4ffd-8ca7-d1ccf4d98560.backup.e8186e17-2214-483f-8aed-135f277aecd4' {{(pid=129697) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Apr 02 21:15:00.311041 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129697) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Apr 02 21:15:00.365382 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] Using --from-snap 'None' for incremental backup of volume a18a946b-4c4d-4ffd-8ca7-d1ccf4d98560. {{(pid=129697) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Apr 02 21:15:00.365804 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] Creating backup snapshot='backup.e8186e17-2214-483f-8aed-135f277aecd4.snap.1648934100.3654075' {{(pid=129697) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Apr 02 21:15:00.971182 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] Performing differential transfer from 'volume-a18a946b-4c4d-4ffd-8ca7-d1ccf4d98560' to 'volume-a18a946b-4c4d-4ffd-8ca7-d1ccf4d98560.backup.e8186e17-2214-483f-8aed-135f277aecd4' {{(pid=129697) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Apr 02 21:15:00.971521 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_2x5zrajq --pool volumes volumes/volume-a18a946b-4c4d-4ffd-8ca7-d1ccf4d98560@backup.e8186e17-2214-483f-8aed-135f277aecd4.snap.1648934100.3654075 -' into... {{(pid=129697) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Apr 02 21:15:00.971903 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-a18a946b-4c4d-4ffd-8ca7-d1ccf4d98560.backup.e8186e17-2214-483f-8aed-135f277aecd4' {{(pid=129697) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Apr 02 21:15:02.027014 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] Differential backup transfer completed in 1.0554s {{(pid=129697) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Apr 02 21:15:02.027724 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] Getting metadata type 'volume-base-metadata' {{(pid=129697) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Apr 02 21:15:02.043523 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129697) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Apr 02 21:15:02.043886 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] Getting metadata type 'volume-metadata' {{(pid=129697) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Apr 02 21:15:02.053590 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] No metadata type 'volume-metadata' available {{(pid=129697) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Apr 02 21:15:02.054457 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] Getting metadata type 'volume-glance-metadata' {{(pid=129697) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Apr 02 21:15:02.061085 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] No metadata type 'volume-glance-metadata' available {{(pid=129697) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Apr 02 21:15:02.061961 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] Backing up metadata for volume a18a946b-4c4d-4ffd-8ca7-d1ccf4d98560. {{(pid=129697) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Apr 02 21:15:02.107651 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] Backup 'e8186e17-2214-483f-8aed-135f277aecd4' of volume a18a946b-4c4d-4ffd-8ca7-d1ccf4d98560 finished. {{(pid=129697) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Apr 02 21:15:02.109731 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-a18a946b-4c4d-4ffd-8ca7-d1ccf4d98560', 'hosts': ['10.222.0.18'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'a18a946b-4c4d-4ffd-8ca7-d1ccf4d98560', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:15:02.142956 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] <== disconnect_volume: return (32ms) None {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:15:02.236017 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-2e274e4a-60b7-4d08-aae6-17b20eb2e634 tempest-VolumesBackupsV39Test-1741259811 None] Create backup finished. backup: e8186e17-2214-483f-8aed-135f277aecd4. Apr 02 21:15:03.159804 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-a8e89360-fac0-465a-adf7-f6ee94ea9696 tempest-VolumesBackupsV39Test-1741259811 None] Delete backup started, backup: e8186e17-2214-483f-8aed-135f277aecd4. Apr 02 21:15:03.168622 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-a8e89360-fac0-465a-adf7-f6ee94ea9696 tempest-VolumesBackupsV39Test-1741259811 None] Delete started for backup=e8186e17-2214-483f-8aed-135f277aecd4 {{(pid=129697) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Apr 02 21:15:03.169054 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-a8e89360-fac0-465a-adf7-f6ee94ea9696 tempest-VolumesBackupsV39Test-1741259811 None] Trying diff format basename='volume-a18a946b-4c4d-4ffd-8ca7-d1ccf4d98560.backup.e8186e17-2214-483f-8aed-135f277aecd4' for backup base image of volume a18a946b-4c4d-4ffd-8ca7-d1ccf4d98560. {{(pid=129697) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Apr 02 21:15:03.302065 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-a8e89360-fac0-465a-adf7-f6ee94ea9696 tempest-VolumesBackupsV39Test-1741259811 None] Looking for snapshot of backup base 'volume-a18a946b-4c4d-4ffd-8ca7-d1ccf4d98560.backup.e8186e17-2214-483f-8aed-135f277aecd4' {{(pid=129697) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Apr 02 21:15:03.302462 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-a8e89360-fac0-465a-adf7-f6ee94ea9696 tempest-VolumesBackupsV39Test-1741259811 None] Found snapshot 'backup.e8186e17-2214-483f-8aed-135f277aecd4.snap.1648934100.3654075' {{(pid=129697) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Apr 02 21:15:03.302843 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-a8e89360-fac0-465a-adf7-f6ee94ea9696 tempest-VolumesBackupsV39Test-1741259811 None] Deleting backup snapshot='backup.e8186e17-2214-483f-8aed-135f277aecd4.snap.1648934100.3654075' {{(pid=129697) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Apr 02 21:15:04.076809 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.drivers.ceph [None req-a8e89360-fac0-465a-adf7-f6ee94ea9696 tempest-VolumesBackupsV39Test-1741259811 None] Deleting backup base image='volume-a18a946b-4c4d-4ffd-8ca7-d1ccf4d98560.backup.e8186e17-2214-483f-8aed-135f277aecd4' of volume a18a946b-4c4d-4ffd-8ca7-d1ccf4d98560. Apr 02 21:15:04.479294 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-a8e89360-fac0-465a-adf7-f6ee94ea9696 tempest-VolumesBackupsV39Test-1741259811 None] Base backup image='volume-a18a946b-4c4d-4ffd-8ca7-d1ccf4d98560.backup.e8186e17-2214-483f-8aed-135f277aecd4' of volume a18a946b-4c4d-4ffd-8ca7-d1ccf4d98560 deleted. {{(pid=129697) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Apr 02 21:15:04.533367 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-a8e89360-fac0-465a-adf7-f6ee94ea9696 tempest-VolumesBackupsV39Test-1741259811 None] Delete of backup 'e8186e17-2214-483f-8aed-135f277aecd4' for volume 'a18a946b-4c4d-4ffd-8ca7-d1ccf4d98560' finished. {{(pid=129697) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Apr 02 21:15:04.541124 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_db.sqlalchemy.engines [None req-a8e89360-fac0-465a-adf7-f6ee94ea9696 tempest-VolumesBackupsV39Test-1741259811 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129697) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 02 21:15:04.573904 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.quota [None req-a8e89360-fac0-465a-adf7-f6ee94ea9696 tempest-VolumesBackupsV39Test-1741259811 None] Created reservations ['de967da4-4891-4d3f-b690-eea1a68cecb1', '645ce0a3-4f4f-40f0-aae5-b7c2f5f3356c'] {{(pid=129697) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 02 21:15:04.623352 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-a8e89360-fac0-465a-adf7-f6ee94ea9696 tempest-VolumesBackupsV39Test-1741259811 None] Delete backup finished, backup e8186e17-2214-483f-8aed-135f277aecd4 deleted. Apr 02 21:15:36.287716 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:15:36.288431 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:16:36.283592 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:16:36.284346 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:17:36.289683 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:17:36.290434 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:18:36.295832 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:18:36.296585 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:19:36.298820 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:19:36.299576 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:20:36.303505 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:20:36.304240 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:21:36.299697 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:21:36.300829 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:22:36.297051 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:22:36.297708 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:23:36.294567 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:23:36.295612 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:23:42.338715 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] Create backup started, backup: dc963741-0ed7-4626-92be-bacfe8ce5f2b volume: 24873dd9-832b-44d1-bbec-069663ce55ea. Apr 02 21:23:42.354148 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-04-02T21:23:42Z,data_timestamp=2022-04-02T21:23:42Z,deleted=False,deleted_at=None,display_description=None,display_name=None,encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=dc963741-0ed7-4626-92be-bacfe8ce5f2b,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='0ac7776ae2fb45cfa015c4056f0ab4a0',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='4cb67e8ec9b44d6f89640b4ccabd1987',volume_id=24873dd9-832b-44d1-bbec-069663ce55ea) Apr 02 21:23:42.395258 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.utils [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.18', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:23:42.396653 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:23:42.404345 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.008s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:23:42.404731 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[0e1844bc-0dcd-4062-9b7b-c5ca1b56711b]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:2a5dbfa3acf\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:23:42.405982 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:23:42.412582 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 02 21:23:42.413173 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[12d8f684-d82d-49ba-8a19-a8f50c75ee81]: [Errno 2] No such file or directory: 'systool' {{(pid=157607) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 02 21:23:42.413173 ncdv-master-1 cinder-backup[129697]: Traceback (most recent call last): Apr 02 21:23:42.413173 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 02 21:23:42.413173 ncdv-master-1 cinder-backup[129697]: ret = func(*f_args, **f_kwargs) Apr 02 21:23:42.413173 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 02 21:23:42.413173 ncdv-master-1 cinder-backup[129697]: return func(*args, **kwargs) Apr 02 21:23:42.413173 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 02 21:23:42.413173 ncdv-master-1 cinder-backup[129697]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 02 21:23:42.413173 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 02 21:23:42.413173 ncdv-master-1 cinder-backup[129697]: return putils.execute(on_execute=on_execute, Apr 02 21:23:42.413173 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 02 21:23:42.413173 ncdv-master-1 cinder-backup[129697]: obj = subprocess.Popen(cmd, Apr 02 21:23:42.413173 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 02 21:23:42.413173 ncdv-master-1 cinder-backup[129697]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 02 21:23:42.413173 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 02 21:23:42.413173 ncdv-master-1 cinder-backup[129697]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 02 21:23:42.413173 ncdv-master-1 cinder-backup[129697]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 02 21:23:42.414372 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[12d8f684-d82d-49ba-8a19-a8f50c75ee81]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:23:42.414661 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:23:42.421216 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 02 21:23:42.421790 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[a63ee491-1e23-4a68-8d54-adc8cd1cbe53]: [Errno 2] No such file or directory: 'systool' {{(pid=157607) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 02 21:23:42.421790 ncdv-master-1 cinder-backup[129697]: Traceback (most recent call last): Apr 02 21:23:42.421790 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 02 21:23:42.421790 ncdv-master-1 cinder-backup[129697]: ret = func(*f_args, **f_kwargs) Apr 02 21:23:42.421790 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 02 21:23:42.421790 ncdv-master-1 cinder-backup[129697]: return func(*args, **kwargs) Apr 02 21:23:42.421790 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 02 21:23:42.421790 ncdv-master-1 cinder-backup[129697]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 02 21:23:42.421790 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 02 21:23:42.421790 ncdv-master-1 cinder-backup[129697]: return putils.execute(on_execute=on_execute, Apr 02 21:23:42.421790 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 02 21:23:42.421790 ncdv-master-1 cinder-backup[129697]: obj = subprocess.Popen(cmd, Apr 02 21:23:42.421790 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 02 21:23:42.421790 ncdv-master-1 cinder-backup[129697]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 02 21:23:42.421790 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 02 21:23:42.421790 ncdv-master-1 cinder-backup[129697]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 02 21:23:42.421790 ncdv-master-1 cinder-backup[129697]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 02 21:23:42.422833 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[a63ee491-1e23-4a68-8d54-adc8cd1cbe53]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:23:42.423429 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:23:42.431850 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.008s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:23:42.432175 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[71ae71e5-685d-42c7-bc01-510dda3a428a]: (4, ('/dev/sda1\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:23:42.433419 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:23:42.442107 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.008s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:23:42.442430 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[1ccafe31-12fc-42b0-ba88-46d50b5ecc3e]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:23:42.443519 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:23:42.452521 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.009s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:23:42.452849 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[f5bd89e1-ae46-4521-9a9d-a79ade831925]: (4, ('c36434ea-c598-ca44-bb67-56d5c2bbf98e\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:23:42.454137 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] Running cmd (subprocess): nvme version {{(pid=129697) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:23:42.478992 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] CMD "nvme version" returned: 0 in 0.025s {{(pid=129697) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:23:42.480873 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129697) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Apr 02 21:23:42.481251 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129697) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Apr 02 21:23:42.481628 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:25346655-e237-4641-aa32-2fe34121db47 dsc: {{(pid=129697) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Apr 02 21:23:42.482291 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.utils [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] <== get_connector_properties: return (86ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.18', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:2a5dbfa3acf', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c36434ea-c598-ca44-bb67-56d5c2bbf98e', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:25346655-e237-4641-aa32-2fe34121db47', 'found_dsc': ''} {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:23:43.157392 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connector [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] Factory for rbd on None {{(pid=129697) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Apr 02 21:23:43.158939 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-24873dd9-832b-44d1-bbec-069663ce55ea', 'hosts': ['10.222.0.18'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '24873dd9-832b-44d1-bbec-069663ce55ea', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:23:43.160052 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.linuxrbd [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] opening connection to ceph cluster (timeout=-1). {{(pid=129697) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Apr 02 21:23:43.245462 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] <== connect_volume: return (86ms) {'path': } {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:23:43.254367 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_db.sqlalchemy.engines [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129697) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 02 21:23:43.274338 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] Starting backup of volume='24873dd9-832b-44d1-bbec-069663ce55ea'. {{(pid=129697) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Apr 02 21:23:43.275264 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] Volume file is RBD: attempting optimized backup {{(pid=129697) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Apr 02 21:23:43.298658 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] Creating base image 'volume-24873dd9-832b-44d1-bbec-069663ce55ea.backup.dc963741-0ed7-4626-92be-bacfe8ce5f2b' {{(pid=129697) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Apr 02 21:23:43.299074 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129697) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Apr 02 21:23:43.348940 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] Using --from-snap 'None' for incremental backup of volume 24873dd9-832b-44d1-bbec-069663ce55ea. {{(pid=129697) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Apr 02 21:23:43.350398 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] Creating backup snapshot='backup.dc963741-0ed7-4626-92be-bacfe8ce5f2b.snap.1648934623.3489687' {{(pid=129697) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Apr 02 21:23:43.857667 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] Performing differential transfer from 'volume-24873dd9-832b-44d1-bbec-069663ce55ea' to 'volume-24873dd9-832b-44d1-bbec-069663ce55ea.backup.dc963741-0ed7-4626-92be-bacfe8ce5f2b' {{(pid=129697) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Apr 02 21:23:43.858209 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_yf3jew92 --pool volumes volumes/volume-24873dd9-832b-44d1-bbec-069663ce55ea@backup.dc963741-0ed7-4626-92be-bacfe8ce5f2b.snap.1648934623.3489687 -' into... {{(pid=129697) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Apr 02 21:23:43.858578 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-24873dd9-832b-44d1-bbec-069663ce55ea.backup.dc963741-0ed7-4626-92be-bacfe8ce5f2b' {{(pid=129697) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Apr 02 21:23:49.138354 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] Differential backup transfer completed in 5.2804s {{(pid=129697) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Apr 02 21:23:49.139065 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] Getting metadata type 'volume-base-metadata' {{(pid=129697) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Apr 02 21:23:49.150489 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129697) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Apr 02 21:23:49.150919 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] Getting metadata type 'volume-metadata' {{(pid=129697) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Apr 02 21:23:49.157263 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] No metadata type 'volume-metadata' available {{(pid=129697) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Apr 02 21:23:49.157645 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] Getting metadata type 'volume-glance-metadata' {{(pid=129697) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Apr 02 21:23:49.164799 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=129697) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}} Apr 02 21:23:49.165641 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] Backing up metadata for volume 24873dd9-832b-44d1-bbec-069663ce55ea. {{(pid=129697) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Apr 02 21:23:49.207474 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] Backup 'dc963741-0ed7-4626-92be-bacfe8ce5f2b' of volume 24873dd9-832b-44d1-bbec-069663ce55ea finished. {{(pid=129697) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Apr 02 21:23:49.209156 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-24873dd9-832b-44d1-bbec-069663ce55ea', 'hosts': ['10.222.0.18'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '24873dd9-832b-44d1-bbec-069663ce55ea', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:23:49.233490 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] <== disconnect_volume: return (23ms) None {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:23:49.305206 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-e1920d7b-ed06-4555-9015-1a277af405ee tempest-TestVolumeBackupRestore-163528694 None] Create backup finished. backup: dc963741-0ed7-4626-92be-bacfe8ce5f2b. Apr 02 21:23:50.928781 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] Restore backup started, backup: dc963741-0ed7-4626-92be-bacfe8ce5f2b volume: 3d159ee6-7514-4ca1-803d-49d79bc9076a. Apr 02 21:23:50.944311 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.utils [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.18', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:23:50.945993 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:23:50.956940 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.010s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:23:50.957234 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[4f803f4e-b736-46c5-8059-960097ea0caf]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:2a5dbfa3acf\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:23:50.967291 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:23:50.970084 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 02 21:23:50.971062 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[75472f22-1e83-4dec-8772-fa77930602e5]: [Errno 2] No such file or directory: 'systool' {{(pid=157607) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 02 21:23:50.971062 ncdv-master-1 cinder-backup[129697]: Traceback (most recent call last): Apr 02 21:23:50.971062 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 02 21:23:50.971062 ncdv-master-1 cinder-backup[129697]: ret = func(*f_args, **f_kwargs) Apr 02 21:23:50.971062 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 02 21:23:50.971062 ncdv-master-1 cinder-backup[129697]: return func(*args, **kwargs) Apr 02 21:23:50.971062 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 02 21:23:50.971062 ncdv-master-1 cinder-backup[129697]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 02 21:23:50.971062 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 02 21:23:50.971062 ncdv-master-1 cinder-backup[129697]: return putils.execute(on_execute=on_execute, Apr 02 21:23:50.971062 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 02 21:23:50.971062 ncdv-master-1 cinder-backup[129697]: obj = subprocess.Popen(cmd, Apr 02 21:23:50.971062 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 02 21:23:50.971062 ncdv-master-1 cinder-backup[129697]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 02 21:23:50.971062 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 02 21:23:50.971062 ncdv-master-1 cinder-backup[129697]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 02 21:23:50.971062 ncdv-master-1 cinder-backup[129697]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 02 21:23:50.972113 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[75472f22-1e83-4dec-8772-fa77930602e5]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:23:50.972652 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:23:50.978665 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 02 21:23:50.979066 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[c8319852-ac7a-40d4-8358-cf3a33819e29]: [Errno 2] No such file or directory: 'systool' {{(pid=157607) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 02 21:23:50.979066 ncdv-master-1 cinder-backup[129697]: Traceback (most recent call last): Apr 02 21:23:50.979066 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 02 21:23:50.979066 ncdv-master-1 cinder-backup[129697]: ret = func(*f_args, **f_kwargs) Apr 02 21:23:50.979066 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 02 21:23:50.979066 ncdv-master-1 cinder-backup[129697]: return func(*args, **kwargs) Apr 02 21:23:50.979066 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 02 21:23:50.979066 ncdv-master-1 cinder-backup[129697]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 02 21:23:50.979066 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 02 21:23:50.979066 ncdv-master-1 cinder-backup[129697]: return putils.execute(on_execute=on_execute, Apr 02 21:23:50.979066 ncdv-master-1 cinder-backup[129697]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 02 21:23:50.979066 ncdv-master-1 cinder-backup[129697]: obj = subprocess.Popen(cmd, Apr 02 21:23:50.979066 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 02 21:23:50.979066 ncdv-master-1 cinder-backup[129697]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 02 21:23:50.979066 ncdv-master-1 cinder-backup[129697]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 02 21:23:50.979066 ncdv-master-1 cinder-backup[129697]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 02 21:23:50.979066 ncdv-master-1 cinder-backup[129697]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 02 21:23:50.980366 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[c8319852-ac7a-40d4-8358-cf3a33819e29]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:23:50.981025 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:23:50.992231 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.011s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:23:50.992709 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[639cd9d0-5426-46b2-b8ab-2938bcd3ca59]: (4, ('/dev/sda1\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:23:50.993880 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:23:51.003612 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.009s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:23:51.004014 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[9663b403-4517-499b-aa88-ace14ee54fce]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:23:51.005214 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:23:51.021586 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.016s {{(pid=157607) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:23:51.022415 ncdv-master-1 cinder-backup[129697]: DEBUG oslo.privsep.daemon [-] privsep: reply[01df5a78-34f2-4537-847c-14e14062e415]: (4, ('c36434ea-c598-ca44-bb67-56d5c2bbf98e\n', '')) {{(pid=157607) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 02 21:23:51.023517 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] Running cmd (subprocess): nvme version {{(pid=129697) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 02 21:23:51.048805 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_concurrency.processutils [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] CMD "nvme version" returned: 0 in 0.025s {{(pid=129697) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 02 21:23:51.050579 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129697) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Apr 02 21:23:51.050950 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129697) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Apr 02 21:23:51.051344 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.lightos [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:25346655-e237-4641-aa32-2fe34121db47 dsc: {{(pid=129697) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Apr 02 21:23:51.052020 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.utils [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] <== get_connector_properties: return (106ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.18', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:2a5dbfa3acf', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'c36434ea-c598-ca44-bb67-56d5c2bbf98e', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:25346655-e237-4641-aa32-2fe34121db47', 'found_dsc': ''} {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:23:51.743986 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connector [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] Factory for rbd on None {{(pid=129697) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Apr 02 21:23:51.745455 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-3d159ee6-7514-4ca1-803d-49d79bc9076a', 'hosts': ['10.222.0.18'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '3d159ee6-7514-4ca1-803d-49d79bc9076a', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:23:51.746490 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.linuxrbd [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] opening connection to ceph cluster (timeout=-1). {{(pid=129697) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Apr 02 21:23:51.784167 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] <== connect_volume: return (38ms) {'path': } {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:23:51.792853 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_db.sqlalchemy.engines [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129697) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 02 21:23:51.804014 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] Starting restore from Ceph backup=dc963741-0ed7-4626-92be-bacfe8ce5f2b to volume=volume-3d159ee6-7514-4ca1-803d-49d79bc9076a {{(pid=129697) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1274}} Apr 02 21:23:51.862913 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] Looking for snapshot of backup base 'volume-24873dd9-832b-44d1-bbec-069663ce55ea.backup.dc963741-0ed7-4626-92be-bacfe8ce5f2b' {{(pid=129697) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Apr 02 21:23:51.863294 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] Found snapshot 'backup.dc963741-0ed7-4626-92be-bacfe8ce5f2b.snap.1648934623.3489687' {{(pid=129697) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Apr 02 21:23:51.867436 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] Volume file is RBD. {{(pid=129697) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1184}} Apr 02 21:23:51.895297 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] Attempting differential restore. {{(pid=129697) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1234}} Apr 02 21:23:51.895777 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] Attempting incremental restore from base='volume-24873dd9-832b-44d1-bbec-069663ce55ea.backup.dc963741-0ed7-4626-92be-bacfe8ce5f2b' snap='backup.dc963741-0ed7-4626-92be-bacfe8ce5f2b.snap.1648934623.3489687' {{(pid=129697) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1089}} Apr 02 21:23:51.896204 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] Performing differential transfer from 'volume-24873dd9-832b-44d1-bbec-069663ce55ea.backup.dc963741-0ed7-4626-92be-bacfe8ce5f2b' to 'volume-3d159ee6-7514-4ca1-803d-49d79bc9076a' {{(pid=129697) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Apr 02 21:23:51.896643 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-24873dd9-832b-44d1-bbec-069663ce55ea.backup.dc963741-0ed7-4626-92be-bacfe8ce5f2b@backup.dc963741-0ed7-4626-92be-bacfe8ce5f2b.snap.1648934623.3489687 -' into... {{(pid=129697) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Apr 02 21:23:51.897014 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_5qajrzgv --pool volumes - volumes/volume-3d159ee6-7514-4ca1-803d-49d79bc9076a' {{(pid=129697) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Apr 02 21:23:55.742643 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] Restore transfer completed in 3.8457s {{(pid=129697) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1111}} Apr 02 21:23:55.747399 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=129697) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1285}} Apr 02 21:23:55.776590 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] Restoring volume base metadata {{(pid=129697) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} Apr 02 21:23:55.798976 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] No metadata of type 'volume-metadata' to restore {{(pid=129697) put /opt/stack/cinder/cinder/backup/driver.py:345}} Apr 02 21:23:55.799382 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.driver [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] Restoring volume glance metadata {{(pid=129697) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}} Apr 02 21:23:55.926912 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] Restore to volume 3d159ee6-7514-4ca1-803d-49d79bc9076a finished successfully. {{(pid=129697) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1292}} Apr 02 21:23:55.928535 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-3d159ee6-7514-4ca1-803d-49d79bc9076a', 'hosts': ['10.222.0.18'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '3d159ee6-7514-4ca1-803d-49d79bc9076a', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 02 21:23:55.938305 ncdv-master-1 cinder-backup[129697]: DEBUG os_brick.initiator.connectors.rbd [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] <== disconnect_volume: return (9ms) None {{(pid=129697) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 02 21:23:55.982600 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.manager [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] Encryption key ID for volume 3d159ee6-7514-4ca1-803d-49d79bc9076a already matches encryption key ID in backup dc963741-0ed7-4626-92be-bacfe8ce5f2b. {{(pid=129697) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} Apr 02 21:23:56.011753 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-4b060606-044e-4b38-9fe5-b5eed4bd7097 tempest-TestVolumeBackupRestore-163528694 None] Finished restoring backup dc963741-0ed7-4626-92be-bacfe8ce5f2b to volume 3d159ee6-7514-4ca1-803d-49d79bc9076a. Apr 02 21:24:36.291240 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:24:36.291930 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:25:36.296192 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:25:36.297372 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:26:36.301434 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:26:36.302102 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:27:36.306762 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:27:36.307408 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:28:36.311877 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:28:36.312591 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:29:36.317439 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:29:36.318139 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:29:59.556442 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-4df24e0a-984e-4a28-9ab6-eb88057a9d30 tempest-TestVolumeBackupRestore-163528694 None] Delete backup started, backup: dc963741-0ed7-4626-92be-bacfe8ce5f2b. Apr 02 21:29:59.558374 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-4df24e0a-984e-4a28-9ab6-eb88057a9d30 tempest-TestVolumeBackupRestore-163528694 None] Delete started for backup=dc963741-0ed7-4626-92be-bacfe8ce5f2b {{(pid=129697) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Apr 02 21:29:59.558796 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-4df24e0a-984e-4a28-9ab6-eb88057a9d30 tempest-TestVolumeBackupRestore-163528694 None] Trying diff format basename='volume-24873dd9-832b-44d1-bbec-069663ce55ea.backup.dc963741-0ed7-4626-92be-bacfe8ce5f2b' for backup base image of volume 24873dd9-832b-44d1-bbec-069663ce55ea. {{(pid=129697) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Apr 02 21:29:59.626105 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-4df24e0a-984e-4a28-9ab6-eb88057a9d30 tempest-TestVolumeBackupRestore-163528694 None] Looking for snapshot of backup base 'volume-24873dd9-832b-44d1-bbec-069663ce55ea.backup.dc963741-0ed7-4626-92be-bacfe8ce5f2b' {{(pid=129697) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Apr 02 21:29:59.626562 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-4df24e0a-984e-4a28-9ab6-eb88057a9d30 tempest-TestVolumeBackupRestore-163528694 None] Found snapshot 'backup.dc963741-0ed7-4626-92be-bacfe8ce5f2b.snap.1648934623.3489687' {{(pid=129697) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Apr 02 21:29:59.627015 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-4df24e0a-984e-4a28-9ab6-eb88057a9d30 tempest-TestVolumeBackupRestore-163528694 None] Deleting backup snapshot='backup.dc963741-0ed7-4626-92be-bacfe8ce5f2b.snap.1648934623.3489687' {{(pid=129697) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Apr 02 21:30:00.650676 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.drivers.ceph [None req-4df24e0a-984e-4a28-9ab6-eb88057a9d30 tempest-TestVolumeBackupRestore-163528694 None] Deleting backup base image='volume-24873dd9-832b-44d1-bbec-069663ce55ea.backup.dc963741-0ed7-4626-92be-bacfe8ce5f2b' of volume 24873dd9-832b-44d1-bbec-069663ce55ea. Apr 02 21:30:01.136265 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-4df24e0a-984e-4a28-9ab6-eb88057a9d30 tempest-TestVolumeBackupRestore-163528694 None] Base backup image='volume-24873dd9-832b-44d1-bbec-069663ce55ea.backup.dc963741-0ed7-4626-92be-bacfe8ce5f2b' of volume 24873dd9-832b-44d1-bbec-069663ce55ea deleted. {{(pid=129697) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Apr 02 21:30:01.172299 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.backup.drivers.ceph [None req-4df24e0a-984e-4a28-9ab6-eb88057a9d30 tempest-TestVolumeBackupRestore-163528694 None] Delete of backup 'dc963741-0ed7-4626-92be-bacfe8ce5f2b' for volume '24873dd9-832b-44d1-bbec-069663ce55ea' finished. {{(pid=129697) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Apr 02 21:30:01.177918 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_db.sqlalchemy.engines [None req-4df24e0a-984e-4a28-9ab6-eb88057a9d30 tempest-TestVolumeBackupRestore-163528694 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129697) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 02 21:30:01.200361 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.quota [None req-4df24e0a-984e-4a28-9ab6-eb88057a9d30 tempest-TestVolumeBackupRestore-163528694 None] Created reservations ['caddd4ae-779f-43a2-ab46-6aabae279f27', 'd17ec68b-a42e-4fa9-a17c-5ea24670ca65'] {{(pid=129697) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 02 21:30:01.223753 ncdv-master-1 cinder-backup[129697]: INFO cinder.backup.manager [None req-4df24e0a-984e-4a28-9ab6-eb88057a9d30 tempest-TestVolumeBackupRestore-163528694 None] Delete backup finished, backup dc963741-0ed7-4626-92be-bacfe8ce5f2b deleted. Apr 02 21:30:36.323670 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:30:36.324356 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:31:36.320580 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:31:36.321335 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:32:36.317530 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:32:36.318284 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:33:36.316234 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:33:36.317022 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:34:36.322355 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:34:36.323087 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:35:36.327354 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:35:36.328200 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:36:36.332227 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:36:36.332905 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:37:36.337110 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:37:36.337845 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:38:36.342124 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:38:36.342863 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:39:36.347332 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:39:36.348122 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:40:36.352193 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:40:36.352919 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:41:36.357262 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:41:36.357968 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:42:36.362457 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:42:36.363118 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:44:36.368335 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:44:36.368997 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:45:36.365171 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:45:36.365770 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:46:36.370376 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:46:36.371096 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:47:36.375850 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:47:36.377495 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:48:36.374131 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:48:36.374799 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:49:36.381797 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:49:36.382858 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:50:36.378106 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:50:36.378713 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:51:36.374201 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:51:36.374846 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:52:36.379394 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:52:36.380039 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:53:36.384245 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:53:36.384962 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:54:36.389320 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:54:36.389986 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:55:36.395279 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:55:36.395982 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:56:36.400341 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:56:36.401057 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:57:36.405247 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:57:36.405822 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:58:36.401110 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:58:36.401722 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 21:59:36.406241 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 21:59:36.406883 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 22:00:36.411179 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 22:00:36.411718 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 22:01:36.416276 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 22:01:36.416956 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 22:02:36.421841 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 22:02:36.422505 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 22:03:36.427341 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 22:03:36.428567 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 22:04:36.425099 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 22:04:36.425794 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 22:05:36.430280 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 22:05:36.430863 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 22:06:36.435245 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 22:06:36.435834 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 22:07:36.440887 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 22:07:36.441538 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 22:08:36.436596 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 22:08:36.437238 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 22:09:36.441191 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 22:09:36.441804 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 22:10:36.446432 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 22:10:36.447036 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 22:11:36.443145 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 22:11:36.443770 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 22:12:36.448315 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 22:12:36.448924 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 22:13:36.444109 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 22:13:36.444696 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 22:14:36.449346 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 22:14:36.449922 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 22:15:36.445193 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 22:15:36.445879 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 22:16:36.450334 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 22:16:36.450990 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 22:17:36.455709 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 22:17:36.456195 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 22:18:36.460165 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 22:18:36.460685 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 02 22:19:36.456054 ncdv-master-1 cinder-backup[129697]: DEBUG oslo_service.periodic_task [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129697) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 02 22:19:36.456572 ncdv-master-1 cinder-backup[129697]: DEBUG cinder.manager [None req-18af85e2-befc-4131-b837-f91b56ebff6b None None] Notifying Schedulers of capabilities ... {{(pid=129697) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}