-- Logs begin at Sat 2022-04-16 20:01:17 UTC, end at Sat 2022-04-16 22:37:19 UTC. -- Apr 16 20:57:19.443930 ncdv-master-1 systemd[1]: Started Devstack devstack@c-bak.service. Apr 16 20:57:22.728445 ncdv-master-1 cinder-backup[130139]: INFO cinder.cmd.backup [-] Backup running with 1 processes. Apr 16 20:57:22.729369 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_concurrency.lockutils [-] Acquired lock "singleton_lock" {{(pid=130139) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:294}} Apr 16 20:57:22.729638 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_concurrency.lockutils [-] Releasing lock "singleton_lock" {{(pid=130139) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:312}} Apr 16 20:57:22.887941 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_db.sqlalchemy.engines [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=130139) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 16 20:57:23.175654 ncdv-master-1 cinder-backup[130139]: INFO cinder.rpc [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] Automatically selected cinder-backup objects version 1.39 as minimum service version. Apr 16 20:57:23.183226 ncdv-master-1 cinder-backup[130139]: INFO cinder.rpc [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] Automatically selected cinder-backup RPC version 2.3 as minimum service version. Apr 16 20:57:23.184501 ncdv-master-1 cinder-backup[130139]: INFO cinder.rpc [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] Automatically selected cinder-volume objects version 1.39 as minimum service version. Apr 16 20:57:23.188113 ncdv-master-1 cinder-backup[130139]: INFO cinder.rpc [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] Automatically selected cinder-volume RPC version 3.18 as minimum service version. Apr 16 20:57:23.192766 ncdv-master-1 cinder-backup[130139]: INFO cinder.rpc [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] Automatically selected cinder-scheduler objects version 1.39 as minimum service version. Apr 16 20:57:23.196717 ncdv-master-1 cinder-backup[130139]: INFO cinder.rpc [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] Automatically selected cinder-scheduler RPC version 3.12 as minimum service version. Apr 16 20:57:23.252691 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_db.sqlalchemy.engines [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=130139) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 16 20:57:23.403328 ncdv-master-1 cinder-backup[130139]: INFO oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] Starting 1 workers Apr 16 20:57:23.415620 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] Started child 130586 {{(pid=130139) _start_child /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:575}} Apr 16 20:57:23.418743 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] Full set of CONF: {{(pid=130139) wait /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:649}} Apr 16 20:57:23.418743 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] ******************************************************************************** {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Apr 16 20:57:23.418743 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] Configuration options gathered from: {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Apr 16 20:57:23.421313 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] command line args: ['--config-file', '/etc/cinder/cinder.conf'] {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Apr 16 20:57:23.422414 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] config files: ['/etc/cinder/cinder.conf'] {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Apr 16 20:57:23.423066 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] ================================================================================ {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Apr 16 20:57:23.423811 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] allow_availability_zone_fallback = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.435322 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] allow_compression_on_image_upload = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.436134 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] allowed_direct_url_schemes = [] {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.437551 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] api_rate_limit = True {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.437661 ncdv-master-1 cinder-backup[130586]: INFO cinder.service [-] Starting cinder-backup node (version 20.1.0) Apr 16 20:57:23.438716 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] auth_strategy = keystone {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.438716 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_availability_zone = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.438716 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backup_api_class = cinder.backup.api.API {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.439831 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backup_ceph_chunk_size = 134217728 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.439831 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backup_ceph_conf = /etc/ceph/ceph.conf {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.440335 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backup_ceph_image_journals = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.440582 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backup_ceph_pool = backups {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.440895 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backup_ceph_stripe_count = 0 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.442825 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backup_ceph_stripe_unit = 0 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.442825 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backup_ceph_user = cinder-bak {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.442825 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backup_driver = cinder.backup.drivers.ceph.CephBackupDriver {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.443305 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backup_driver_init_check_interval = 60 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.443787 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backup_driver_stats_polling_interval = 60 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.444290 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backup_manager = cinder.backup.manager.BackupManager {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.444753 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backup_max_operations = 15 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.448273 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backup_metadata_version = 2 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.450033 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backup_name_template = backup-%s {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.450875 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backup_native_threads_pool_size = 60 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.451699 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backup_object_number_per_notification = 10 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.452417 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backup_service_inithost_offload = True {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.453362 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backup_timer_interval = 120 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.454508 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backup_use_same_host = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.455484 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backup_use_temp_snapshot = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.456266 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backup_workers = 1 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.457070 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] chap_password = **** {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.495732 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] chap_username = {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.495732 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.495732 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cinder_internal_tenant_project_id = 4cb034afa3444ac9bfb020f072f6e801 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.495732 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cinder_internal_tenant_user_id = 6245e1a1bf6e41fb8660d5f4a3439068 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.495732 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] compression_format = gzip {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.495732 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] config_dir = [] {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.496836 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] config_file = ['/etc/cinder/cinder.conf'] {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.496908 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.lockutils [None req-4e597849-00b1-4463-9395-58785da2511f None None] Lock "cleanup_incomplete_backups_130139" acquired by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: waited 0.001s {{(pid=130586) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:386}} Apr 16 20:57:23.496908 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-4e597849-00b1-4463-9395-58785da2511f None None] Cleaning up incomplete backup operations. Apr 16 20:57:23.497046 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] config_source = [] {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.497046 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] consistencygroup_api_class = cinder.consistencygroup.api.API {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.497046 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] control_exchange = openstack {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.497046 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] debug = True {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.497046 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] default_availability_zone = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.497046 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] default_group_type = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.497440 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'swiftclient=WARN', 'botocore=WARN'] {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.497440 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] default_volume_type = ceph {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.497440 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] driver_client_cert = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.497440 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] driver_client_cert_key = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.497440 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] driver_data_namespace = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.497781 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] driver_ssl_cert_path = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.497781 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] driver_ssl_cert_verify = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.497781 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] driver_use_ssl = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.497781 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] enable_new_services = True {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.497781 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] enable_unsupported_driver = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.497781 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] enabled_backends = ['ceph'] {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.497781 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] enforce_multipath_for_image_xfer = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.498186 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] filter_function = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.498186 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] glance_api_insecure = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.498186 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] glance_api_servers = ['http://10.222.0.20/image'] {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.498186 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] glance_api_ssl_compression = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.498186 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] glance_ca_certificates_file = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.498186 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] glance_catalog_info = image:glance:publicURL {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.498591 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] glance_certfile = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.498591 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] glance_core_properties = ['checksum', 'container_format', 'disk_format', 'image_name', 'image_id', 'min_disk', 'min_ram', 'name', 'size'] {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.504422 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] glance_keyfile = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.504422 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] glance_num_retries = 3 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.504422 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] glance_request_timeout = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.504422 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] goodness_function = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.504422 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] graceful_shutdown_timeout = 5 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.504422 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] group_api_class = cinder.group.api.API {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.504942 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] host = ncdv-master-1 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.505154 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] iet_conf = /etc/iet/ietd.conf {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.505551 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] image_compress_on_upload = True {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.505907 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] image_conversion_address_space_limit = 1 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.506276 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] image_conversion_cpu_limit = 60 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.506749 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] image_conversion_dir = /opt/stack/data/cinder/conversion {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.507100 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] image_upload_use_cinder_backend = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.507449 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] image_upload_use_internal_tenant = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.507801 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] image_volume_cache_enabled = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.508164 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] image_volume_cache_max_count = 0 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.508551 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] image_volume_cache_max_size_gb = 0 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.508912 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] instance_format = [instance: %(uuid)s] {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.509273 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] instance_uuid_format = [instance: %(uuid)s] {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.509650 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] iscsi_iotype = fileio {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.510003 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] iscsi_secondary_ip_addresses = [] {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.510354 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] iscsi_target_flags = {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.511156 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] iscsi_write_cache = on {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.511522 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] iser_helper = tgtadm {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.513028 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] iser_ip_address = 10.222.0.20 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.513435 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] iser_port = 3260 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.513798 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.514176 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] log_config_append = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.514532 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.514881 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] log_dir = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.515249 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] log_file = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.515538 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] log_options = True {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.515884 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] log_rotate_interval = 1 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.516772 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] log_rotate_interval_type = days {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.517168 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] log_rotation_type = none {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.517469 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.517758 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.518247 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.518602 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.518890 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] logging_user_identity_format = %(user)s %(project)s %(domain)s %(user_domain)s %(project_domain)s {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.519545 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] max_age = 0 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.519929 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] max_logfile_count = 30 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.520318 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] max_logfile_size_mb = 200 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.520708 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] max_over_subscription_ratio = 20.0 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.521075 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] message_reap_interval = 86400 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.521434 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] message_ttl = 2592000 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.521781 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] monkey_patch = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.522144 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] monkey_patch_modules = [] {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.522427 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] my_ip = 10.222.0.20 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.522799 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] no_snapshot_gb_quota = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.523159 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] num_iser_scan_tries = 3 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.523508 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] num_shell_tries = 3 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.523856 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] num_volume_device_scan_tries = 3 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.524455 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] nvmet_ns_id = 10 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.524824 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] nvmet_port_id = 1 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.525212 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] osapi_max_limit = 1000 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.525553 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] osapi_volume_ext_list = [] {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.533562 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] osapi_volume_extension = ['cinder.api.contrib.standard_extensions'] {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.533562 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] osapi_volume_listen = 0.0.0.0 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.533562 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] osapi_volume_listen_port = 8776 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.533562 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] osapi_volume_use_ssl = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.533562 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] osapi_volume_workers = 2 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.533562 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] per_volume_size_limit = -1 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.534031 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] periodic_fuzzy_delay = 60 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.534031 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] periodic_interval = 60 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.534031 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] project_id_regex = [0-9a-f\-]+ {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.534031 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] public_endpoint = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.534031 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] publish_errors = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.534031 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] quota_backup_gigabytes = 1000 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.534031 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] quota_backups = 10 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.534493 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] quota_consistencygroups = 10 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.534493 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] quota_driver = cinder.quota.DbQuotaDriver {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.534493 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] quota_gigabytes = 1000 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.534493 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] quota_groups = 10 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.534493 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] quota_snapshots = 10 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.534493 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] quota_volumes = 10 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.534493 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] rate_limit_burst = 0 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.535778 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] rate_limit_except_level = CRITICAL {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.535778 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] rate_limit_interval = 0 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.535778 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] replication_device = **** {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.535778 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] report_discard_supported = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.535778 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] report_interval = 10 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.537421 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] reservation_clean_interval = 86400 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.537421 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] reservation_expire = 86400 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.537421 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] reserved_percentage = 0 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.537421 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] resource_query_filters_file = /etc/cinder/resource_filters.json {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.537664 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] restore_discard_excess_bytes = True {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.537985 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] rootwrap_config = /etc/cinder/rootwrap.conf {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.538539 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] rpc_response_timeout = 120 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.539126 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] run_external_periodic_tasks = True {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.539697 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] scheduler_manager = cinder.scheduler.manager.SchedulerManager {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.540315 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] scst_target_driver = iscsi {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.540863 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] scst_target_iqn_name = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.541157 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] service_down_time = 60 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.541521 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] snapshot_name_template = snapshot-%s {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.541867 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] split_loggers = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.542155 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] state_path = /opt/stack/data/cinder {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.542527 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] storage_availability_zone = nova {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.543009 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] storage_protocol = iscsi {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.543368 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] syslog_log_facility = LOG_USER {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.543721 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] target_helper = lioadm {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.544188 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] target_ip_address = 10.222.0.20 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.544592 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] target_port = 3260 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.544946 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] target_prefix = iqn.2010-10.org.openstack: {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.545317 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] target_protocol = iscsi {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.545659 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] trace_flags = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.546008 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] transfer_api_class = cinder.transfer.api.API {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.546351 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] transport_url = **** {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.546886 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] until_refresh = 0 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.547394 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] use_chap_auth = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.547970 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] use_default_quota_class = True {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.548609 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] use_eventlog = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.549153 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] use_journal = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.549850 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] use_json = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.550671 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] use_multipath_for_image_xfer = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.551784 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] use_stderr = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.552972 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] use_syslog = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.553883 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] verify_glance_signatures = enabled {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.554788 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] volume_api_class = cinder.volume.api.API {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.555563 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] volume_backend_name = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.556339 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] volume_clear = zero {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.576433 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] volume_clear_ionice = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.576433 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] volume_clear_size = 0 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.576433 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.576433 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] volume_copy_bps_limit = 0 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.576433 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] volume_dd_blocksize = 1M {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.576433 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] volume_manager = cinder.volume.manager.VolumeManager {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.577062 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] volume_name_template = volume-%s {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.577062 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] volume_usage_audit_period = month {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.577062 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] volumes_dir = /opt/stack/data/cinder/volumes {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.577062 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] watch_log_file = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Apr 16 20:57:23.577062 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.577062 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.577435 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_policy.enforce_new_defaults = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.577435 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_policy.enforce_scope = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.577435 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_policy.policy_default_rule = default {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.577435 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_policy.policy_dirs = ['policy.d'] {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.577435 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_policy.policy_file = policy.yaml {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.577435 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.577801 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.577801 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.577801 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.577801 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.577801 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_concurrency.disable_process_locking = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.577801 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_concurrency.lock_path = /opt/stack/data/cinder {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.578153 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.admin_password = **** {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.578153 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.admin_tenant_name = admin {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.578153 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.admin_token = **** {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.578153 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.admin_user = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.578153 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.auth_admin_prefix = {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.578153 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.578153 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.auth_port = 35357 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.578584 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.auth_protocol = https {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.578584 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.auth_section = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.578584 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.auth_type = password {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.578584 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.auth_uri = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.578584 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.auth_version = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.578584 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.cache = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.579384 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.cafile = /opt/stack/data/ca-bundle.pem {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.579384 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.certfile = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.579384 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.delay_auth_decision = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.579384 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.enforce_token_bind = permissive {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.579384 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.http_connect_timeout = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.579384 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.http_request_max_retries = 3 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.579819 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.identity_uri = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.579819 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.include_service_catalog = True {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.579819 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.insecure = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.579819 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.interface = public {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.579819 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.keyfile = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.579819 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.580203 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.580203 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.580622 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.581376 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.582208 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.memcache_secret_key = **** {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.583122 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.memcache_security_strategy = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.584079 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.585011 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.memcached_servers = ['localhost:11211'] {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.585838 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.region_name = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.586521 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.service_token_roles = ['service'] {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.587083 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.service_token_roles_required = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.587651 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.service_type = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.588200 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.token_cache_time = 300 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.588840 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] keystone_authtoken.www_authenticate_uri = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.589524 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.backend = dogpile.cache.null {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.590097 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.backend_argument = **** {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.590619 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.config_prefix = cache.oslo {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.591206 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.dead_timeout = 60.0 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.591835 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.debug_cache_backend = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.592477 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.enable_retry_client = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.593001 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.enable_socket_keepalive = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.593509 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.enabled = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.594050 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.expiration_time = 600 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.594586 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.hashclient_retry_attempts = 2 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.595126 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.hashclient_retry_delay = 1.0 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.595656 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.memcache_dead_retry = 300 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.596296 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.memcache_pool_connection_get_timeout = 10 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.596826 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.memcache_pool_flush_on_reconnect = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.597335 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.memcache_pool_maxsize = 10 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.597853 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.memcache_pool_unused_timeout = 60 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.598419 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.memcache_servers = ['localhost:11211'] {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.598954 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.memcache_socket_timeout = 1.0 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.599676 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.proxies = [] {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.600238 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.retry_attempts = 2 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.600822 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.retry_delay = 0.0 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.601349 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.socket_keepalive_count = 1 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.601898 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.socket_keepalive_idle = 1 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.602419 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.socket_keepalive_interval = 1 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.602926 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.tls_allowed_ciphers = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.603456 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.tls_cafile = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.603973 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.tls_certfile = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.604537 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.tls_enabled = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.605055 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cache.tls_keyfile = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.605717 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] coordination.backend_url = etcd3+http://10.222.0.20:2379 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.606329 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] database.backend = sqlalchemy {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.606841 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] database.connection = **** {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.607424 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] database.connection_debug = 0 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.607968 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] database.connection_parameters = {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.608592 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] database.connection_recycle_time = 3600 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.609139 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] database.connection_trace = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.609655 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] database.db_inc_retry_interval = True {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.610174 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] database.db_max_retries = 20 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.610714 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] database.db_max_retry_interval = 10 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.615136 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] database.db_retry_interval = 1 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.615739 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] database.max_overflow = 50 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.616314 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] database.max_pool_size = 5 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.621733 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] database.max_retries = 10 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.621733 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] database.mysql_enable_ndb = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.621733 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] database.mysql_sql_mode = TRADITIONAL {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.621733 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] database.pool_timeout = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.621733 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] database.retry_interval = 10 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.621733 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] database.slave_connection = **** {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.621733 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] database.sqlite_synchronous = True {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.622202 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] database.use_db_reconnect = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.622202 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.622202 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.623193 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_metrics.metrics_process_name = {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.623766 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.624357 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.624952 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] profiler.connection_string = messaging:// {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.625446 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] profiler.enabled = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.626012 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] profiler.es_doc_type = notification {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.626554 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] profiler.es_scroll_size = 10000 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.627088 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] profiler.es_scroll_time = 2m {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.627656 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] profiler.filter_error_trace = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.628240 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] profiler.hmac_keys = SECRET_KEY {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.640094 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] profiler.sentinel_service_name = mymaster {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.640094 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] profiler.socket_timeout = 0.1 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.640094 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] profiler.trace_sqlalchemy = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.640094 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] privsep_osbrick.capabilities = [21] {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.640094 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] privsep_osbrick.group = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.640094 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] privsep_osbrick.helper_command = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.640094 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] privsep_osbrick.logger_name = oslo_privsep.daemon {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.641256 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] privsep_osbrick.thread_pool_size = 4 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.641256 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] privsep_osbrick.user = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.641256 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cinder_sys_admin.capabilities = [0, 1, 2, 3, 12, 21] {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.641256 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cinder_sys_admin.group = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.641256 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cinder_sys_admin.helper_command = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.641256 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cinder_sys_admin.logger_name = oslo_privsep.daemon {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.641256 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cinder_sys_admin.thread_pool_size = 4 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.641991 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] cinder_sys_admin.user = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.641991 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] service_user.auth_section = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.641991 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] service_user.auth_type = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.641991 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] service_user.cafile = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.641991 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] service_user.certfile = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.641991 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] service_user.collect_timing = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.641991 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] service_user.insecure = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.642617 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] service_user.keyfile = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.642617 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] service_user.send_service_user_token = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.642617 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] service_user.split_loggers = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.642617 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] service_user.timeout = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.642617 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.backend_availability_zone = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.642617 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.chap_password = **** {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.642617 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.chap_username = {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.643070 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.643070 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.deferred_deletion_delay = 0 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.643545 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.deferred_deletion_purge_interval = 60 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.644276 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.driver_client_cert = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.644832 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.driver_client_cert_key = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.645364 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.driver_data_namespace = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.645918 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.driver_ssl_cert_path = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.646475 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.driver_ssl_cert_verify = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.647056 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.driver_use_ssl = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.647863 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.enable_deferred_deletion = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.648488 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.enable_unsupported_driver = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.649774 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.enforce_multipath_for_image_xfer = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.650324 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.filter_function = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.650870 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.goodness_function = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.651428 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.iet_conf = /etc/iet/ietd.conf {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.651961 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.image_upload_use_cinder_backend = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.652552 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.image_upload_use_internal_tenant = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.653098 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.image_volume_cache_enabled = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.653648 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.image_volume_cache_max_count = 0 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.654191 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.image_volume_cache_max_size_gb = 0 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.654742 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.iscsi_iotype = fileio {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.655269 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.iscsi_secondary_ip_addresses = [] {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.655803 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.iscsi_target_flags = {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.656410 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.iscsi_write_cache = on {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.656977 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.iser_helper = tgtadm {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.657705 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.iser_ip_address = 10.222.0.20 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.658442 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.iser_port = 3260 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.658997 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.659549 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.max_over_subscription_ratio = 20.0 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.660085 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.num_iser_scan_tries = 3 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.660653 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.num_shell_tries = 3 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.662478 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.num_volume_device_scan_tries = 3 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.663029 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.nvmet_ns_id = 10 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.663568 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.nvmet_port_id = 1 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.664102 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.rados_connect_timeout = -1 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.664679 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.rados_connection_interval = 5 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.665218 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.rados_connection_retries = 3 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.665755 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.rbd_ceph_conf = {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.666304 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.rbd_cluster_name = ceph {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.666825 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.rbd_exclusive_cinder_pool = True {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.667359 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_db.sqlalchemy.engines [None req-4e597849-00b1-4463-9395-58785da2511f None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=130586) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 16 20:57:23.667536 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.rbd_flatten_volume_from_snapshot = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.668214 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.rbd_max_clone_depth = 5 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.668806 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.rbd_pool = rbd {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.669326 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.rbd_secret_uuid = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.672077 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.rbd_store_chunk_size = 4 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.672657 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.rbd_user = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.673205 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.replication_connect_timeout = 5 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.674087 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.replication_device = **** {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.675707 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.report_discard_supported = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.676252 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.report_dynamic_total_capacity = True {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.676941 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.reserved_percentage = 0 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.677478 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.scst_target_driver = iscsi {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.678024 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.scst_target_iqn_name = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.678629 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.storage_protocol = iscsi {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.679219 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.target_helper = tgtadm {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.679930 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.target_ip_address = 10.222.0.20 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.680541 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.target_port = 3260 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.681081 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.target_prefix = iqn.2010-10.org.openstack: {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.681636 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.target_protocol = iscsi {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.682180 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.trace_flags = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.682717 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.unique_fqdn_network = True {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.683257 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.use_chap_auth = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.684336 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.use_multipath_for_image_xfer = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.684954 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.volume_backend_name = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.685860 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.volume_clear = zero {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.686639 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.volume_clear_ionice = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.687457 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.volume_clear_size = 0 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.688314 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.689100 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.volume_copy_bps_limit = 0 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.690692 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.volume_dd_blocksize = 1M {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.691495 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] backend_defaults.volumes_dir = /opt/stack/data/cinder/volumes {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.692088 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_reports.file_event_handler = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.692709 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_reports.file_event_handler_interval = 1 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.693490 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_reports.log_dir = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.694142 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.694695 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.695470 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.696008 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.697130 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.697669 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.698328 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_rabbit.heartbeat_in_pthread = True {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.698881 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_rabbit.heartbeat_rate = 2 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.699993 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.700550 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.701121 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.701657 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.702274 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.702814 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.703371 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.703929 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.704512 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.705027 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.705653 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.706205 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.706760 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.707290 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.707864 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_rabbit.ssl = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.708437 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_rabbit.ssl_ca_file = {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.709001 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_rabbit.ssl_cert_file = {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.709557 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_rabbit.ssl_key_file = {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.710101 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_rabbit.ssl_version = {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.710695 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_notifications.driver = [] {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.711249 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_notifications.retry = -1 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.711835 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.712399 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] oslo_messaging_notifications.transport_url = **** {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.713106 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] key_manager.backend = cinder.keymgr.conf_key_mgr.ConfKeyManager {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.713727 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] barbican.auth_endpoint = http://localhost/identity/v3 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.720131 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] barbican.barbican_api_version = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.720131 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] barbican.barbican_endpoint = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.720131 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] barbican.barbican_endpoint_type = public {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.720131 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] barbican.barbican_region_name = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.720131 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] barbican.cafile = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.720131 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] barbican.certfile = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.720131 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] barbican.collect_timing = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.721140 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] barbican.insecure = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.721140 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] barbican.keyfile = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.721140 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] barbican.number_of_retries = 60 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.721140 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] barbican.retry_delay = 1 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.721140 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] barbican.send_service_user_token = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.721140 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] barbican.split_loggers = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.721140 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] barbican.timeout = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.721587 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] barbican.verify_ssl = True {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.721587 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] barbican.verify_ssl_path = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.721587 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] barbican_service_user.auth_section = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.722261 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] barbican_service_user.auth_type = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.722854 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] barbican_service_user.cafile = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.723149 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.lockutils [None req-4e597849-00b1-4463-9395-58785da2511f None None] Lock "cleanup_incomplete_backups_130139" "released" by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: held 0.253s {{(pid=130586) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:400}} Apr 16 20:57:23.723562 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] barbican_service_user.certfile = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.724150 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] barbican_service_user.collect_timing = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.724773 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] barbican_service_user.insecure = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.725331 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] barbican_service_user.keyfile = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.725871 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] barbican_service_user.split_loggers = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.726388 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] barbican_service_user.timeout = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.727132 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] vault.approle_role_id = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.727668 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] vault.approle_secret_id = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.728017 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] vault.cafile = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.728388 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] vault.certfile = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.737104 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] vault.collect_timing = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.737104 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] vault.insecure = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.737104 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] vault.keyfile = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.737104 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] vault.kv_mountpoint = secret {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.737104 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] vault.kv_version = 2 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.737104 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] vault.namespace = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.737104 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] vault.root_token_id = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.738227 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] vault.split_loggers = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.738227 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] vault.ssl_ca_crt_file = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.738227 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] vault.timeout = None {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.738227 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] vault.use_ssl = False {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.738227 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] vault.vault_url = http://127.0.0.1:8200 {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Apr 16 20:57:23.738227 ncdv-master-1 cinder-backup[130139]: DEBUG oslo_service.service [None req-8a368fd7-5606-4e7c-8049-dc63bbb570aa None None] ******************************************************************************** {{(pid=130139) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Apr 16 20:57:23.739608 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_db.sqlalchemy.engines [None req-4e597849-00b1-4463-9395-58785da2511f None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=130586) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 16 20:57:23.745041 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-4e597849-00b1-4463-9395-58785da2511f None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 20:57:23.767307 ncdv-master-1 cinder-backup[130586]: INFO cinder.keymgr.migration [None req-9f328603-54de-4a45-8bfe-d69a3e90ce5a None None] Not migrating encryption keys because the ConfKeyManager is still in use. Apr 16 20:57:23.780609 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.service [None req-4e597849-00b1-4463-9395-58785da2511f None None] Creating RPC server for service cinder-backup {{(pid=130586) start /opt/stack/cinder/cinder/service.py:231}} Apr 16 20:57:23.790115 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.service [None req-4e597849-00b1-4463-9395-58785da2511f None None] Pinning object versions for RPC server serializer to 1.39 {{(pid=130586) start /opt/stack/cinder/cinder/service.py:237}} Apr 16 20:59:18.821208 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 20:59:18.821817 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:00:18.816861 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:00:18.817435 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:01:18.822133 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:01:18.822814 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:02:18.826785 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:02:18.827345 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:03:18.831748 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:03:18.832359 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:04:18.836409 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:04:18.836975 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:05:18.842823 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:05:18.843393 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:06:18.848645 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:06:18.849335 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:07:18.859217 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:07:18.859217 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:08:18.858744 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:08:18.862848 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:09:18.863946 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:09:18.864731 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:10:18.869837 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:10:18.871589 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:11:18.873084 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:11:18.873840 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:12:18.873710 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:12:18.874992 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:13:18.874805 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:13:18.879129 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:14:18.878312 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:14:18.878984 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:15:18.882872 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:15:18.883600 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:16:18.887577 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:16:18.888464 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:17:18.893176 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:17:18.893775 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:18:18.894950 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:18:18.895590 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:19:18.902973 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:19:18.902973 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:20:18.896322 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:20:18.897771 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:21:18.895230 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:21:18.899538 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:22:18.891633 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:22:18.892305 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:23:18.898946 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:23:18.902561 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:24:18.894810 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:24:18.895889 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:25:18.891694 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:25:18.892290 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:26:18.895351 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:26:18.896395 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:27:18.012361 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] Create backup started, backup: f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b volume: 8a31c3b5-5649-44bf-8b54-3af9e8bcbc48. Apr 16 21:27:18.107277 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-04-16T21:27:17Z,data_timestamp=2022-04-16T21:27:17Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsAdminTest-Backup-1156029311',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='b2b6e01c6acc41eeb83106b2e9830fe4',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='d59fae3205f9483d850750bfe5a304b7',volume_id=8a31c3b5-5649-44bf-8b54-3af9e8bcbc48) Apr 16 21:27:18.169571 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.utils [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.20', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:27:18.171582 ncdv-master-1 cinder-backup[130586]: INFO oslo.privsep.daemon [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] Running privsep helper: ['sudo', 'cinder-rootwrap', '/etc/cinder/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/cinder/cinder.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmp5d2tm8ix/privsep.sock'] Apr 16 21:27:18.227176 ncdv-master-1 sudo[154438]: ubuntu : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf privsep-helper --config-file /etc/cinder/cinder.conf --privsep_context os_brick.privileged.default --privsep_sock_path /tmp/tmp5d2tm8ix/privsep.sock Apr 16 21:27:18.228151 ncdv-master-1 sudo[154438]: pam_unix(sudo:session): session opened for user root by (uid=0) Apr 16 21:27:18.891970 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:27:18.892749 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:27:22.210911 ncdv-master-1 sudo[154438]: pam_unix(sudo:session): session closed for user root Apr 16 21:27:22.217176 ncdv-master-1 cinder-backup[130586]: INFO oslo.privsep.daemon [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] Spawned new privsep daemon via rootwrap Apr 16 21:27:22.218158 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] Accepted privsep connection to /tmp/tmp5d2tm8ix/privsep.sock {{(pid=130586) __init__ /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:362}} Apr 16 21:27:22.219633 ncdv-master-1 cinder-backup[130586]: INFO oslo.privsep.daemon [-] privsep daemon starting Apr 16 21:27:22.219912 ncdv-master-1 cinder-backup[130586]: INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 Apr 16 21:27:22.221027 ncdv-master-1 cinder-backup[130586]: INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none Apr 16 21:27:22.221027 ncdv-master-1 cinder-backup[130586]: INFO oslo.privsep.daemon [-] privsep daemon running as pid 154475 Apr 16 21:27:22.231244 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[08444c88-6ff2-4be6-a4e7-fd687eb9ecdd]: (2,) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:27:22.755632 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:27:22.788278 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.032s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:27:22.788806 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[c34f9116-8000-41ba-b385-9af89b5eb694]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ff85ebdbdb9\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:27:22.790448 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:27:22.808987 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 16 21:27:22.829186 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[2f900394-300a-4023-8471-360ca3bbb1a9]: [Errno 2] No such file or directory: 'systool' {{(pid=154475) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 16 21:27:22.829186 ncdv-master-1 cinder-backup[130586]: Traceback (most recent call last): Apr 16 21:27:22.829186 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 16 21:27:22.829186 ncdv-master-1 cinder-backup[130586]: ret = func(*f_args, **f_kwargs) Apr 16 21:27:22.829186 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 16 21:27:22.829186 ncdv-master-1 cinder-backup[130586]: return func(*args, **kwargs) Apr 16 21:27:22.829186 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 16 21:27:22.829186 ncdv-master-1 cinder-backup[130586]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 16 21:27:22.829186 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 16 21:27:22.829186 ncdv-master-1 cinder-backup[130586]: return putils.execute(on_execute=on_execute, Apr 16 21:27:22.829186 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 16 21:27:22.829186 ncdv-master-1 cinder-backup[130586]: obj = subprocess.Popen(cmd, Apr 16 21:27:22.829186 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 16 21:27:22.829186 ncdv-master-1 cinder-backup[130586]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 16 21:27:22.829186 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 16 21:27:22.829186 ncdv-master-1 cinder-backup[130586]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 16 21:27:22.829186 ncdv-master-1 cinder-backup[130586]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 16 21:27:22.830537 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[2f900394-300a-4023-8471-360ca3bbb1a9]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:27:22.830973 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:27:22.839114 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 16 21:27:22.839449 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[71f5601f-a831-4b8e-87e4-4fb3aea3b6f5]: [Errno 2] No such file or directory: 'systool' {{(pid=154475) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 16 21:27:22.839449 ncdv-master-1 cinder-backup[130586]: Traceback (most recent call last): Apr 16 21:27:22.839449 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 16 21:27:22.839449 ncdv-master-1 cinder-backup[130586]: ret = func(*f_args, **f_kwargs) Apr 16 21:27:22.839449 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 16 21:27:22.839449 ncdv-master-1 cinder-backup[130586]: return func(*args, **kwargs) Apr 16 21:27:22.839449 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 16 21:27:22.839449 ncdv-master-1 cinder-backup[130586]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 16 21:27:22.839449 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 16 21:27:22.839449 ncdv-master-1 cinder-backup[130586]: return putils.execute(on_execute=on_execute, Apr 16 21:27:22.839449 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 16 21:27:22.839449 ncdv-master-1 cinder-backup[130586]: obj = subprocess.Popen(cmd, Apr 16 21:27:22.839449 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 16 21:27:22.839449 ncdv-master-1 cinder-backup[130586]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 16 21:27:22.839449 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 16 21:27:22.839449 ncdv-master-1 cinder-backup[130586]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 16 21:27:22.839449 ncdv-master-1 cinder-backup[130586]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 16 21:27:22.845664 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[71f5601f-a831-4b8e-87e4-4fb3aea3b6f5]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:27:22.845664 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:27:22.878488 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.036s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:27:22.878827 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[774302ee-54c3-4e8c-9525-49a734963b84]: (4, ('/dev/sda1\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:27:22.880106 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:27:22.952491 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.072s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:27:22.953355 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[8f378046-f291-40d3-868b-200f8dc5f627]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:27:22.958951 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:27:22.984457 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.025s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:27:22.984801 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[9ca48683-0f02-496f-b501-05f8c93ac662]: (4, ('efede189-2323-8540-b968-072c0c8c1315\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:27:22.986486 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] Running cmd (subprocess): nvme version {{(pid=130586) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:27:23.058054 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] CMD "nvme version" returned: 0 in 0.070s {{(pid=130586) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:27:23.084646 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=130586) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Apr 16 21:27:23.085822 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=130586) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Apr 16 21:27:23.086303 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:3a91007f-78fd-42da-ba0d-d5efe907ca61 dsc: {{(pid=130586) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Apr 16 21:27:23.088554 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.utils [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] <== get_connector_properties: return (4916ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.20', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ff85ebdbdb9', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'efede189-2323-8540-b968-072c0c8c1315', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:3a91007f-78fd-42da-ba0d-d5efe907ca61', 'found_dsc': ''} {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:27:23.974896 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_db.sqlalchemy.engines [None req-61e8a934-990b-4a99-9b63-11946b0b3861 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=130586) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 16 21:27:26.394108 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connector [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] Factory for rbd on None {{(pid=130586) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Apr 16 21:27:26.396346 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-8a31c3b5-5649-44bf-8b54-3af9e8bcbc48', 'hosts': ['10.222.0.20'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '8a31c3b5-5649-44bf-8b54-3af9e8bcbc48', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:27:26.397701 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.linuxrbd [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] opening connection to ceph cluster (timeout=-1). {{(pid=130586) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Apr 16 21:27:26.807066 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] <== connect_volume: return (406ms) {'path': } {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:27:26.947326 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] Starting backup of volume='8a31c3b5-5649-44bf-8b54-3af9e8bcbc48'. {{(pid=130586) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} Apr 16 21:27:26.948367 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] Volume file is RBD: attempting optimized backup {{(pid=130586) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}} Apr 16 21:27:26.978511 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] Creating base image 'volume-8a31c3b5-5649-44bf-8b54-3af9e8bcbc48.backup.f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b' {{(pid=130586) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}} Apr 16 21:27:26.979076 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=130586) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} Apr 16 21:27:31.255850 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] Using --from-snap 'None' for incremental backup of volume 8a31c3b5-5649-44bf-8b54-3af9e8bcbc48. {{(pid=130586) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}} Apr 16 21:27:31.256541 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] Creating backup snapshot='backup.f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b.snap.1650144451.2558749' {{(pid=130586) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}} Apr 16 21:27:33.825758 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] Performing differential transfer from 'volume-8a31c3b5-5649-44bf-8b54-3af9e8bcbc48' to 'volume-8a31c3b5-5649-44bf-8b54-3af9e8bcbc48.backup.f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b' {{(pid=130586) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} Apr 16 21:27:33.826380 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_ija9znmx --pool volumes volumes/volume-8a31c3b5-5649-44bf-8b54-3af9e8bcbc48@backup.f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b.snap.1650144451.2558749 -' into... {{(pid=130586) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} Apr 16 21:27:33.826556 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-8a31c3b5-5649-44bf-8b54-3af9e8bcbc48.backup.f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b' {{(pid=130586) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} Apr 16 21:27:37.752827 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] Differential backup transfer completed in 3.9262s {{(pid=130586) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}} Apr 16 21:27:37.753469 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] Getting metadata type 'volume-base-metadata' {{(pid=130586) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Apr 16 21:27:37.764961 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=130586) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Apr 16 21:27:37.765378 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] Getting metadata type 'volume-metadata' {{(pid=130586) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Apr 16 21:27:37.776733 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] No metadata type 'volume-metadata' available {{(pid=130586) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Apr 16 21:27:37.777125 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] Getting metadata type 'volume-glance-metadata' {{(pid=130586) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Apr 16 21:27:37.786810 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] No metadata type 'volume-glance-metadata' available {{(pid=130586) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Apr 16 21:27:37.787357 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] Backing up metadata for volume 8a31c3b5-5649-44bf-8b54-3af9e8bcbc48. {{(pid=130586) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} Apr 16 21:27:38.078039 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] Backup 'f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b' of volume 8a31c3b5-5649-44bf-8b54-3af9e8bcbc48 finished. {{(pid=130586) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} Apr 16 21:27:38.080391 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-8a31c3b5-5649-44bf-8b54-3af9e8bcbc48', 'hosts': ['10.222.0.20'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '8a31c3b5-5649-44bf-8b54-3af9e8bcbc48', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:27:38.939466 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] <== disconnect_volume: return (857ms) None {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:27:39.298408 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-64df4430-da36-4739-b705-952b193b9734 tempest-VolumesBackupsAdminTest-1425647500 None] Create backup finished. backup: f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b. Apr 16 21:27:40.355155 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-295358b5-c782-41ae-92ec-5520cb2aaf85 tempest-VolumesBackupsAdminTest-1714170116 None] Export record started, backup: f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b. Apr 16 21:27:40.357431 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-295358b5-c782-41ae-92ec-5520cb2aaf85 tempest-VolumesBackupsAdminTest-1714170116 None] Export record finished, backup f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b exported. Apr 16 21:27:41.099092 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-3f10a497-f376-4318-ac24-c7d4a3e58464 tempest-VolumesBackupsAdminTest-1714170116 None] Import record started, backup_url: eyJkcml2ZXJfaW5mbyI6IHt9LCAiaWQiOiAiZWM2MzY4MWQtNTZmNC00NGY0LWFmY2YtZTYxNmUwYmI3MTQzIiwgInVzZXJfaWQiOiAiZDU5ZmFlMzIwNWY5NDgzZDg1MDc1MGJmZTVhMzA0YjciLCAicHJvamVjdF9pZCI6ICJiMmI2ZTAxYzZhY2M0MWVlYjgzMTA2YjJlOTgzMGZlNCIsICJ2b2x1bWVfaWQiOiAiOGEzMWMzYjUtNTY0OS00NGJmLThiNTQtM2FmOWU4YmNiYzQ4IiwgImhvc3QiOiAibmNkdi1tYXN0ZXItMSIsICJhdmFpbGFiaWxpdHlfem9uZSI6IG51bGwsICJjb250YWluZXIiOiAiYmFja3VwcyIsICJwYXJlbnRfaWQiOiBudWxsLCAic3RhdHVzIjogImF2YWlsYWJsZSIsICJmYWlsX3JlYXNvbiI6IG51bGwsICJzaXplIjogMSwgImRpc3BsYXlfbmFtZSI6ICJ0ZW1wZXN0LVZvbHVtZXNCYWNrdXBzQWRtaW5UZXN0LUJhY2t1cC0xMTU2MDI5MzExIiwgImRpc3BsYXlfZGVzY3JpcHRpb24iOiBudWxsLCAic2VydmljZV9tZXRhZGF0YSI6ICJ7XCJiYXNlXCI6IFwidm9sdW1lLThhMzFjM2I1LTU2NDktNDRiZi04YjU0LTNhZjllOGJjYmM0OC5iYWNrdXAuZjFiYTdkNTUtZmU1YS00YjgxLWJhODYtM2Q2ZTc1MDRmYjViXCJ9IiwgInNlcnZpY2UiOiAiY2luZGVyLmJhY2t1cC5kcml2ZXJzLmNlcGguQ2VwaEJhY2t1cERyaXZlciIsICJvYmplY3RfY291bnQiOiAwLCAidGVtcF92b2x1bWVfaWQiOiBudWxsLCAidGVtcF9zbmFwc2hvdF9pZCI6IG51bGwsICJudW1fZGVwZW5kZW50X2JhY2t1cHMiOiAwLCAic25hcHNob3RfaWQiOiBudWxsLCAiZGF0YV90aW1lc3RhbXAiOiAiMjAyMi0wNC0xNlQyMToyNzoxN1oiLCAicmVzdG9yZV92b2x1bWVfaWQiOiBudWxsLCAibWV0YWRhdGEiOiB7fSwgImVuY3J5cHRpb25fa2V5X2lkIjogbnVsbCwgImNyZWF0ZWRfYXQiOiAiMjAyMi0wNC0xNlQyMToyNzoxN1oiLCAidXBkYXRlZF9hdCI6ICIyMDIyLTA0LTE2VDIxOjI3OjM5WiIsICJkZWxldGVkX2F0IjogbnVsbCwgImRlbGV0ZWQiOiBmYWxzZX0=. Apr 16 21:27:41.106966 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_db.sqlalchemy.engines [None req-3f10a497-f376-4318-ac24-c7d4a3e58464 tempest-VolumesBackupsAdminTest-1714170116 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=130586) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 16 21:27:41.374326 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-3f10a497-f376-4318-ac24-c7d4a3e58464 tempest-VolumesBackupsAdminTest-1714170116 None] Import record id ec63681d-56f4-44f4-afcf-e616e0bb7143 metadata from driver finished. Apr 16 21:27:45.085566 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] Restore backup started, backup: f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b volume: bbbd6e6a-ae33-44ca-ba1e-ec6d7832b6c4. Apr 16 21:27:45.120333 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.utils [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.20', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:27:45.124563 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:27:45.134288 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.012s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:27:45.135295 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[830b568a-171c-4bb5-8fc3-043099ecc86f]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ff85ebdbdb9\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:27:45.142532 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:27:45.154010 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 16 21:27:45.154504 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[299b1a79-b8ba-42c3-b9e4-182facd827c5]: [Errno 2] No such file or directory: 'systool' {{(pid=154475) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 16 21:27:45.154504 ncdv-master-1 cinder-backup[130586]: Traceback (most recent call last): Apr 16 21:27:45.154504 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 16 21:27:45.154504 ncdv-master-1 cinder-backup[130586]: ret = func(*f_args, **f_kwargs) Apr 16 21:27:45.154504 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 16 21:27:45.154504 ncdv-master-1 cinder-backup[130586]: return func(*args, **kwargs) Apr 16 21:27:45.154504 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 16 21:27:45.154504 ncdv-master-1 cinder-backup[130586]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 16 21:27:45.154504 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 16 21:27:45.154504 ncdv-master-1 cinder-backup[130586]: return putils.execute(on_execute=on_execute, Apr 16 21:27:45.154504 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 16 21:27:45.154504 ncdv-master-1 cinder-backup[130586]: obj = subprocess.Popen(cmd, Apr 16 21:27:45.154504 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 16 21:27:45.154504 ncdv-master-1 cinder-backup[130586]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 16 21:27:45.154504 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 16 21:27:45.154504 ncdv-master-1 cinder-backup[130586]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 16 21:27:45.154504 ncdv-master-1 cinder-backup[130586]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 16 21:27:45.155961 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[299b1a79-b8ba-42c3-b9e4-182facd827c5]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:27:45.157244 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:27:45.166086 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 16 21:27:45.166629 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[7561d69a-5068-406f-8078-5891a49c6aa7]: [Errno 2] No such file or directory: 'systool' {{(pid=154475) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 16 21:27:45.166629 ncdv-master-1 cinder-backup[130586]: Traceback (most recent call last): Apr 16 21:27:45.166629 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 16 21:27:45.166629 ncdv-master-1 cinder-backup[130586]: ret = func(*f_args, **f_kwargs) Apr 16 21:27:45.166629 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 16 21:27:45.166629 ncdv-master-1 cinder-backup[130586]: return func(*args, **kwargs) Apr 16 21:27:45.166629 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 16 21:27:45.166629 ncdv-master-1 cinder-backup[130586]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 16 21:27:45.166629 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 16 21:27:45.166629 ncdv-master-1 cinder-backup[130586]: return putils.execute(on_execute=on_execute, Apr 16 21:27:45.166629 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 16 21:27:45.166629 ncdv-master-1 cinder-backup[130586]: obj = subprocess.Popen(cmd, Apr 16 21:27:45.166629 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 16 21:27:45.166629 ncdv-master-1 cinder-backup[130586]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 16 21:27:45.166629 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 16 21:27:45.166629 ncdv-master-1 cinder-backup[130586]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 16 21:27:45.166629 ncdv-master-1 cinder-backup[130586]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 16 21:27:45.168115 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[7561d69a-5068-406f-8078-5891a49c6aa7]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:27:45.169267 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:27:45.180600 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.011s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:27:45.180821 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[5230a449-2d9b-40b9-a5c6-dca68dca2c81]: (4, ('/dev/sda1\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:27:45.182189 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:27:45.193085 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.011s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:27:45.194075 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[1c7ada92-4db1-475b-b06d-5665233e0079]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:27:45.196058 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:27:45.207692 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.012s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:27:45.208190 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[16a078b8-5610-41f4-8158-49ae0305840c]: (4, ('efede189-2323-8540-b968-072c0c8c1315\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:27:45.209349 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] Running cmd (subprocess): nvme version {{(pid=130586) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:27:45.241118 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] CMD "nvme version" returned: 0 in 0.031s {{(pid=130586) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:27:45.242982 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=130586) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Apr 16 21:27:45.243710 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=130586) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Apr 16 21:27:45.244119 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:3a91007f-78fd-42da-ba0d-d5efe907ca61 dsc: {{(pid=130586) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Apr 16 21:27:45.245058 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.utils [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] <== get_connector_properties: return (123ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.20', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ff85ebdbdb9', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'efede189-2323-8540-b968-072c0c8c1315', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:3a91007f-78fd-42da-ba0d-d5efe907ca61', 'found_dsc': ''} {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:27:46.114444 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connector [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] Factory for rbd on None {{(pid=130586) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Apr 16 21:27:46.118096 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-bbbd6e6a-ae33-44ca-ba1e-ec6d7832b6c4', 'hosts': ['10.222.0.20'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'bbbd6e6a-ae33-44ca-ba1e-ec6d7832b6c4', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:27:46.118791 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.linuxrbd [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] opening connection to ceph cluster (timeout=-1). {{(pid=130586) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Apr 16 21:27:46.355622 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] <== connect_volume: return (235ms) {'path': } {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:27:46.360460 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_db.sqlalchemy.engines [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=130586) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 16 21:27:46.373291 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] Starting restore from Ceph backup=f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b to volume=volume-bbbd6e6a-ae33-44ca-ba1e-ec6d7832b6c4 {{(pid=130586) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1343}} Apr 16 21:27:46.547034 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] Looking for snapshot of backup base 'volume-8a31c3b5-5649-44bf-8b54-3af9e8bcbc48.backup.f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b' {{(pid=130586) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} Apr 16 21:27:46.549723 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] Found snapshot 'backup.f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b.snap.1650144451.2558749' {{(pid=130586) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} Apr 16 21:27:46.552510 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] Volume file is RBD. {{(pid=130586) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1245}} Apr 16 21:27:46.597559 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] Attempting differential restore. {{(pid=130586) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1298}} Apr 16 21:27:46.598201 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] Attempting incremental restore from base='volume-8a31c3b5-5649-44bf-8b54-3af9e8bcbc48.backup.f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b' snap='backup.f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b.snap.1650144451.2558749' {{(pid=130586) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1145}} Apr 16 21:27:46.598793 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] Performing differential transfer from 'volume-8a31c3b5-5649-44bf-8b54-3af9e8bcbc48.backup.f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b' to 'volume-bbbd6e6a-ae33-44ca-ba1e-ec6d7832b6c4' {{(pid=130586) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} Apr 16 21:27:46.599333 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-8a31c3b5-5649-44bf-8b54-3af9e8bcbc48.backup.f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b@backup.f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b.snap.1650144451.2558749 -' into... {{(pid=130586) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} Apr 16 21:27:46.599804 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_88tu9_ms --pool volumes - volumes/volume-bbbd6e6a-ae33-44ca-ba1e-ec6d7832b6c4' {{(pid=130586) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} Apr 16 21:27:49.915673 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] Restore transfer completed in 3.3164s {{(pid=130586) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1167}} Apr 16 21:27:49.917466 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=130586) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1354}} Apr 16 21:27:49.960056 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] Restoring volume base metadata {{(pid=130586) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} Apr 16 21:27:50.031383 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] No metadata of type 'volume-metadata' to restore {{(pid=130586) put /opt/stack/cinder/cinder/backup/driver.py:345}} Apr 16 21:27:50.031933 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] No metadata of type 'volume-glance-metadata' to restore {{(pid=130586) put /opt/stack/cinder/cinder/backup/driver.py:345}} Apr 16 21:27:50.035925 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] Restore to volume bbbd6e6a-ae33-44ca-ba1e-ec6d7832b6c4 finished successfully. {{(pid=130586) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1361}} Apr 16 21:27:50.037786 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-bbbd6e6a-ae33-44ca-ba1e-ec6d7832b6c4', 'hosts': ['10.222.0.20'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'bbbd6e6a-ae33-44ca-ba1e-ec6d7832b6c4', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:27:50.201013 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] <== disconnect_volume: return (162ms) None {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:27:50.295551 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.manager [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] Encryption key ID for volume bbbd6e6a-ae33-44ca-ba1e-ec6d7832b6c4 already matches encryption key ID in backup f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b. {{(pid=130586) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} Apr 16 21:27:50.602853 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-88c18561-10ed-4592-b66c-6c351b6c653b tempest-VolumesBackupsAdminTest-1425647500 None] Finished restoring backup f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b to volume bbbd6e6a-ae33-44ca-ba1e-ec6d7832b6c4. Apr 16 21:27:51.981636 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-4a8883b1-c502-4862-b73e-6af14dc99cb1 tempest-VolumesBackupsAdminTest-1714170116 None] Delete backup started, backup: ec63681d-56f4-44f4-afcf-e616e0bb7143. Apr 16 21:27:51.986110 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-4a8883b1-c502-4862-b73e-6af14dc99cb1 tempest-VolumesBackupsAdminTest-1714170116 None] Delete started for backup=ec63681d-56f4-44f4-afcf-e616e0bb7143 {{(pid=130586) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} Apr 16 21:27:51.988344 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-4a8883b1-c502-4862-b73e-6af14dc99cb1 tempest-VolumesBackupsAdminTest-1714170116 None] Trying diff format basename='volume-8a31c3b5-5649-44bf-8b54-3af9e8bcbc48.backup.f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b' for backup base image of volume 8a31c3b5-5649-44bf-8b54-3af9e8bcbc48. {{(pid=130586) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} Apr 16 21:27:52.259022 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-4a8883b1-c502-4862-b73e-6af14dc99cb1 tempest-VolumesBackupsAdminTest-1714170116 None] Looking for snapshot of backup base 'volume-8a31c3b5-5649-44bf-8b54-3af9e8bcbc48.backup.f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b' {{(pid=130586) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} Apr 16 21:27:52.259861 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-4a8883b1-c502-4862-b73e-6af14dc99cb1 tempest-VolumesBackupsAdminTest-1714170116 None] Backup 'ec63681d-56f4-44f4-afcf-e616e0bb7143' has no snapshot {{(pid=130586) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:962}} Apr 16 21:27:52.260399 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-4a8883b1-c502-4862-b73e-6af14dc99cb1 tempest-VolumesBackupsAdminTest-1714170116 None] No backup snapshot to delete {{(pid=130586) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:500}} Apr 16 21:27:52.504758 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.drivers.ceph [None req-4a8883b1-c502-4862-b73e-6af14dc99cb1 tempest-VolumesBackupsAdminTest-1714170116 None] Backup base image of volume 8a31c3b5-5649-44bf-8b54-3af9e8bcbc48 still has 1 snapshots so skipping base image delete. Apr 16 21:27:52.794987 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-4a8883b1-c502-4862-b73e-6af14dc99cb1 tempest-VolumesBackupsAdminTest-1714170116 None] Metadata backup object 'backup.ec63681d-56f4-44f4-afcf-e616e0bb7143.meta' not found - ignoring {{(pid=130586) remove_if_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:162}} Apr 16 21:27:52.800470 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-4a8883b1-c502-4862-b73e-6af14dc99cb1 tempest-VolumesBackupsAdminTest-1714170116 None] Delete of backup 'ec63681d-56f4-44f4-afcf-e616e0bb7143' for volume '8a31c3b5-5649-44bf-8b54-3af9e8bcbc48' finished. {{(pid=130586) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} Apr 16 21:27:52.812590 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_db.sqlalchemy.engines [None req-4a8883b1-c502-4862-b73e-6af14dc99cb1 tempest-VolumesBackupsAdminTest-1714170116 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=130586) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 16 21:27:52.850359 ncdv-master-1 cinder-backup[130586]: WARNING cinder.quota [None req-4a8883b1-c502-4862-b73e-6af14dc99cb1 tempest-VolumesBackupsAdminTest-1714170116 None] Deprecated: Default quota for resource: backups is set by the default quota flag: quota_backups, it is now deprecated. Please use the default quota class for default quota. Apr 16 21:27:52.851424 ncdv-master-1 cinder-backup[130586]: WARNING cinder.quota [None req-4a8883b1-c502-4862-b73e-6af14dc99cb1 tempest-VolumesBackupsAdminTest-1714170116 None] Deprecated: Default quota for resource: backup_gigabytes is set by the default quota flag: quota_backup_gigabytes, it is now deprecated. Please use the default quota class for default quota. Apr 16 21:27:52.968774 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.quota [None req-4a8883b1-c502-4862-b73e-6af14dc99cb1 tempest-VolumesBackupsAdminTest-1714170116 None] Created reservations ['00499a4a-b8e6-47ec-bb8e-15083a86aa7b', 'f0862427-857e-4e86-90f6-5e7310e22718'] {{(pid=130586) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 16 21:27:53.161041 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-4a8883b1-c502-4862-b73e-6af14dc99cb1 tempest-VolumesBackupsAdminTest-1714170116 None] Delete backup finished, backup ec63681d-56f4-44f4-afcf-e616e0bb7143 deleted. Apr 16 21:27:54.229168 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-49fddf09-72ec-4e86-829c-323c20b6f838 tempest-VolumesBackupsAdminTest-1425647500 None] Delete backup started, backup: f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b. Apr 16 21:27:54.234587 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-49fddf09-72ec-4e86-829c-323c20b6f838 tempest-VolumesBackupsAdminTest-1425647500 None] Delete started for backup=f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b {{(pid=130586) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} Apr 16 21:27:54.234587 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-49fddf09-72ec-4e86-829c-323c20b6f838 tempest-VolumesBackupsAdminTest-1425647500 None] Trying diff format basename='volume-8a31c3b5-5649-44bf-8b54-3af9e8bcbc48.backup.f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b' for backup base image of volume 8a31c3b5-5649-44bf-8b54-3af9e8bcbc48. {{(pid=130586) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} Apr 16 21:27:54.466543 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-49fddf09-72ec-4e86-829c-323c20b6f838 tempest-VolumesBackupsAdminTest-1425647500 None] Looking for snapshot of backup base 'volume-8a31c3b5-5649-44bf-8b54-3af9e8bcbc48.backup.f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b' {{(pid=130586) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} Apr 16 21:27:54.466929 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-49fddf09-72ec-4e86-829c-323c20b6f838 tempest-VolumesBackupsAdminTest-1425647500 None] Found snapshot 'backup.f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b.snap.1650144451.2558749' {{(pid=130586) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} Apr 16 21:27:54.467303 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-49fddf09-72ec-4e86-829c-323c20b6f838 tempest-VolumesBackupsAdminTest-1425647500 None] Deleting backup snapshot='backup.f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b.snap.1650144451.2558749' {{(pid=130586) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}} Apr 16 21:27:57.502840 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.drivers.ceph [None req-49fddf09-72ec-4e86-829c-323c20b6f838 tempest-VolumesBackupsAdminTest-1425647500 None] Deleting backup base image='volume-8a31c3b5-5649-44bf-8b54-3af9e8bcbc48.backup.f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b' of volume 8a31c3b5-5649-44bf-8b54-3af9e8bcbc48. Apr 16 21:28:03.969648 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_db.sqlalchemy.engines [None req-61e8a934-990b-4a99-9b63-11946b0b3861 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=130586) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 16 21:28:05.824452 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-49fddf09-72ec-4e86-829c-323c20b6f838 tempest-VolumesBackupsAdminTest-1425647500 None] Base backup image='volume-8a31c3b5-5649-44bf-8b54-3af9e8bcbc48.backup.f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b' of volume 8a31c3b5-5649-44bf-8b54-3af9e8bcbc48 deleted. {{(pid=130586) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} Apr 16 21:28:06.065514 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-49fddf09-72ec-4e86-829c-323c20b6f838 tempest-VolumesBackupsAdminTest-1425647500 None] Delete of backup 'f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b' for volume '8a31c3b5-5649-44bf-8b54-3af9e8bcbc48' finished. {{(pid=130586) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} Apr 16 21:28:06.180144 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.quota [None req-49fddf09-72ec-4e86-829c-323c20b6f838 tempest-VolumesBackupsAdminTest-1425647500 None] Created reservations ['27eb8a9b-1ba7-4396-baba-fa510c7d55f7', 'ab3e6e4c-0194-4681-8763-fe7342e07741'] {{(pid=130586) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 16 21:28:06.340072 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-49fddf09-72ec-4e86-829c-323c20b6f838 tempest-VolumesBackupsAdminTest-1425647500 None] Delete backup finished, backup f1ba7d55-fe5a-4b81-ba86-3d6e7504fb5b deleted. Apr 16 21:28:10.597539 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] Create backup started, backup: 424ec57b-a5e2-458a-9a4e-5906dac3b04b volume: 3970c74e-3013-4b29-b21d-fdea46bf38d1. Apr 16 21:28:10.729254 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-04-16T21:28:09Z,data_timestamp=2022-04-16T21:28:09Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsAdminTest-Backup-31396608',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=424ec57b-a5e2-458a-9a4e-5906dac3b04b,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='b2b6e01c6acc41eeb83106b2e9830fe4',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='d59fae3205f9483d850750bfe5a304b7',volume_id=3970c74e-3013-4b29-b21d-fdea46bf38d1) Apr 16 21:28:10.797799 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.utils [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.20', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:28:10.800894 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:28:10.809955 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.009s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:28:10.810553 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[a94fe4ab-339c-4ae7-81ae-7cd18814e16d]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ff85ebdbdb9\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:28:10.811909 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:28:10.819805 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 16 21:28:10.820598 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[5988dcbe-d53d-41c8-9af1-a3e823691658]: [Errno 2] No such file or directory: 'systool' {{(pid=154475) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 16 21:28:10.820598 ncdv-master-1 cinder-backup[130586]: Traceback (most recent call last): Apr 16 21:28:10.820598 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 16 21:28:10.820598 ncdv-master-1 cinder-backup[130586]: ret = func(*f_args, **f_kwargs) Apr 16 21:28:10.820598 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 16 21:28:10.820598 ncdv-master-1 cinder-backup[130586]: return func(*args, **kwargs) Apr 16 21:28:10.820598 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 16 21:28:10.820598 ncdv-master-1 cinder-backup[130586]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 16 21:28:10.820598 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 16 21:28:10.820598 ncdv-master-1 cinder-backup[130586]: return putils.execute(on_execute=on_execute, Apr 16 21:28:10.820598 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 16 21:28:10.820598 ncdv-master-1 cinder-backup[130586]: obj = subprocess.Popen(cmd, Apr 16 21:28:10.820598 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 16 21:28:10.820598 ncdv-master-1 cinder-backup[130586]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 16 21:28:10.820598 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 16 21:28:10.820598 ncdv-master-1 cinder-backup[130586]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 16 21:28:10.820598 ncdv-master-1 cinder-backup[130586]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 16 21:28:10.821733 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[5988dcbe-d53d-41c8-9af1-a3e823691658]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:28:10.823196 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:28:10.829699 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 16 21:28:10.835586 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[1b1e4efb-c73c-4dd4-8505-4efb7e53fbb6]: [Errno 2] No such file or directory: 'systool' {{(pid=154475) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 16 21:28:10.835586 ncdv-master-1 cinder-backup[130586]: Traceback (most recent call last): Apr 16 21:28:10.835586 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 16 21:28:10.835586 ncdv-master-1 cinder-backup[130586]: ret = func(*f_args, **f_kwargs) Apr 16 21:28:10.835586 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 16 21:28:10.835586 ncdv-master-1 cinder-backup[130586]: return func(*args, **kwargs) Apr 16 21:28:10.835586 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 16 21:28:10.835586 ncdv-master-1 cinder-backup[130586]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 16 21:28:10.835586 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 16 21:28:10.835586 ncdv-master-1 cinder-backup[130586]: return putils.execute(on_execute=on_execute, Apr 16 21:28:10.835586 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 16 21:28:10.835586 ncdv-master-1 cinder-backup[130586]: obj = subprocess.Popen(cmd, Apr 16 21:28:10.835586 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 16 21:28:10.835586 ncdv-master-1 cinder-backup[130586]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 16 21:28:10.835586 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 16 21:28:10.835586 ncdv-master-1 cinder-backup[130586]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 16 21:28:10.835586 ncdv-master-1 cinder-backup[130586]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 16 21:28:10.835586 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[1b1e4efb-c73c-4dd4-8505-4efb7e53fbb6]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:28:10.840981 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:28:10.875284 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.038s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:28:10.876064 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[cecf6289-bed7-4240-9a38-f60eba0039fb]: (4, ('/dev/sda1\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:28:10.877364 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:28:10.889465 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.012s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:28:10.889702 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[e8aaf6b9-c0f9-41f9-b3d9-2e985aff4477]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:28:10.891215 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:28:10.900124 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.009s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:28:10.901528 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[0aaee0d2-da3e-403d-8600-58915d63f825]: (4, ('efede189-2323-8540-b968-072c0c8c1315\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:28:10.902414 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] Running cmd (subprocess): nvme version {{(pid=130586) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:28:10.928046 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] CMD "nvme version" returned: 0 in 0.025s {{(pid=130586) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:28:10.929897 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=130586) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Apr 16 21:28:10.930346 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=130586) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Apr 16 21:28:10.930870 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:3a91007f-78fd-42da-ba0d-d5efe907ca61 dsc: {{(pid=130586) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Apr 16 21:28:10.931756 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.utils [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] <== get_connector_properties: return (133ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.20', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ff85ebdbdb9', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'efede189-2323-8540-b968-072c0c8c1315', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:3a91007f-78fd-42da-ba0d-d5efe907ca61', 'found_dsc': ''} {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:28:11.921770 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connector [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] Factory for rbd on None {{(pid=130586) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Apr 16 21:28:11.925074 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-3970c74e-3013-4b29-b21d-fdea46bf38d1', 'hosts': ['10.222.0.20'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '3970c74e-3013-4b29-b21d-fdea46bf38d1', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:28:11.926470 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.linuxrbd [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] opening connection to ceph cluster (timeout=-1). {{(pid=130586) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Apr 16 21:28:12.157182 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] <== connect_volume: return (231ms) {'path': } {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:28:12.165337 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_db.sqlalchemy.engines [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=130586) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 16 21:28:12.242432 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] Starting backup of volume='3970c74e-3013-4b29-b21d-fdea46bf38d1'. {{(pid=130586) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} Apr 16 21:28:12.243702 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] Volume file is RBD: attempting optimized backup {{(pid=130586) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}} Apr 16 21:28:12.296160 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] Creating base image 'volume-3970c74e-3013-4b29-b21d-fdea46bf38d1.backup.424ec57b-a5e2-458a-9a4e-5906dac3b04b' {{(pid=130586) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}} Apr 16 21:28:12.297152 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=130586) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} Apr 16 21:28:13.426179 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] Using --from-snap 'None' for incremental backup of volume 3970c74e-3013-4b29-b21d-fdea46bf38d1. {{(pid=130586) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}} Apr 16 21:28:13.426742 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] Creating backup snapshot='backup.424ec57b-a5e2-458a-9a4e-5906dac3b04b.snap.1650144493.426209' {{(pid=130586) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}} Apr 16 21:28:16.444621 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] Performing differential transfer from 'volume-3970c74e-3013-4b29-b21d-fdea46bf38d1' to 'volume-3970c74e-3013-4b29-b21d-fdea46bf38d1.backup.424ec57b-a5e2-458a-9a4e-5906dac3b04b' {{(pid=130586) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} Apr 16 21:28:16.445835 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd__px56_b3 --pool volumes volumes/volume-3970c74e-3013-4b29-b21d-fdea46bf38d1@backup.424ec57b-a5e2-458a-9a4e-5906dac3b04b.snap.1650144493.426209 -' into... {{(pid=130586) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} Apr 16 21:28:16.446394 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-3970c74e-3013-4b29-b21d-fdea46bf38d1.backup.424ec57b-a5e2-458a-9a4e-5906dac3b04b' {{(pid=130586) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} Apr 16 21:28:18.892989 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:28:18.894297 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:28:20.361939 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] Differential backup transfer completed in 3.9173s {{(pid=130586) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}} Apr 16 21:28:20.363867 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] Getting metadata type 'volume-base-metadata' {{(pid=130586) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Apr 16 21:28:20.383084 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=130586) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Apr 16 21:28:20.384235 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] Getting metadata type 'volume-metadata' {{(pid=130586) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Apr 16 21:28:20.392869 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] No metadata type 'volume-metadata' available {{(pid=130586) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Apr 16 21:28:20.393503 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] Getting metadata type 'volume-glance-metadata' {{(pid=130586) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Apr 16 21:28:20.401275 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] No metadata type 'volume-glance-metadata' available {{(pid=130586) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Apr 16 21:28:20.401923 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] Backing up metadata for volume 3970c74e-3013-4b29-b21d-fdea46bf38d1. {{(pid=130586) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} Apr 16 21:28:20.857221 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] Backup '424ec57b-a5e2-458a-9a4e-5906dac3b04b' of volume 3970c74e-3013-4b29-b21d-fdea46bf38d1 finished. {{(pid=130586) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} Apr 16 21:28:20.859104 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-3970c74e-3013-4b29-b21d-fdea46bf38d1', 'hosts': ['10.222.0.20'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '3970c74e-3013-4b29-b21d-fdea46bf38d1', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:28:21.603491 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] <== disconnect_volume: return (743ms) None {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:28:21.989538 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-86b60595-4f7a-4b80-814f-2d9d6d09f316 tempest-VolumesBackupsAdminTest-1425647500 None] Create backup finished. backup: 424ec57b-a5e2-458a-9a4e-5906dac3b04b. Apr 16 21:28:23.323708 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-60be6426-b1cd-499f-b666-701b653dc98b tempest-VolumesBackupsAdminTest-1714170116 None] Reset backup status started, backup_id: 424ec57b-a5e2-458a-9a4e-5906dac3b04b, status: error. Apr 16 21:28:23.324320 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-60be6426-b1cd-499f-b666-701b653dc98b tempest-VolumesBackupsAdminTest-1714170116 None] Backup service: cinder.backup.drivers.ceph.CephBackupDriver. Apr 16 21:28:24.634294 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-3bd9c557-baac-4336-90bd-509d07d952bb tempest-VolumesBackupsAdminTest-1425647500 None] Delete backup started, backup: 424ec57b-a5e2-458a-9a4e-5906dac3b04b. Apr 16 21:28:24.637884 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-3bd9c557-baac-4336-90bd-509d07d952bb tempest-VolumesBackupsAdminTest-1425647500 None] Delete started for backup=424ec57b-a5e2-458a-9a4e-5906dac3b04b {{(pid=130586) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} Apr 16 21:28:24.638324 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-3bd9c557-baac-4336-90bd-509d07d952bb tempest-VolumesBackupsAdminTest-1425647500 None] Trying diff format basename='volume-3970c74e-3013-4b29-b21d-fdea46bf38d1.backup.424ec57b-a5e2-458a-9a4e-5906dac3b04b' for backup base image of volume 3970c74e-3013-4b29-b21d-fdea46bf38d1. {{(pid=130586) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} Apr 16 21:28:25.077121 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-3bd9c557-baac-4336-90bd-509d07d952bb tempest-VolumesBackupsAdminTest-1425647500 None] Looking for snapshot of backup base 'volume-3970c74e-3013-4b29-b21d-fdea46bf38d1.backup.424ec57b-a5e2-458a-9a4e-5906dac3b04b' {{(pid=130586) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} Apr 16 21:28:25.077514 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-3bd9c557-baac-4336-90bd-509d07d952bb tempest-VolumesBackupsAdminTest-1425647500 None] Found snapshot 'backup.424ec57b-a5e2-458a-9a4e-5906dac3b04b.snap.1650144493.426209' {{(pid=130586) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} Apr 16 21:28:25.077890 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-3bd9c557-baac-4336-90bd-509d07d952bb tempest-VolumesBackupsAdminTest-1425647500 None] Deleting backup snapshot='backup.424ec57b-a5e2-458a-9a4e-5906dac3b04b.snap.1650144493.426209' {{(pid=130586) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}} Apr 16 21:28:29.604958 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.drivers.ceph [None req-3bd9c557-baac-4336-90bd-509d07d952bb tempest-VolumesBackupsAdminTest-1425647500 None] Deleting backup base image='volume-3970c74e-3013-4b29-b21d-fdea46bf38d1.backup.424ec57b-a5e2-458a-9a4e-5906dac3b04b' of volume 3970c74e-3013-4b29-b21d-fdea46bf38d1. Apr 16 21:28:33.978942 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_db.sqlalchemy.engines [None req-61e8a934-990b-4a99-9b63-11946b0b3861 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=130586) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 16 21:28:36.644666 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-3bd9c557-baac-4336-90bd-509d07d952bb tempest-VolumesBackupsAdminTest-1425647500 None] Base backup image='volume-3970c74e-3013-4b29-b21d-fdea46bf38d1.backup.424ec57b-a5e2-458a-9a4e-5906dac3b04b' of volume 3970c74e-3013-4b29-b21d-fdea46bf38d1 deleted. {{(pid=130586) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} Apr 16 21:28:36.885061 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-3bd9c557-baac-4336-90bd-509d07d952bb tempest-VolumesBackupsAdminTest-1425647500 None] Delete of backup '424ec57b-a5e2-458a-9a4e-5906dac3b04b' for volume '3970c74e-3013-4b29-b21d-fdea46bf38d1' finished. {{(pid=130586) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} Apr 16 21:28:37.050192 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.quota [None req-3bd9c557-baac-4336-90bd-509d07d952bb tempest-VolumesBackupsAdminTest-1425647500 None] Created reservations ['03402f3c-7fd4-4fab-ba81-b31b18cf7c47', '6d5a0564-8e2b-476d-b113-e64a9db9a9b9'] {{(pid=130586) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 16 21:28:37.317180 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-3bd9c557-baac-4336-90bd-509d07d952bb tempest-VolumesBackupsAdminTest-1425647500 None] Delete backup finished, backup 424ec57b-a5e2-458a-9a4e-5906dac3b04b deleted. Apr 16 21:29:18.892532 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:29:18.893200 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:30:18.898302 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:30:18.900719 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:31:18.898995 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:31:18.899691 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:32:18.896904 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:32:18.897616 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:33:18.902904 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:33:18.904359 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:34:18.908753 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:34:18.909990 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:35:18.913407 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:35:18.915078 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:36:18.916380 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:36:18.920290 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:37:18.872782 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] Create backup started, backup: 8e05f40b-c063-449b-8140-ed75064781b3 volume: 0dc9c2b9-1dc0-4ef0-b0d6-2a09be69e861. Apr 16 21:37:18.915448 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:37:18.915766 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:37:18.960669 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-04-16T21:37:18Z,data_timestamp=2022-04-16T21:37:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsTest-Backup-1035261800',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=8e05f40b-c063-449b-8140-ed75064781b3,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='d51b576f9c0241c1a254f15308a455ec',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='77f61ccbf5114e6080e33f0c5f500468',volume_id=0dc9c2b9-1dc0-4ef0-b0d6-2a09be69e861) Apr 16 21:37:19.019435 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.utils [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.20', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:37:19.020860 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:37:19.031036 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.010s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:37:19.031326 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[a7582a1c-54d4-48ef-840e-5e56e9b1a10e]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ff85ebdbdb9\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:37:19.032646 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:37:19.040099 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 16 21:37:19.041382 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[9ef2af64-5d64-4e03-ba5b-baf97a5f2c2f]: [Errno 2] No such file or directory: 'systool' {{(pid=154475) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 16 21:37:19.041382 ncdv-master-1 cinder-backup[130586]: Traceback (most recent call last): Apr 16 21:37:19.041382 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 16 21:37:19.041382 ncdv-master-1 cinder-backup[130586]: ret = func(*f_args, **f_kwargs) Apr 16 21:37:19.041382 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 16 21:37:19.041382 ncdv-master-1 cinder-backup[130586]: return func(*args, **kwargs) Apr 16 21:37:19.041382 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 16 21:37:19.041382 ncdv-master-1 cinder-backup[130586]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 16 21:37:19.041382 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 16 21:37:19.041382 ncdv-master-1 cinder-backup[130586]: return putils.execute(on_execute=on_execute, Apr 16 21:37:19.041382 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 16 21:37:19.041382 ncdv-master-1 cinder-backup[130586]: obj = subprocess.Popen(cmd, Apr 16 21:37:19.041382 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 16 21:37:19.041382 ncdv-master-1 cinder-backup[130586]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 16 21:37:19.041382 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 16 21:37:19.041382 ncdv-master-1 cinder-backup[130586]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 16 21:37:19.041382 ncdv-master-1 cinder-backup[130586]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 16 21:37:19.042433 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[9ef2af64-5d64-4e03-ba5b-baf97a5f2c2f]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:37:19.043238 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:37:19.049915 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 16 21:37:19.051358 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[bb530fbe-6d6f-4f7a-a595-80f44756d097]: [Errno 2] No such file or directory: 'systool' {{(pid=154475) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 16 21:37:19.051358 ncdv-master-1 cinder-backup[130586]: Traceback (most recent call last): Apr 16 21:37:19.051358 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 16 21:37:19.051358 ncdv-master-1 cinder-backup[130586]: ret = func(*f_args, **f_kwargs) Apr 16 21:37:19.051358 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 16 21:37:19.051358 ncdv-master-1 cinder-backup[130586]: return func(*args, **kwargs) Apr 16 21:37:19.051358 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 16 21:37:19.051358 ncdv-master-1 cinder-backup[130586]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 16 21:37:19.051358 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 16 21:37:19.051358 ncdv-master-1 cinder-backup[130586]: return putils.execute(on_execute=on_execute, Apr 16 21:37:19.051358 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 16 21:37:19.051358 ncdv-master-1 cinder-backup[130586]: obj = subprocess.Popen(cmd, Apr 16 21:37:19.051358 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 16 21:37:19.051358 ncdv-master-1 cinder-backup[130586]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 16 21:37:19.051358 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 16 21:37:19.051358 ncdv-master-1 cinder-backup[130586]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 16 21:37:19.051358 ncdv-master-1 cinder-backup[130586]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 16 21:37:19.056625 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[bb530fbe-6d6f-4f7a-a595-80f44756d097]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:37:19.056625 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:37:19.063661 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.010s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:37:19.064277 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[b9038864-029c-4354-a566-680b983317f7]: (4, ('/dev/sda1\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:37:19.065664 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:37:19.076780 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.011s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:37:19.077337 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[f1d70cdb-5e40-47f1-85d1-aa42c63983eb]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:37:19.078804 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:37:19.088774 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.010s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:37:19.089099 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[23a92abd-7c36-4d00-9e95-647120741997]: (4, ('efede189-2323-8540-b968-072c0c8c1315\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:37:19.089879 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] Running cmd (subprocess): nvme version {{(pid=130586) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:37:19.110177 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] CMD "nvme version" returned: 0 in 0.020s {{(pid=130586) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:37:19.111951 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=130586) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Apr 16 21:37:19.112376 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=130586) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Apr 16 21:37:19.112755 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:3a91007f-78fd-42da-ba0d-d5efe907ca61 dsc: {{(pid=130586) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Apr 16 21:37:19.113393 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.utils [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] <== get_connector_properties: return (93ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.20', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ff85ebdbdb9', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'efede189-2323-8540-b968-072c0c8c1315', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:3a91007f-78fd-42da-ba0d-d5efe907ca61', 'found_dsc': ''} {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:37:19.818697 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connector [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] Factory for rbd on None {{(pid=130586) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Apr 16 21:37:19.821395 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-0dc9c2b9-1dc0-4ef0-b0d6-2a09be69e861', 'hosts': ['10.222.0.20'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '0dc9c2b9-1dc0-4ef0-b0d6-2a09be69e861', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:37:19.822673 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.linuxrbd [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] opening connection to ceph cluster (timeout=-1). {{(pid=130586) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Apr 16 21:37:19.949154 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] <== connect_volume: return (127ms) {'path': } {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:37:19.957067 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_db.sqlalchemy.engines [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=130586) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 16 21:37:20.041898 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] Starting backup of volume='0dc9c2b9-1dc0-4ef0-b0d6-2a09be69e861'. {{(pid=130586) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} Apr 16 21:37:20.042834 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] Volume file is RBD: attempting optimized backup {{(pid=130586) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}} Apr 16 21:37:20.135575 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] Creating base image 'volume-0dc9c2b9-1dc0-4ef0-b0d6-2a09be69e861.backup.8e05f40b-c063-449b-8140-ed75064781b3' {{(pid=130586) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}} Apr 16 21:37:20.136110 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=130586) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} Apr 16 21:37:20.993245 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] Using --from-snap 'None' for incremental backup of volume 0dc9c2b9-1dc0-4ef0-b0d6-2a09be69e861. {{(pid=130586) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}} Apr 16 21:37:20.993802 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] Creating backup snapshot='backup.8e05f40b-c063-449b-8140-ed75064781b3.snap.1650145040.99327' {{(pid=130586) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}} Apr 16 21:37:22.462825 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] Performing differential transfer from 'volume-0dc9c2b9-1dc0-4ef0-b0d6-2a09be69e861' to 'volume-0dc9c2b9-1dc0-4ef0-b0d6-2a09be69e861.backup.8e05f40b-c063-449b-8140-ed75064781b3' {{(pid=130586) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} Apr 16 21:37:22.463427 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_81xcu3uw --pool volumes volumes/volume-0dc9c2b9-1dc0-4ef0-b0d6-2a09be69e861@backup.8e05f40b-c063-449b-8140-ed75064781b3.snap.1650145040.99327 -' into... {{(pid=130586) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} Apr 16 21:37:22.463675 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-0dc9c2b9-1dc0-4ef0-b0d6-2a09be69e861.backup.8e05f40b-c063-449b-8140-ed75064781b3' {{(pid=130586) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} Apr 16 21:37:26.226531 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] Differential backup transfer completed in 3.7629s {{(pid=130586) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}} Apr 16 21:37:26.227147 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] Getting metadata type 'volume-base-metadata' {{(pid=130586) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Apr 16 21:37:26.240958 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=130586) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Apr 16 21:37:26.241955 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] Getting metadata type 'volume-metadata' {{(pid=130586) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Apr 16 21:37:26.250746 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] No metadata type 'volume-metadata' available {{(pid=130586) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Apr 16 21:37:26.251434 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] Getting metadata type 'volume-glance-metadata' {{(pid=130586) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Apr 16 21:37:26.264378 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] No metadata type 'volume-glance-metadata' available {{(pid=130586) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Apr 16 21:37:26.265155 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] Backing up metadata for volume 0dc9c2b9-1dc0-4ef0-b0d6-2a09be69e861. {{(pid=130586) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} Apr 16 21:37:26.502535 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] Backup '8e05f40b-c063-449b-8140-ed75064781b3' of volume 0dc9c2b9-1dc0-4ef0-b0d6-2a09be69e861 finished. {{(pid=130586) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} Apr 16 21:37:26.504138 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-0dc9c2b9-1dc0-4ef0-b0d6-2a09be69e861', 'hosts': ['10.222.0.20'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '0dc9c2b9-1dc0-4ef0-b0d6-2a09be69e861', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:37:26.868997 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] <== disconnect_volume: return (363ms) None {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:37:27.041573 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-6d6b2bbc-f07d-454b-a500-f68b157c0071 tempest-VolumesBackupsTest-493584864 None] Create backup finished. backup: 8e05f40b-c063-449b-8140-ed75064781b3. Apr 16 21:37:27.373395 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-1561807e-c9b3-4bdf-91e9-4be60d3119c5 tempest-VolumesBackupsTest-493584864 None] Delete backup started, backup: 8e05f40b-c063-449b-8140-ed75064781b3. Apr 16 21:37:27.374946 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-1561807e-c9b3-4bdf-91e9-4be60d3119c5 tempest-VolumesBackupsTest-493584864 None] Delete started for backup=8e05f40b-c063-449b-8140-ed75064781b3 {{(pid=130586) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} Apr 16 21:37:27.375377 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-1561807e-c9b3-4bdf-91e9-4be60d3119c5 tempest-VolumesBackupsTest-493584864 None] Trying diff format basename='volume-0dc9c2b9-1dc0-4ef0-b0d6-2a09be69e861.backup.8e05f40b-c063-449b-8140-ed75064781b3' for backup base image of volume 0dc9c2b9-1dc0-4ef0-b0d6-2a09be69e861. {{(pid=130586) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} Apr 16 21:37:27.666175 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-1561807e-c9b3-4bdf-91e9-4be60d3119c5 tempest-VolumesBackupsTest-493584864 None] Looking for snapshot of backup base 'volume-0dc9c2b9-1dc0-4ef0-b0d6-2a09be69e861.backup.8e05f40b-c063-449b-8140-ed75064781b3' {{(pid=130586) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} Apr 16 21:37:27.666578 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-1561807e-c9b3-4bdf-91e9-4be60d3119c5 tempest-VolumesBackupsTest-493584864 None] Found snapshot 'backup.8e05f40b-c063-449b-8140-ed75064781b3.snap.1650145040.99327' {{(pid=130586) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} Apr 16 21:37:27.667607 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-1561807e-c9b3-4bdf-91e9-4be60d3119c5 tempest-VolumesBackupsTest-493584864 None] Deleting backup snapshot='backup.8e05f40b-c063-449b-8140-ed75064781b3.snap.1650145040.99327' {{(pid=130586) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}} Apr 16 21:37:30.660151 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.drivers.ceph [None req-1561807e-c9b3-4bdf-91e9-4be60d3119c5 tempest-VolumesBackupsTest-493584864 None] Deleting backup base image='volume-0dc9c2b9-1dc0-4ef0-b0d6-2a09be69e861.backup.8e05f40b-c063-449b-8140-ed75064781b3' of volume 0dc9c2b9-1dc0-4ef0-b0d6-2a09be69e861. Apr 16 21:37:34.023557 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_db.sqlalchemy.engines [None req-61e8a934-990b-4a99-9b63-11946b0b3861 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=130586) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 16 21:37:35.556336 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-1561807e-c9b3-4bdf-91e9-4be60d3119c5 tempest-VolumesBackupsTest-493584864 None] Base backup image='volume-0dc9c2b9-1dc0-4ef0-b0d6-2a09be69e861.backup.8e05f40b-c063-449b-8140-ed75064781b3' of volume 0dc9c2b9-1dc0-4ef0-b0d6-2a09be69e861 deleted. {{(pid=130586) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} Apr 16 21:37:35.713102 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-1561807e-c9b3-4bdf-91e9-4be60d3119c5 tempest-VolumesBackupsTest-493584864 None] Delete of backup '8e05f40b-c063-449b-8140-ed75064781b3' for volume '0dc9c2b9-1dc0-4ef0-b0d6-2a09be69e861' finished. {{(pid=130586) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} Apr 16 21:37:35.816920 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.quota [None req-1561807e-c9b3-4bdf-91e9-4be60d3119c5 tempest-VolumesBackupsTest-493584864 None] Created reservations ['ef3770b2-88ef-45bf-92fe-083d0575935b', '65cc0a2f-df5b-45c1-b886-97f923180f6b'] {{(pid=130586) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 16 21:37:36.000755 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-1561807e-c9b3-4bdf-91e9-4be60d3119c5 tempest-VolumesBackupsTest-493584864 None] Delete backup finished, backup 8e05f40b-c063-449b-8140-ed75064781b3 deleted. Apr 16 21:38:00.002536 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] Create backup started, backup: ab06f9fe-c532-404f-b0e9-d1e0343b39c8 volume: 2a52b0c3-1018-488f-bff4-8d3025861720. Apr 16 21:38:00.101152 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-04-16T21:37:59Z,data_timestamp=2022-04-16T21:37:59Z,deleted=False,deleted_at=None,display_description=None,display_name=None,encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=ab06f9fe-c532-404f-b0e9-d1e0343b39c8,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='80715ea8284b48959a67d33e8be6b560',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='52bbf82f2ac64a4c9e3b86975f8173b2',volume_id=2a52b0c3-1018-488f-bff4-8d3025861720) Apr 16 21:38:00.149654 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.utils [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.20', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:38:00.153922 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:38:00.161222 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.010s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:38:00.161571 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[3cf3ee98-bbe6-4d1f-9ed4-41f4132b59e8]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ff85ebdbdb9\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:38:00.163706 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:38:00.172529 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 16 21:38:00.173940 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[08b6a57c-5800-4d4a-8647-783c07054f7c]: [Errno 2] No such file or directory: 'systool' {{(pid=154475) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 16 21:38:00.173940 ncdv-master-1 cinder-backup[130586]: Traceback (most recent call last): Apr 16 21:38:00.173940 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 16 21:38:00.173940 ncdv-master-1 cinder-backup[130586]: ret = func(*f_args, **f_kwargs) Apr 16 21:38:00.173940 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 16 21:38:00.173940 ncdv-master-1 cinder-backup[130586]: return func(*args, **kwargs) Apr 16 21:38:00.173940 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 16 21:38:00.173940 ncdv-master-1 cinder-backup[130586]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 16 21:38:00.173940 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 16 21:38:00.173940 ncdv-master-1 cinder-backup[130586]: return putils.execute(on_execute=on_execute, Apr 16 21:38:00.173940 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 16 21:38:00.173940 ncdv-master-1 cinder-backup[130586]: obj = subprocess.Popen(cmd, Apr 16 21:38:00.173940 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 16 21:38:00.173940 ncdv-master-1 cinder-backup[130586]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 16 21:38:00.173940 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 16 21:38:00.173940 ncdv-master-1 cinder-backup[130586]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 16 21:38:00.173940 ncdv-master-1 cinder-backup[130586]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 16 21:38:00.175544 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[08b6a57c-5800-4d4a-8647-783c07054f7c]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:38:00.180302 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:38:00.189896 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 16 21:38:00.190200 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[6155cbb5-08f4-41a1-9c1d-bd8c5e40c163]: [Errno 2] No such file or directory: 'systool' {{(pid=154475) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 16 21:38:00.190200 ncdv-master-1 cinder-backup[130586]: Traceback (most recent call last): Apr 16 21:38:00.190200 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 16 21:38:00.190200 ncdv-master-1 cinder-backup[130586]: ret = func(*f_args, **f_kwargs) Apr 16 21:38:00.190200 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 16 21:38:00.190200 ncdv-master-1 cinder-backup[130586]: return func(*args, **kwargs) Apr 16 21:38:00.190200 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 16 21:38:00.190200 ncdv-master-1 cinder-backup[130586]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 16 21:38:00.190200 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 16 21:38:00.190200 ncdv-master-1 cinder-backup[130586]: return putils.execute(on_execute=on_execute, Apr 16 21:38:00.190200 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 16 21:38:00.190200 ncdv-master-1 cinder-backup[130586]: obj = subprocess.Popen(cmd, Apr 16 21:38:00.190200 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 16 21:38:00.190200 ncdv-master-1 cinder-backup[130586]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 16 21:38:00.190200 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 16 21:38:00.190200 ncdv-master-1 cinder-backup[130586]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 16 21:38:00.190200 ncdv-master-1 cinder-backup[130586]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 16 21:38:00.192125 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[6155cbb5-08f4-41a1-9c1d-bd8c5e40c163]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:38:00.194695 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:38:00.205535 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.013s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:38:00.205852 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[04ac2dfd-84ab-4eff-abf5-791109a51f50]: (4, ('/dev/sda1\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:38:00.207756 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:38:00.225773 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.018s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:38:00.226185 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[2d021d0f-222e-4e98-be53-21fb6c7adc25]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:38:00.230086 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:38:00.239771 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.012s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:38:00.240286 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[43d3fc0f-4da4-4ecb-a28f-88b2a1747a9a]: (4, ('efede189-2323-8540-b968-072c0c8c1315\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:38:00.241160 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] Running cmd (subprocess): nvme version {{(pid=130586) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:38:00.259711 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] CMD "nvme version" returned: 0 in 0.018s {{(pid=130586) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:38:00.261541 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=130586) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Apr 16 21:38:00.261928 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=130586) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Apr 16 21:38:00.262422 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:3a91007f-78fd-42da-ba0d-d5efe907ca61 dsc: {{(pid=130586) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Apr 16 21:38:00.263067 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.utils [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] <== get_connector_properties: return (112ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.20', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ff85ebdbdb9', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'efede189-2323-8540-b968-072c0c8c1315', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:3a91007f-78fd-42da-ba0d-d5efe907ca61', 'found_dsc': ''} {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:38:01.063250 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connector [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] Factory for rbd on None {{(pid=130586) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Apr 16 21:38:01.064783 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-2a52b0c3-1018-488f-bff4-8d3025861720', 'hosts': ['10.222.0.20'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '2a52b0c3-1018-488f-bff4-8d3025861720', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:38:01.065847 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.linuxrbd [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] opening connection to ceph cluster (timeout=-1). {{(pid=130586) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Apr 16 21:38:01.473452 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] <== connect_volume: return (408ms) {'path': } {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:38:01.482496 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_db.sqlalchemy.engines [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=130586) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 16 21:38:01.555259 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] Create backup started, backup: 9c0f0465-f3dc-45d2-801b-939c0d169abe volume: 46ed5edf-8b2e-4f49-a9bc-d26d3e657685. Apr 16 21:38:01.564540 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] Starting backup of volume='2a52b0c3-1018-488f-bff4-8d3025861720'. {{(pid=130586) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} Apr 16 21:38:01.565422 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] Volume file is RBD: attempting optimized backup {{(pid=130586) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}} Apr 16 21:38:01.586295 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] Creating base image 'volume-2a52b0c3-1018-488f-bff4-8d3025861720.backup.ab06f9fe-c532-404f-b0e9-d1e0343b39c8' {{(pid=130586) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}} Apr 16 21:38:01.586741 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=130586) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} Apr 16 21:38:01.649986 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-04-16T21:38:01Z,data_timestamp=2022-04-16T21:38:01Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsTest-Backup-961156290',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=9c0f0465-f3dc-45d2-801b-939c0d169abe,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='d51b576f9c0241c1a254f15308a455ec',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='77f61ccbf5114e6080e33f0c5f500468',volume_id=46ed5edf-8b2e-4f49-a9bc-d26d3e657685) Apr 16 21:38:01.697484 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.utils [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.20', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:38:01.698820 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:38:01.708140 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.009s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:38:01.708569 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[0f1dbf60-1432-448f-8640-eaf5bd4b11db]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ff85ebdbdb9\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:38:01.710120 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:38:01.715603 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 16 21:38:01.716272 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[0a3d535b-f5f9-44e7-a16b-389f844febe8]: [Errno 2] No such file or directory: 'systool' {{(pid=154475) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 16 21:38:01.716272 ncdv-master-1 cinder-backup[130586]: Traceback (most recent call last): Apr 16 21:38:01.716272 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 16 21:38:01.716272 ncdv-master-1 cinder-backup[130586]: ret = func(*f_args, **f_kwargs) Apr 16 21:38:01.716272 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 16 21:38:01.716272 ncdv-master-1 cinder-backup[130586]: return func(*args, **kwargs) Apr 16 21:38:01.716272 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 16 21:38:01.716272 ncdv-master-1 cinder-backup[130586]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 16 21:38:01.716272 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 16 21:38:01.716272 ncdv-master-1 cinder-backup[130586]: return putils.execute(on_execute=on_execute, Apr 16 21:38:01.716272 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 16 21:38:01.716272 ncdv-master-1 cinder-backup[130586]: obj = subprocess.Popen(cmd, Apr 16 21:38:01.716272 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 16 21:38:01.716272 ncdv-master-1 cinder-backup[130586]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 16 21:38:01.716272 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 16 21:38:01.716272 ncdv-master-1 cinder-backup[130586]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 16 21:38:01.716272 ncdv-master-1 cinder-backup[130586]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 16 21:38:01.717436 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[0a3d535b-f5f9-44e7-a16b-389f844febe8]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:38:01.717893 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:38:01.723808 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 16 21:38:01.724403 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[0283d78f-40cb-4828-917d-1cd6fdc4c109]: [Errno 2] No such file or directory: 'systool' {{(pid=154475) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 16 21:38:01.724403 ncdv-master-1 cinder-backup[130586]: Traceback (most recent call last): Apr 16 21:38:01.724403 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 16 21:38:01.724403 ncdv-master-1 cinder-backup[130586]: ret = func(*f_args, **f_kwargs) Apr 16 21:38:01.724403 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 16 21:38:01.724403 ncdv-master-1 cinder-backup[130586]: return func(*args, **kwargs) Apr 16 21:38:01.724403 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 16 21:38:01.724403 ncdv-master-1 cinder-backup[130586]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 16 21:38:01.724403 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 16 21:38:01.724403 ncdv-master-1 cinder-backup[130586]: return putils.execute(on_execute=on_execute, Apr 16 21:38:01.724403 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 16 21:38:01.724403 ncdv-master-1 cinder-backup[130586]: obj = subprocess.Popen(cmd, Apr 16 21:38:01.724403 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 16 21:38:01.724403 ncdv-master-1 cinder-backup[130586]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 16 21:38:01.724403 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 16 21:38:01.724403 ncdv-master-1 cinder-backup[130586]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 16 21:38:01.724403 ncdv-master-1 cinder-backup[130586]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 16 21:38:01.725481 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[0283d78f-40cb-4828-917d-1cd6fdc4c109]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:38:01.726017 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:38:01.734144 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.008s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:38:01.734533 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[654d4adf-e40d-482a-8b0c-00cd60c0b00f]: (4, ('/dev/sda1\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:38:01.735651 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:38:01.742790 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.007s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:38:01.743106 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[f0036eab-f05f-4529-ab2c-11e2041fd42c]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:38:01.744287 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:38:01.751192 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.007s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:38:01.751545 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[85e4e92b-4b90-42f9-b5d8-5e76fbd24578]: (4, ('efede189-2323-8540-b968-072c0c8c1315\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:38:01.756018 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] Running cmd (subprocess): nvme version {{(pid=130586) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:38:01.771983 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] CMD "nvme version" returned: 0 in 0.019s {{(pid=130586) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:38:01.773835 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=130586) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Apr 16 21:38:01.774244 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=130586) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Apr 16 21:38:01.774623 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:3a91007f-78fd-42da-ba0d-d5efe907ca61 dsc: {{(pid=130586) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Apr 16 21:38:01.779027 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.utils [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] <== get_connector_properties: return (77ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.20', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ff85ebdbdb9', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'efede189-2323-8540-b968-072c0c8c1315', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:3a91007f-78fd-42da-ba0d-d5efe907ca61', 'found_dsc': ''} {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:38:02.455883 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connector [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] Factory for rbd on None {{(pid=130586) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Apr 16 21:38:02.458098 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-46ed5edf-8b2e-4f49-a9bc-d26d3e657685', 'hosts': ['10.222.0.20'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '46ed5edf-8b2e-4f49-a9bc-d26d3e657685', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:38:02.459315 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.linuxrbd [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] opening connection to ceph cluster (timeout=-1). {{(pid=130586) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Apr 16 21:38:02.879631 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] <== connect_volume: return (419ms) {'path': } {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:38:02.889015 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_db.sqlalchemy.engines [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=130586) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 16 21:38:02.890140 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] Using --from-snap 'None' for incremental backup of volume 2a52b0c3-1018-488f-bff4-8d3025861720. {{(pid=130586) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}} Apr 16 21:38:02.890845 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] Creating backup snapshot='backup.ab06f9fe-c532-404f-b0e9-d1e0343b39c8.snap.1650145082.8901744' {{(pid=130586) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}} Apr 16 21:38:02.976628 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] Starting backup of volume='46ed5edf-8b2e-4f49-a9bc-d26d3e657685'. {{(pid=130586) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} Apr 16 21:38:02.979884 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] Volume file is RBD: attempting optimized backup {{(pid=130586) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}} Apr 16 21:38:03.008137 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] Creating base image 'volume-46ed5edf-8b2e-4f49-a9bc-d26d3e657685.backup.9c0f0465-f3dc-45d2-801b-939c0d169abe' {{(pid=130586) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}} Apr 16 21:38:03.008807 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=130586) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} Apr 16 21:38:04.105462 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] Using --from-snap 'None' for incremental backup of volume 46ed5edf-8b2e-4f49-a9bc-d26d3e657685. {{(pid=130586) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}} Apr 16 21:38:04.106550 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] Creating backup snapshot='backup.9c0f0465-f3dc-45d2-801b-939c0d169abe.snap.1650145084.1061025' {{(pid=130586) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}} Apr 16 21:38:05.406639 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] Performing differential transfer from 'volume-2a52b0c3-1018-488f-bff4-8d3025861720' to 'volume-2a52b0c3-1018-488f-bff4-8d3025861720.backup.ab06f9fe-c532-404f-b0e9-d1e0343b39c8' {{(pid=130586) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} Apr 16 21:38:05.407284 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_8iav95bz --pool volumes volumes/volume-2a52b0c3-1018-488f-bff4-8d3025861720@backup.ab06f9fe-c532-404f-b0e9-d1e0343b39c8.snap.1650145082.8901744 -' into... {{(pid=130586) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} Apr 16 21:38:05.407449 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-2a52b0c3-1018-488f-bff4-8d3025861720.backup.ab06f9fe-c532-404f-b0e9-d1e0343b39c8' {{(pid=130586) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} Apr 16 21:38:07.227517 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] Performing differential transfer from 'volume-46ed5edf-8b2e-4f49-a9bc-d26d3e657685' to 'volume-46ed5edf-8b2e-4f49-a9bc-d26d3e657685.backup.9c0f0465-f3dc-45d2-801b-939c0d169abe' {{(pid=130586) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} Apr 16 21:38:07.228937 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_pu85r0ti --pool volumes volumes/volume-46ed5edf-8b2e-4f49-a9bc-d26d3e657685@backup.9c0f0465-f3dc-45d2-801b-939c0d169abe.snap.1650145084.1061025 -' into... {{(pid=130586) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} Apr 16 21:38:07.229755 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-46ed5edf-8b2e-4f49-a9bc-d26d3e657685.backup.9c0f0465-f3dc-45d2-801b-939c0d169abe' {{(pid=130586) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} Apr 16 21:38:18.920805 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:38:18.921447 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:39:18.572098 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] Differential backup transfer completed in 73.1654s {{(pid=130586) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}} Apr 16 21:39:18.572851 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] Getting metadata type 'volume-base-metadata' {{(pid=130586) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Apr 16 21:39:18.596009 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=130586) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Apr 16 21:39:18.596478 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] Getting metadata type 'volume-metadata' {{(pid=130586) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Apr 16 21:39:18.603786 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] No metadata type 'volume-metadata' available {{(pid=130586) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Apr 16 21:39:18.604318 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] Getting metadata type 'volume-glance-metadata' {{(pid=130586) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Apr 16 21:39:18.620335 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=130586) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}} Apr 16 21:39:18.621103 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] Backing up metadata for volume 2a52b0c3-1018-488f-bff4-8d3025861720. {{(pid=130586) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} Apr 16 21:39:18.918760 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:39:18.919085 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:39:19.054475 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] Backup 'ab06f9fe-c532-404f-b0e9-d1e0343b39c8' of volume 2a52b0c3-1018-488f-bff4-8d3025861720 finished. {{(pid=130586) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} Apr 16 21:39:19.056200 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-2a52b0c3-1018-488f-bff4-8d3025861720', 'hosts': ['10.222.0.20'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '2a52b0c3-1018-488f-bff4-8d3025861720', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:39:19.937974 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] <== disconnect_volume: return (880ms) None {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:39:20.221537 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-c8b3e760-4eec-4d04-bf57-1b0b212994f0 tempest-TestVolumeBackupRestore-1675082881 None] Create backup finished. backup: ab06f9fe-c532-404f-b0e9-d1e0343b39c8. Apr 16 21:39:20.453441 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] Differential backup transfer completed in 73.2278s {{(pid=130586) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}} Apr 16 21:39:20.454094 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] Getting metadata type 'volume-base-metadata' {{(pid=130586) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Apr 16 21:39:20.469858 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=130586) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Apr 16 21:39:20.470936 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] Getting metadata type 'volume-metadata' {{(pid=130586) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Apr 16 21:39:20.479894 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] No metadata type 'volume-metadata' available {{(pid=130586) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Apr 16 21:39:20.481037 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] Getting metadata type 'volume-glance-metadata' {{(pid=130586) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Apr 16 21:39:20.491184 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=130586) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}} Apr 16 21:39:20.491979 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] Backing up metadata for volume 46ed5edf-8b2e-4f49-a9bc-d26d3e657685. {{(pid=130586) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} Apr 16 21:39:20.690508 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] Backup '9c0f0465-f3dc-45d2-801b-939c0d169abe' of volume 46ed5edf-8b2e-4f49-a9bc-d26d3e657685 finished. {{(pid=130586) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} Apr 16 21:39:20.692088 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-46ed5edf-8b2e-4f49-a9bc-d26d3e657685', 'hosts': ['10.222.0.20'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '46ed5edf-8b2e-4f49-a9bc-d26d3e657685', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:39:21.538210 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] <== disconnect_volume: return (844ms) None {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:39:21.828327 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-cb2ef55b-4223-4da7-9fd0-1d3d20e5290b tempest-VolumesBackupsTest-493584864 None] Create backup finished. backup: 9c0f0465-f3dc-45d2-801b-939c0d169abe. Apr 16 21:39:24.346441 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] Restore backup started, backup: ab06f9fe-c532-404f-b0e9-d1e0343b39c8 volume: 9312ec94-740a-4dce-bd71-1c9d823af28d. Apr 16 21:39:24.353182 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.utils [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.20', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:39:24.354795 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:39:24.376198 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.020s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:39:24.377592 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[7c5d7545-1338-472e-95e7-68563e638000]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ff85ebdbdb9\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:39:24.380365 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:39:24.403584 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 16 21:39:24.403584 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[84833aa8-c2ae-4bfe-a7ae-abda08268101]: [Errno 2] No such file or directory: 'systool' {{(pid=154475) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 16 21:39:24.403584 ncdv-master-1 cinder-backup[130586]: Traceback (most recent call last): Apr 16 21:39:24.403584 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 16 21:39:24.403584 ncdv-master-1 cinder-backup[130586]: ret = func(*f_args, **f_kwargs) Apr 16 21:39:24.403584 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 16 21:39:24.403584 ncdv-master-1 cinder-backup[130586]: return func(*args, **kwargs) Apr 16 21:39:24.403584 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 16 21:39:24.403584 ncdv-master-1 cinder-backup[130586]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 16 21:39:24.403584 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 16 21:39:24.403584 ncdv-master-1 cinder-backup[130586]: return putils.execute(on_execute=on_execute, Apr 16 21:39:24.403584 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 16 21:39:24.403584 ncdv-master-1 cinder-backup[130586]: obj = subprocess.Popen(cmd, Apr 16 21:39:24.403584 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 16 21:39:24.403584 ncdv-master-1 cinder-backup[130586]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 16 21:39:24.403584 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 16 21:39:24.403584 ncdv-master-1 cinder-backup[130586]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 16 21:39:24.403584 ncdv-master-1 cinder-backup[130586]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 16 21:39:24.403584 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[84833aa8-c2ae-4bfe-a7ae-abda08268101]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:39:24.406708 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:39:24.432456 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 16 21:39:24.433086 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[5f67a835-8b45-4d63-9ea9-a22d2844da00]: [Errno 2] No such file or directory: 'systool' {{(pid=154475) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 16 21:39:24.433086 ncdv-master-1 cinder-backup[130586]: Traceback (most recent call last): Apr 16 21:39:24.433086 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 16 21:39:24.433086 ncdv-master-1 cinder-backup[130586]: ret = func(*f_args, **f_kwargs) Apr 16 21:39:24.433086 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 16 21:39:24.433086 ncdv-master-1 cinder-backup[130586]: return func(*args, **kwargs) Apr 16 21:39:24.433086 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 16 21:39:24.433086 ncdv-master-1 cinder-backup[130586]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 16 21:39:24.433086 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 16 21:39:24.433086 ncdv-master-1 cinder-backup[130586]: return putils.execute(on_execute=on_execute, Apr 16 21:39:24.433086 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 16 21:39:24.433086 ncdv-master-1 cinder-backup[130586]: obj = subprocess.Popen(cmd, Apr 16 21:39:24.433086 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 16 21:39:24.433086 ncdv-master-1 cinder-backup[130586]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 16 21:39:24.433086 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 16 21:39:24.433086 ncdv-master-1 cinder-backup[130586]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 16 21:39:24.433086 ncdv-master-1 cinder-backup[130586]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 16 21:39:24.434643 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[5f67a835-8b45-4d63-9ea9-a22d2844da00]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:39:24.436552 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:39:24.465009 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.028s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:39:24.465532 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[801e5e35-648b-496c-9e19-4e571523d567]: (4, ('/dev/sda1\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:39:24.466656 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:39:24.487012 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.020s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:39:24.487571 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[96e9e25e-222f-46e1-96d1-5a20e899541e]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:39:24.488848 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:39:24.502161 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.012s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:39:24.502456 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[91a8b5d2-8612-49d7-9ac2-9e492fcd027b]: (4, ('efede189-2323-8540-b968-072c0c8c1315\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:39:24.503295 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] Running cmd (subprocess): nvme version {{(pid=130586) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:39:24.535829 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] CMD "nvme version" returned: 0 in 0.032s {{(pid=130586) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:39:24.537705 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=130586) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Apr 16 21:39:24.538122 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=130586) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Apr 16 21:39:24.538497 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:3a91007f-78fd-42da-ba0d-d5efe907ca61 dsc: {{(pid=130586) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Apr 16 21:39:24.539925 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.utils [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] <== get_connector_properties: return (185ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.20', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ff85ebdbdb9', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'efede189-2323-8540-b968-072c0c8c1315', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:3a91007f-78fd-42da-ba0d-d5efe907ca61', 'found_dsc': ''} {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:39:25.475034 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connector [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] Factory for rbd on None {{(pid=130586) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Apr 16 21:39:25.477532 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-9312ec94-740a-4dce-bd71-1c9d823af28d', 'hosts': ['10.222.0.20'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '9312ec94-740a-4dce-bd71-1c9d823af28d', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:39:25.482366 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.linuxrbd [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] opening connection to ceph cluster (timeout=-1). {{(pid=130586) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Apr 16 21:39:25.881021 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] <== connect_volume: return (400ms) {'path': } {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:39:25.900891 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] Restore backup started, backup: 9c0f0465-f3dc-45d2-801b-939c0d169abe volume: e84da828-0f94-4f59-8e11-fac1f7cb4977. Apr 16 21:39:25.916827 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_db.sqlalchemy.engines [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=130586) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 16 21:39:25.947112 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.utils [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.20', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:39:25.953260 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:39:25.954240 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] Starting restore from Ceph backup=ab06f9fe-c532-404f-b0e9-d1e0343b39c8 to volume=volume-9312ec94-740a-4dce-bd71-1c9d823af28d {{(pid=130586) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1343}} Apr 16 21:39:25.969399 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.021s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:39:25.969752 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[ea0d28fa-4886-4422-92a7-169966d43a17]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ff85ebdbdb9\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:39:25.980970 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:39:25.988094 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 16 21:39:25.988664 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[2b42c6bc-4249-4f20-b8a6-ca6f2e331680]: [Errno 2] No such file or directory: 'systool' {{(pid=154475) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 16 21:39:25.988664 ncdv-master-1 cinder-backup[130586]: Traceback (most recent call last): Apr 16 21:39:25.988664 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 16 21:39:25.988664 ncdv-master-1 cinder-backup[130586]: ret = func(*f_args, **f_kwargs) Apr 16 21:39:25.988664 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 16 21:39:25.988664 ncdv-master-1 cinder-backup[130586]: return func(*args, **kwargs) Apr 16 21:39:25.988664 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 16 21:39:25.988664 ncdv-master-1 cinder-backup[130586]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 16 21:39:25.988664 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 16 21:39:25.988664 ncdv-master-1 cinder-backup[130586]: return putils.execute(on_execute=on_execute, Apr 16 21:39:25.988664 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 16 21:39:25.988664 ncdv-master-1 cinder-backup[130586]: obj = subprocess.Popen(cmd, Apr 16 21:39:25.988664 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 16 21:39:25.988664 ncdv-master-1 cinder-backup[130586]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 16 21:39:25.988664 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 16 21:39:25.988664 ncdv-master-1 cinder-backup[130586]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 16 21:39:25.988664 ncdv-master-1 cinder-backup[130586]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 16 21:39:25.991158 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[2b42c6bc-4249-4f20-b8a6-ca6f2e331680]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:39:25.992884 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:39:26.002302 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 16 21:39:26.003882 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[c37f859e-2819-4f4d-a29d-66bb01ecb44e]: [Errno 2] No such file or directory: 'systool' {{(pid=154475) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 16 21:39:26.003882 ncdv-master-1 cinder-backup[130586]: Traceback (most recent call last): Apr 16 21:39:26.003882 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 16 21:39:26.003882 ncdv-master-1 cinder-backup[130586]: ret = func(*f_args, **f_kwargs) Apr 16 21:39:26.003882 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 16 21:39:26.003882 ncdv-master-1 cinder-backup[130586]: return func(*args, **kwargs) Apr 16 21:39:26.003882 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 16 21:39:26.003882 ncdv-master-1 cinder-backup[130586]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 16 21:39:26.003882 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 16 21:39:26.003882 ncdv-master-1 cinder-backup[130586]: return putils.execute(on_execute=on_execute, Apr 16 21:39:26.003882 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 16 21:39:26.003882 ncdv-master-1 cinder-backup[130586]: obj = subprocess.Popen(cmd, Apr 16 21:39:26.003882 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 16 21:39:26.003882 ncdv-master-1 cinder-backup[130586]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 16 21:39:26.003882 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 16 21:39:26.003882 ncdv-master-1 cinder-backup[130586]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 16 21:39:26.003882 ncdv-master-1 cinder-backup[130586]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 16 21:39:26.006003 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[c37f859e-2819-4f4d-a29d-66bb01ecb44e]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:39:26.007425 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:39:26.025166 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.017s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:39:26.025802 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[9dc98ac2-320b-406d-96ba-b79b2a71da89]: (4, ('/dev/sda1\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:39:26.029041 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:39:26.043213 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.014s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:39:26.043553 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[1e91b1f9-008a-4e4a-87f4-c3042e0b378a]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:39:26.047628 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:39:26.086532 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.014s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:39:26.087032 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[d1a3c9de-92bc-42f5-9f79-b61700d5cc7d]: (4, ('efede189-2323-8540-b968-072c0c8c1315\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:39:26.088825 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] Running cmd (subprocess): nvme version {{(pid=130586) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:39:26.115624 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] Looking for snapshot of backup base 'volume-2a52b0c3-1018-488f-bff4-8d3025861720.backup.ab06f9fe-c532-404f-b0e9-d1e0343b39c8' {{(pid=130586) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} Apr 16 21:39:26.116026 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] Found snapshot 'backup.ab06f9fe-c532-404f-b0e9-d1e0343b39c8.snap.1650145082.8901744' {{(pid=130586) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} Apr 16 21:39:26.117710 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] CMD "nvme version" returned: 0 in 0.029s {{(pid=130586) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:39:26.119363 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=130586) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Apr 16 21:39:26.119780 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=130586) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Apr 16 21:39:26.120113 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:3a91007f-78fd-42da-ba0d-d5efe907ca61 dsc: {{(pid=130586) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Apr 16 21:39:26.120765 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.utils [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] <== get_connector_properties: return (172ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.20', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ff85ebdbdb9', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'efede189-2323-8540-b968-072c0c8c1315', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:3a91007f-78fd-42da-ba0d-d5efe907ca61', 'found_dsc': ''} {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:39:26.137399 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] Volume file is RBD. {{(pid=130586) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1245}} Apr 16 21:39:26.228952 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] Attempting differential restore. {{(pid=130586) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1298}} Apr 16 21:39:26.229486 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] Attempting incremental restore from base='volume-2a52b0c3-1018-488f-bff4-8d3025861720.backup.ab06f9fe-c532-404f-b0e9-d1e0343b39c8' snap='backup.ab06f9fe-c532-404f-b0e9-d1e0343b39c8.snap.1650145082.8901744' {{(pid=130586) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1145}} Apr 16 21:39:26.229896 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] Performing differential transfer from 'volume-2a52b0c3-1018-488f-bff4-8d3025861720.backup.ab06f9fe-c532-404f-b0e9-d1e0343b39c8' to 'volume-9312ec94-740a-4dce-bd71-1c9d823af28d' {{(pid=130586) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} Apr 16 21:39:26.230302 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-2a52b0c3-1018-488f-bff4-8d3025861720.backup.ab06f9fe-c532-404f-b0e9-d1e0343b39c8@backup.ab06f9fe-c532-404f-b0e9-d1e0343b39c8.snap.1650145082.8901744 -' into... {{(pid=130586) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} Apr 16 21:39:26.230671 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_otzodb73 --pool volumes - volumes/volume-9312ec94-740a-4dce-bd71-1c9d823af28d' {{(pid=130586) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} Apr 16 21:39:27.206842 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connector [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] Factory for rbd on None {{(pid=130586) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Apr 16 21:39:27.210091 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-e84da828-0f94-4f59-8e11-fac1f7cb4977', 'hosts': ['10.222.0.20'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'e84da828-0f94-4f59-8e11-fac1f7cb4977', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:39:27.212354 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.linuxrbd [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] opening connection to ceph cluster (timeout=-1). {{(pid=130586) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Apr 16 21:39:27.634549 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] <== connect_volume: return (423ms) {'path': } {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:39:27.646915 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_db.sqlalchemy.engines [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=130586) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 16 21:39:27.656147 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] Starting restore from Ceph backup=9c0f0465-f3dc-45d2-801b-939c0d169abe to volume=volume-e84da828-0f94-4f59-8e11-fac1f7cb4977 {{(pid=130586) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1343}} Apr 16 21:39:27.775321 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] Looking for snapshot of backup base 'volume-46ed5edf-8b2e-4f49-a9bc-d26d3e657685.backup.9c0f0465-f3dc-45d2-801b-939c0d169abe' {{(pid=130586) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} Apr 16 21:39:27.775723 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] Found snapshot 'backup.9c0f0465-f3dc-45d2-801b-939c0d169abe.snap.1650145084.1061025' {{(pid=130586) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} Apr 16 21:39:27.789499 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] Volume file is RBD. {{(pid=130586) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1245}} Apr 16 21:39:27.847920 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] Attempting differential restore. {{(pid=130586) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1298}} Apr 16 21:39:27.847920 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] Attempting incremental restore from base='volume-46ed5edf-8b2e-4f49-a9bc-d26d3e657685.backup.9c0f0465-f3dc-45d2-801b-939c0d169abe' snap='backup.9c0f0465-f3dc-45d2-801b-939c0d169abe.snap.1650145084.1061025' {{(pid=130586) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1145}} Apr 16 21:39:27.847920 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] Performing differential transfer from 'volume-46ed5edf-8b2e-4f49-a9bc-d26d3e657685.backup.9c0f0465-f3dc-45d2-801b-939c0d169abe' to 'volume-e84da828-0f94-4f59-8e11-fac1f7cb4977' {{(pid=130586) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} Apr 16 21:39:27.847920 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-46ed5edf-8b2e-4f49-a9bc-d26d3e657685.backup.9c0f0465-f3dc-45d2-801b-939c0d169abe@backup.9c0f0465-f3dc-45d2-801b-939c0d169abe.snap.1650145084.1061025 -' into... {{(pid=130586) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} Apr 16 21:39:27.849114 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_goy4kx0l --pool volumes - volumes/volume-e84da828-0f94-4f59-8e11-fac1f7cb4977' {{(pid=130586) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} Apr 16 21:40:18.924269 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:40:18.924891 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:40:44.820931 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] Restore transfer completed in 78.5908s {{(pid=130586) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1167}} Apr 16 21:40:44.821981 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=130586) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1354}} Apr 16 21:40:44.844427 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] Restoring volume base metadata {{(pid=130586) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} Apr 16 21:40:44.930002 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] No metadata of type 'volume-metadata' to restore {{(pid=130586) put /opt/stack/cinder/cinder/backup/driver.py:345}} Apr 16 21:40:44.930332 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] Restoring volume glance metadata {{(pid=130586) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}} Apr 16 21:40:46.116391 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] Restore to volume 9312ec94-740a-4dce-bd71-1c9d823af28d finished successfully. {{(pid=130586) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1361}} Apr 16 21:40:46.118262 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-9312ec94-740a-4dce-bd71-1c9d823af28d', 'hosts': ['10.222.0.20'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '9312ec94-740a-4dce-bd71-1c9d823af28d', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:40:46.376300 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] <== disconnect_volume: return (256ms) None {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:40:46.439483 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.manager [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] Encryption key ID for volume 9312ec94-740a-4dce-bd71-1c9d823af28d already matches encryption key ID in backup ab06f9fe-c532-404f-b0e9-d1e0343b39c8. {{(pid=130586) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} Apr 16 21:40:46.467734 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] Restore transfer completed in 78.6204s {{(pid=130586) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1167}} Apr 16 21:40:46.470919 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=130586) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1354}} Apr 16 21:40:46.527049 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] Restoring volume base metadata {{(pid=130586) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} Apr 16 21:40:46.623743 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] No metadata of type 'volume-metadata' to restore {{(pid=130586) put /opt/stack/cinder/cinder/backup/driver.py:345}} Apr 16 21:40:46.624046 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] Restoring volume glance metadata {{(pid=130586) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}} Apr 16 21:40:46.725045 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-076de2d0-d325-4c1d-8cdd-b04ea03a1420 tempest-TestVolumeBackupRestore-1675082881 None] Finished restoring backup ab06f9fe-c532-404f-b0e9-d1e0343b39c8 to volume 9312ec94-740a-4dce-bd71-1c9d823af28d. Apr 16 21:40:47.759380 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] Restore to volume e84da828-0f94-4f59-8e11-fac1f7cb4977 finished successfully. {{(pid=130586) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1361}} Apr 16 21:40:47.760892 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-e84da828-0f94-4f59-8e11-fac1f7cb4977', 'hosts': ['10.222.0.20'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'e84da828-0f94-4f59-8e11-fac1f7cb4977', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:40:47.940872 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] <== disconnect_volume: return (179ms) None {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:40:48.002997 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.manager [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] Encryption key ID for volume e84da828-0f94-4f59-8e11-fac1f7cb4977 already matches encryption key ID in backup 9c0f0465-f3dc-45d2-801b-939c0d169abe. {{(pid=130586) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} Apr 16 21:40:48.246764 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-698107ed-98a0-4ce4-add6-08003883f586 tempest-VolumesBackupsTest-493584864 None] Finished restoring backup 9c0f0465-f3dc-45d2-801b-939c0d169abe to volume e84da828-0f94-4f59-8e11-fac1f7cb4977. Apr 16 21:41:00.564936 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-1d01c999-cd6a-4405-a9c6-cfdc7dd1a92e tempest-VolumesBackupsTest-493584864 None] Delete backup started, backup: 9c0f0465-f3dc-45d2-801b-939c0d169abe. Apr 16 21:41:00.567867 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-1d01c999-cd6a-4405-a9c6-cfdc7dd1a92e tempest-VolumesBackupsTest-493584864 None] Delete started for backup=9c0f0465-f3dc-45d2-801b-939c0d169abe {{(pid=130586) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} Apr 16 21:41:00.568345 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-1d01c999-cd6a-4405-a9c6-cfdc7dd1a92e tempest-VolumesBackupsTest-493584864 None] Trying diff format basename='volume-46ed5edf-8b2e-4f49-a9bc-d26d3e657685.backup.9c0f0465-f3dc-45d2-801b-939c0d169abe' for backup base image of volume 46ed5edf-8b2e-4f49-a9bc-d26d3e657685. {{(pid=130586) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} Apr 16 21:41:00.838683 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-1d01c999-cd6a-4405-a9c6-cfdc7dd1a92e tempest-VolumesBackupsTest-493584864 None] Looking for snapshot of backup base 'volume-46ed5edf-8b2e-4f49-a9bc-d26d3e657685.backup.9c0f0465-f3dc-45d2-801b-939c0d169abe' {{(pid=130586) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} Apr 16 21:41:00.839074 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-1d01c999-cd6a-4405-a9c6-cfdc7dd1a92e tempest-VolumesBackupsTest-493584864 None] Found snapshot 'backup.9c0f0465-f3dc-45d2-801b-939c0d169abe.snap.1650145084.1061025' {{(pid=130586) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} Apr 16 21:41:00.839448 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-1d01c999-cd6a-4405-a9c6-cfdc7dd1a92e tempest-VolumesBackupsTest-493584864 None] Deleting backup snapshot='backup.9c0f0465-f3dc-45d2-801b-939c0d169abe.snap.1650145084.1061025' {{(pid=130586) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}} Apr 16 21:41:03.890176 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.drivers.ceph [None req-1d01c999-cd6a-4405-a9c6-cfdc7dd1a92e tempest-VolumesBackupsTest-493584864 None] Deleting backup base image='volume-46ed5edf-8b2e-4f49-a9bc-d26d3e657685.backup.9c0f0465-f3dc-45d2-801b-939c0d169abe' of volume 46ed5edf-8b2e-4f49-a9bc-d26d3e657685. Apr 16 21:41:04.031954 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_db.sqlalchemy.engines [None req-61e8a934-990b-4a99-9b63-11946b0b3861 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=130586) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 16 21:41:12.880901 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-1d01c999-cd6a-4405-a9c6-cfdc7dd1a92e tempest-VolumesBackupsTest-493584864 None] Base backup image='volume-46ed5edf-8b2e-4f49-a9bc-d26d3e657685.backup.9c0f0465-f3dc-45d2-801b-939c0d169abe' of volume 46ed5edf-8b2e-4f49-a9bc-d26d3e657685 deleted. {{(pid=130586) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} Apr 16 21:41:13.157148 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-1d01c999-cd6a-4405-a9c6-cfdc7dd1a92e tempest-VolumesBackupsTest-493584864 None] Delete of backup '9c0f0465-f3dc-45d2-801b-939c0d169abe' for volume '46ed5edf-8b2e-4f49-a9bc-d26d3e657685' finished. {{(pid=130586) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} Apr 16 21:41:13.262444 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.quota [None req-1d01c999-cd6a-4405-a9c6-cfdc7dd1a92e tempest-VolumesBackupsTest-493584864 None] Created reservations ['4dfdff7a-93c5-48fe-bbfc-963c78f84849', '5323cab3-afd1-4246-a98d-9a40da33c488'] {{(pid=130586) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 16 21:41:13.448756 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-1d01c999-cd6a-4405-a9c6-cfdc7dd1a92e tempest-VolumesBackupsTest-493584864 None] Delete backup finished, backup 9c0f0465-f3dc-45d2-801b-939c0d169abe deleted. Apr 16 21:41:18.919826 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:41:18.920449 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:42:05.477920 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] Create backup started, backup: cf9d0423-efc6-4554-ac62-83484c45a336 volume: b419c8ec-d8d6-495e-9cec-3da461c92db9 snapshot: 4201fdce-fcae-4c9c-856b-2507c4ae1a5c. Apr 16 21:42:05.580142 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-04-16T21:42:05Z,data_timestamp=2022-04-16T21:42:01Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Backup-1939402058',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=cf9d0423-efc6-4554-ac62-83484c45a336,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='cda4f5ee41d2439e935ee0f658f465d4',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id='4201fdce-fcae-4c9c-856b-2507c4ae1a5c',status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='95bf99de7a594e209f995e62db441850',volume_id=b419c8ec-d8d6-495e-9cec-3da461c92db9) Apr 16 21:42:07.970610 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.utils [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.20', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:42:07.976833 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:42:07.980857 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.008s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:42:07.981434 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[3982cc04-3241-4391-a291-8632bf070feb]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ff85ebdbdb9\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:42:07.982617 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:42:07.988898 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 16 21:42:07.989485 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[f05987d3-f16a-4fe4-bc97-75b6209d95e5]: [Errno 2] No such file or directory: 'systool' {{(pid=154475) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 16 21:42:07.989485 ncdv-master-1 cinder-backup[130586]: Traceback (most recent call last): Apr 16 21:42:07.989485 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 16 21:42:07.989485 ncdv-master-1 cinder-backup[130586]: ret = func(*f_args, **f_kwargs) Apr 16 21:42:07.989485 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 16 21:42:07.989485 ncdv-master-1 cinder-backup[130586]: return func(*args, **kwargs) Apr 16 21:42:07.989485 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 16 21:42:07.989485 ncdv-master-1 cinder-backup[130586]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 16 21:42:07.989485 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 16 21:42:07.989485 ncdv-master-1 cinder-backup[130586]: return putils.execute(on_execute=on_execute, Apr 16 21:42:07.989485 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 16 21:42:07.989485 ncdv-master-1 cinder-backup[130586]: obj = subprocess.Popen(cmd, Apr 16 21:42:07.989485 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 16 21:42:07.989485 ncdv-master-1 cinder-backup[130586]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 16 21:42:07.989485 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 16 21:42:07.989485 ncdv-master-1 cinder-backup[130586]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 16 21:42:07.989485 ncdv-master-1 cinder-backup[130586]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 16 21:42:07.990560 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[f05987d3-f16a-4fe4-bc97-75b6209d95e5]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:42:07.990975 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:42:07.998070 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 16 21:42:07.998582 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[558a286a-46a4-44d8-9388-e60149d9bed2]: [Errno 2] No such file or directory: 'systool' {{(pid=154475) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 16 21:42:07.998582 ncdv-master-1 cinder-backup[130586]: Traceback (most recent call last): Apr 16 21:42:07.998582 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 16 21:42:07.998582 ncdv-master-1 cinder-backup[130586]: ret = func(*f_args, **f_kwargs) Apr 16 21:42:07.998582 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 16 21:42:07.998582 ncdv-master-1 cinder-backup[130586]: return func(*args, **kwargs) Apr 16 21:42:07.998582 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 16 21:42:07.998582 ncdv-master-1 cinder-backup[130586]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 16 21:42:07.998582 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 16 21:42:07.998582 ncdv-master-1 cinder-backup[130586]: return putils.execute(on_execute=on_execute, Apr 16 21:42:07.998582 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 16 21:42:07.998582 ncdv-master-1 cinder-backup[130586]: obj = subprocess.Popen(cmd, Apr 16 21:42:07.998582 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 16 21:42:07.998582 ncdv-master-1 cinder-backup[130586]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 16 21:42:07.998582 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 16 21:42:07.998582 ncdv-master-1 cinder-backup[130586]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 16 21:42:07.998582 ncdv-master-1 cinder-backup[130586]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 16 21:42:07.999655 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[558a286a-46a4-44d8-9388-e60149d9bed2]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:42:08.000650 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:42:08.010275 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.009s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:42:08.010592 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[6dea6a62-f0e5-424f-aa46-c7de803451eb]: (4, ('/dev/sda1\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:42:08.011804 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:42:08.020893 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.009s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:42:08.027323 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[a279550a-9d47-438a-bb92-6a3cda21a3aa]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:42:08.027323 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:42:08.030338 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.007s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:42:08.030656 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[315dfc8e-a16c-4a77-ac76-3de0e8a5dda1]: (4, ('efede189-2323-8540-b968-072c0c8c1315\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:42:08.031407 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] Running cmd (subprocess): nvme version {{(pid=130586) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:42:08.049873 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] CMD "nvme version" returned: 0 in 0.018s {{(pid=130586) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:42:08.051768 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=130586) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Apr 16 21:42:08.052210 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=130586) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Apr 16 21:42:08.052623 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:3a91007f-78fd-42da-ba0d-d5efe907ca61 dsc: {{(pid=130586) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Apr 16 21:42:08.053275 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.utils [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] <== get_connector_properties: return (82ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.20', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ff85ebdbdb9', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'efede189-2323-8540-b968-072c0c8c1315', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:3a91007f-78fd-42da-ba0d-d5efe907ca61', 'found_dsc': ''} {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:42:08.706771 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connector [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] Factory for rbd on None {{(pid=130586) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Apr 16 21:42:08.708304 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-de773626-a602-4242-b991-bfa71523c235', 'hosts': ['10.222.0.20'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'de773626-a602-4242-b991-bfa71523c235', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:42:08.709357 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.linuxrbd [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] opening connection to ceph cluster (timeout=-1). {{(pid=130586) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Apr 16 21:42:08.957258 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] <== connect_volume: return (248ms) {'path': } {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:42:08.965008 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_db.sqlalchemy.engines [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=130586) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 16 21:42:09.040027 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] Starting backup of volume='b419c8ec-d8d6-495e-9cec-3da461c92db9'. {{(pid=130586) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} Apr 16 21:42:09.082194 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=130586) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} Apr 16 21:42:09.084634 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] Creating backup base image='volume-b419c8ec-d8d6-495e-9cec-3da461c92db9.backup.base' for volume b419c8ec-d8d6-495e-9cec-3da461c92db9. {{(pid=130586) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:875}} Apr 16 21:42:10.140045 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] Copying data from volume b419c8ec-d8d6-495e-9cec-3da461c92db9. {{(pid=130586) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:887}} Apr 16 21:42:10.333928 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] Transferring data between 'volume-b419c8ec-d8d6-495e-9cec-3da461c92db9' and 'volume-b419c8ec-d8d6-495e-9cec-3da461c92db9.backup.base' {{(pid=130586) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:417}} Apr 16 21:42:10.334352 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] 8 chunks of 134217728 bytes to be transferred {{(pid=130586) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:421}} Apr 16 21:42:18.924927 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:42:18.925670 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:42:22.503481 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] Transferred chunk 1 of 8 (10771K/s) {{(pid=130586) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} Apr 16 21:42:33.886653 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] Transferred chunk 2 of 8 (11515K/s) {{(pid=130586) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} Apr 16 21:42:45.295298 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] Transferred chunk 3 of 8 (11490K/s) {{(pid=130586) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} Apr 16 21:42:56.374007 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] Transferred chunk 4 of 8 (11831K/s) {{(pid=130586) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} Apr 16 21:43:08.179227 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] Transferred chunk 5 of 8 (11104K/s) {{(pid=130586) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} Apr 16 21:43:16.324539 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-e2bbab43-2d8b-45fc-a379-f2163f0073a2 tempest-TestVolumeBackupRestore-1675082881 None] Delete backup started, backup: ab06f9fe-c532-404f-b0e9-d1e0343b39c8. Apr 16 21:43:16.330300 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-e2bbab43-2d8b-45fc-a379-f2163f0073a2 tempest-TestVolumeBackupRestore-1675082881 None] Delete started for backup=ab06f9fe-c532-404f-b0e9-d1e0343b39c8 {{(pid=130586) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} Apr 16 21:43:16.330811 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-e2bbab43-2d8b-45fc-a379-f2163f0073a2 tempest-TestVolumeBackupRestore-1675082881 None] Trying diff format basename='volume-2a52b0c3-1018-488f-bff4-8d3025861720.backup.ab06f9fe-c532-404f-b0e9-d1e0343b39c8' for backup base image of volume 2a52b0c3-1018-488f-bff4-8d3025861720. {{(pid=130586) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} Apr 16 21:43:16.976636 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-e2bbab43-2d8b-45fc-a379-f2163f0073a2 tempest-TestVolumeBackupRestore-1675082881 None] Looking for snapshot of backup base 'volume-2a52b0c3-1018-488f-bff4-8d3025861720.backup.ab06f9fe-c532-404f-b0e9-d1e0343b39c8' {{(pid=130586) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} Apr 16 21:43:16.976943 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-e2bbab43-2d8b-45fc-a379-f2163f0073a2 tempest-TestVolumeBackupRestore-1675082881 None] Found snapshot 'backup.ab06f9fe-c532-404f-b0e9-d1e0343b39c8.snap.1650145082.8901744' {{(pid=130586) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} Apr 16 21:43:16.977515 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-e2bbab43-2d8b-45fc-a379-f2163f0073a2 tempest-TestVolumeBackupRestore-1675082881 None] Deleting backup snapshot='backup.ab06f9fe-c532-404f-b0e9-d1e0343b39c8.snap.1650145082.8901744' {{(pid=130586) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}} Apr 16 21:43:18.921695 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:43:18.922403 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:43:19.933762 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] Transferred chunk 6 of 8 (11151K/s) {{(pid=130586) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} Apr 16 21:43:22.107186 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.drivers.ceph [None req-e2bbab43-2d8b-45fc-a379-f2163f0073a2 tempest-TestVolumeBackupRestore-1675082881 None] Deleting backup base image='volume-2a52b0c3-1018-488f-bff4-8d3025861720.backup.ab06f9fe-c532-404f-b0e9-d1e0343b39c8' of volume 2a52b0c3-1018-488f-bff4-8d3025861720. Apr 16 21:43:24.060771 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_db.sqlalchemy.engines [None req-61e8a934-990b-4a99-9b63-11946b0b3861 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=130586) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 16 21:43:30.524095 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] Transferred chunk 7 of 8 (12377K/s) {{(pid=130586) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} Apr 16 21:43:39.196556 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-e2bbab43-2d8b-45fc-a379-f2163f0073a2 tempest-TestVolumeBackupRestore-1675082881 None] Base backup image='volume-2a52b0c3-1018-488f-bff4-8d3025861720.backup.ab06f9fe-c532-404f-b0e9-d1e0343b39c8' of volume 2a52b0c3-1018-488f-bff4-8d3025861720 deleted. {{(pid=130586) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} Apr 16 21:43:39.536243 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-e2bbab43-2d8b-45fc-a379-f2163f0073a2 tempest-TestVolumeBackupRestore-1675082881 None] Delete of backup 'ab06f9fe-c532-404f-b0e9-d1e0343b39c8' for volume '2a52b0c3-1018-488f-bff4-8d3025861720' finished. {{(pid=130586) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} Apr 16 21:43:39.671064 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.quota [None req-e2bbab43-2d8b-45fc-a379-f2163f0073a2 tempest-TestVolumeBackupRestore-1675082881 None] Created reservations ['5ae724a7-11d5-4c77-8d6a-be2bf0aa44fd', '9d742bdf-ff11-452d-9d20-dcb4da50b41f'] {{(pid=130586) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 16 21:43:39.847288 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-e2bbab43-2d8b-45fc-a379-f2163f0073a2 tempest-TestVolumeBackupRestore-1675082881 None] Delete backup finished, backup ab06f9fe-c532-404f-b0e9-d1e0343b39c8 deleted. Apr 16 21:43:42.530720 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] Transferred chunk 8 of 8 (10917K/s) {{(pid=130586) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} Apr 16 21:43:43.277336 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] Getting metadata type 'volume-base-metadata' {{(pid=130586) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Apr 16 21:43:43.288774 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=130586) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Apr 16 21:43:43.289222 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] Getting metadata type 'volume-metadata' {{(pid=130586) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Apr 16 21:43:43.295630 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] No metadata type 'volume-metadata' available {{(pid=130586) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Apr 16 21:43:43.296019 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] Getting metadata type 'volume-glance-metadata' {{(pid=130586) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Apr 16 21:43:43.302322 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] No metadata type 'volume-glance-metadata' available {{(pid=130586) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Apr 16 21:43:43.302879 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] Backing up metadata for volume b419c8ec-d8d6-495e-9cec-3da461c92db9. {{(pid=130586) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} Apr 16 21:43:43.679187 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] Backup 'cf9d0423-efc6-4554-ac62-83484c45a336' of volume b419c8ec-d8d6-495e-9cec-3da461c92db9 finished. {{(pid=130586) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} Apr 16 21:43:43.680828 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-de773626-a602-4242-b991-bfa71523c235', 'hosts': ['10.222.0.20'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'de773626-a602-4242-b991-bfa71523c235', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:43:43.793937 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] <== disconnect_volume: return (112ms) None {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:43:45.293042 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-897539d2-32f9-4295-b94c-259fce1118d3 tempest-VolumesSnapshotTestJSON-2078807312 None] Create backup finished. backup: cf9d0423-efc6-4554-ac62-83484c45a336. Apr 16 21:43:46.362304 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-269da992-a29a-47cc-832a-aeeccdf416fa tempest-VolumesSnapshotTestJSON-2078807312 None] Delete backup started, backup: cf9d0423-efc6-4554-ac62-83484c45a336. Apr 16 21:43:46.363230 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-269da992-a29a-47cc-832a-aeeccdf416fa tempest-VolumesSnapshotTestJSON-2078807312 None] Delete started for backup=cf9d0423-efc6-4554-ac62-83484c45a336 {{(pid=130586) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} Apr 16 21:43:46.363729 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-269da992-a29a-47cc-832a-aeeccdf416fa tempest-VolumesSnapshotTestJSON-2078807312 None] Trying diff format basename='volume-b419c8ec-d8d6-495e-9cec-3da461c92db9.backup.cf9d0423-efc6-4554-ac62-83484c45a336' for backup base image of volume b419c8ec-d8d6-495e-9cec-3da461c92db9. {{(pid=130586) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} Apr 16 21:43:46.398761 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-269da992-a29a-47cc-832a-aeeccdf416fa tempest-VolumesSnapshotTestJSON-2078807312 None] Image 'volume-b419c8ec-d8d6-495e-9cec-3da461c92db9.backup.cf9d0423-efc6-4554-ac62-83484c45a336' not found - trying diff format name {{(pid=130586) _rbd_image_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:693}} Apr 16 21:43:46.711621 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-269da992-a29a-47cc-832a-aeeccdf416fa tempest-VolumesSnapshotTestJSON-2078807312 None] Looking for snapshot of backup base 'volume-b419c8ec-d8d6-495e-9cec-3da461c92db9.backup.base' {{(pid=130586) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} Apr 16 21:43:46.711931 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-269da992-a29a-47cc-832a-aeeccdf416fa tempest-VolumesSnapshotTestJSON-2078807312 None] Backup base 'volume-b419c8ec-d8d6-495e-9cec-3da461c92db9.backup.base' has no snapshots {{(pid=130586) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:955}} Apr 16 21:43:46.712339 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-269da992-a29a-47cc-832a-aeeccdf416fa tempest-VolumesSnapshotTestJSON-2078807312 None] No backup snapshot to delete {{(pid=130586) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:500}} Apr 16 21:43:46.953425 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.drivers.ceph [None req-269da992-a29a-47cc-832a-aeeccdf416fa tempest-VolumesSnapshotTestJSON-2078807312 None] Deleting backup base image='volume-b419c8ec-d8d6-495e-9cec-3da461c92db9.backup.base' of volume b419c8ec-d8d6-495e-9cec-3da461c92db9. Apr 16 21:43:54.058737 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_db.sqlalchemy.engines [None req-61e8a934-990b-4a99-9b63-11946b0b3861 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=130586) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 16 21:44:00.433275 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-269da992-a29a-47cc-832a-aeeccdf416fa tempest-VolumesSnapshotTestJSON-2078807312 None] Base backup image='volume-b419c8ec-d8d6-495e-9cec-3da461c92db9.backup.base' of volume b419c8ec-d8d6-495e-9cec-3da461c92db9 deleted. {{(pid=130586) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} Apr 16 21:44:00.592443 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-269da992-a29a-47cc-832a-aeeccdf416fa tempest-VolumesSnapshotTestJSON-2078807312 None] Delete of backup 'cf9d0423-efc6-4554-ac62-83484c45a336' for volume 'b419c8ec-d8d6-495e-9cec-3da461c92db9' finished. {{(pid=130586) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} Apr 16 21:44:00.668869 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.quota [None req-269da992-a29a-47cc-832a-aeeccdf416fa tempest-VolumesSnapshotTestJSON-2078807312 None] Created reservations ['1b47a7f9-e5a2-435a-b3fe-e2eb0502e399', 'eb6caa11-7a0f-498d-a664-f67388ecf7f3'] {{(pid=130586) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 16 21:44:00.846214 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-269da992-a29a-47cc-832a-aeeccdf416fa tempest-VolumesSnapshotTestJSON-2078807312 None] Delete backup finished, backup cf9d0423-efc6-4554-ac62-83484c45a336 deleted. Apr 16 21:44:18.918687 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:44:18.919309 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:45:18.923369 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:45:18.924043 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:46:18.927782 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:46:18.928420 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:47:18.933602 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:47:18.934275 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:48:18.937991 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:48:18.938729 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:48:24.602892 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] Create backup started, backup: d25e3166-72cc-45f7-9af4-7f91b9cbcd3f volume: 277643c8-e229-4a42-9dab-8ebf5d0634e2. Apr 16 21:48:24.671369 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-04-16T21:48:24Z,data_timestamp=2022-04-16T21:48:24Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsV39Test-Backup-281360062',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=d25e3166-72cc-45f7-9af4-7f91b9cbcd3f,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='475ffde0ba2f432b918f959d413bb874',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='de128399333c4a2481f61c2f0b5f310b',volume_id=277643c8-e229-4a42-9dab-8ebf5d0634e2) Apr 16 21:48:24.711911 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.utils [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.20', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:48:24.713327 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:48:24.721116 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.008s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:48:24.721442 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[e87eb031-f4db-4ced-833c-4162d9384e3f]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ff85ebdbdb9\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:48:24.727292 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:48:24.729096 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 16 21:48:24.730847 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[a572a56e-22da-4d84-8185-b2c13ba34de3]: [Errno 2] No such file or directory: 'systool' {{(pid=154475) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 16 21:48:24.730847 ncdv-master-1 cinder-backup[130586]: Traceback (most recent call last): Apr 16 21:48:24.730847 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 16 21:48:24.730847 ncdv-master-1 cinder-backup[130586]: ret = func(*f_args, **f_kwargs) Apr 16 21:48:24.730847 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 16 21:48:24.730847 ncdv-master-1 cinder-backup[130586]: return func(*args, **kwargs) Apr 16 21:48:24.730847 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 16 21:48:24.730847 ncdv-master-1 cinder-backup[130586]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 16 21:48:24.730847 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 16 21:48:24.730847 ncdv-master-1 cinder-backup[130586]: return putils.execute(on_execute=on_execute, Apr 16 21:48:24.730847 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 16 21:48:24.730847 ncdv-master-1 cinder-backup[130586]: obj = subprocess.Popen(cmd, Apr 16 21:48:24.730847 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 16 21:48:24.730847 ncdv-master-1 cinder-backup[130586]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 16 21:48:24.730847 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 16 21:48:24.730847 ncdv-master-1 cinder-backup[130586]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 16 21:48:24.730847 ncdv-master-1 cinder-backup[130586]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 16 21:48:24.732094 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[a572a56e-22da-4d84-8185-b2c13ba34de3]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:48:24.732474 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:48:24.738843 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Apr 16 21:48:24.739431 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[28705e20-a380-49bf-9cf5-dda354e7dd67]: [Errno 2] No such file or directory: 'systool' {{(pid=154475) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Apr 16 21:48:24.739431 ncdv-master-1 cinder-backup[130586]: Traceback (most recent call last): Apr 16 21:48:24.739431 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Apr 16 21:48:24.739431 ncdv-master-1 cinder-backup[130586]: ret = func(*f_args, **f_kwargs) Apr 16 21:48:24.739431 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Apr 16 21:48:24.739431 ncdv-master-1 cinder-backup[130586]: return func(*args, **kwargs) Apr 16 21:48:24.739431 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Apr 16 21:48:24.739431 ncdv-master-1 cinder-backup[130586]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Apr 16 21:48:24.739431 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Apr 16 21:48:24.739431 ncdv-master-1 cinder-backup[130586]: return putils.execute(on_execute=on_execute, Apr 16 21:48:24.739431 ncdv-master-1 cinder-backup[130586]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Apr 16 21:48:24.739431 ncdv-master-1 cinder-backup[130586]: obj = subprocess.Popen(cmd, Apr 16 21:48:24.739431 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Apr 16 21:48:24.739431 ncdv-master-1 cinder-backup[130586]: self._execute_child(args, executable, preexec_fn, close_fds, Apr 16 21:48:24.739431 ncdv-master-1 cinder-backup[130586]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Apr 16 21:48:24.739431 ncdv-master-1 cinder-backup[130586]: raise child_exception_type(errno_num, err_msg, err_filename) Apr 16 21:48:24.739431 ncdv-master-1 cinder-backup[130586]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Apr 16 21:48:24.740859 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[28705e20-a380-49bf-9cf5-dda354e7dd67]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:48:24.741145 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:48:24.749763 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.008s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:48:24.750153 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[d3cd53e2-dfd1-4059-93ac-b6c46e90e37e]: (4, ('/dev/sda1\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:48:24.751617 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:48:24.761909 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.010s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:48:24.762331 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[fef357cc-6f19-4602-b24b-599f9a07c749]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:48:24.763395 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:48:24.770278 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.007s {{(pid=154475) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:48:24.773137 ncdv-master-1 cinder-backup[130586]: DEBUG oslo.privsep.daemon [-] privsep: reply[54df0086-b1ab-4287-bf09-ec9baf73b8c1]: (4, ('efede189-2323-8540-b968-072c0c8c1315\n', '')) {{(pid=154475) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Apr 16 21:48:24.773137 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] Running cmd (subprocess): nvme version {{(pid=130586) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Apr 16 21:48:24.791292 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_concurrency.processutils [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] CMD "nvme version" returned: 0 in 0.020s {{(pid=130586) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Apr 16 21:48:24.793097 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=130586) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Apr 16 21:48:24.793484 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=130586) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Apr 16 21:48:24.793870 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.lightos [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:3a91007f-78fd-42da-ba0d-d5efe907ca61 dsc: {{(pid=130586) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Apr 16 21:48:24.794526 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.utils [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] <== get_connector_properties: return (82ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.20', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ff85ebdbdb9', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'efede189-2323-8540-b968-072c0c8c1315', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:3a91007f-78fd-42da-ba0d-d5efe907ca61', 'found_dsc': ''} {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:48:25.455617 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connector [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] Factory for rbd on None {{(pid=130586) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Apr 16 21:48:25.457140 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-277643c8-e229-4a42-9dab-8ebf5d0634e2', 'hosts': ['10.222.0.20'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '277643c8-e229-4a42-9dab-8ebf5d0634e2', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:48:25.458215 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.linuxrbd [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] opening connection to ceph cluster (timeout=-1). {{(pid=130586) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Apr 16 21:48:25.716801 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] <== connect_volume: return (258ms) {'path': } {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:48:25.725395 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_db.sqlalchemy.engines [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=130586) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 16 21:48:25.842108 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] Starting backup of volume='277643c8-e229-4a42-9dab-8ebf5d0634e2'. {{(pid=130586) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} Apr 16 21:48:25.843046 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] Volume file is RBD: attempting optimized backup {{(pid=130586) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}} Apr 16 21:48:25.864068 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] Creating base image 'volume-277643c8-e229-4a42-9dab-8ebf5d0634e2.backup.d25e3166-72cc-45f7-9af4-7f91b9cbcd3f' {{(pid=130586) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}} Apr 16 21:48:25.869792 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=130586) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} Apr 16 21:48:26.965689 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] Using --from-snap 'None' for incremental backup of volume 277643c8-e229-4a42-9dab-8ebf5d0634e2. {{(pid=130586) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}} Apr 16 21:48:26.966341 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] Creating backup snapshot='backup.d25e3166-72cc-45f7-9af4-7f91b9cbcd3f.snap.1650145706.9657145' {{(pid=130586) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}} Apr 16 21:48:29.802967 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] Performing differential transfer from 'volume-277643c8-e229-4a42-9dab-8ebf5d0634e2' to 'volume-277643c8-e229-4a42-9dab-8ebf5d0634e2.backup.d25e3166-72cc-45f7-9af4-7f91b9cbcd3f' {{(pid=130586) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} Apr 16 21:48:29.802967 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_lln__nbr --pool volumes volumes/volume-277643c8-e229-4a42-9dab-8ebf5d0634e2@backup.d25e3166-72cc-45f7-9af4-7f91b9cbcd3f.snap.1650145706.9657145 -' into... {{(pid=130586) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} Apr 16 21:48:29.804070 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-277643c8-e229-4a42-9dab-8ebf5d0634e2.backup.d25e3166-72cc-45f7-9af4-7f91b9cbcd3f' {{(pid=130586) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} Apr 16 21:48:32.598477 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] Differential backup transfer completed in 2.7956s {{(pid=130586) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}} Apr 16 21:48:32.599171 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] Getting metadata type 'volume-base-metadata' {{(pid=130586) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Apr 16 21:48:32.613218 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=130586) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Apr 16 21:48:32.613626 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] Getting metadata type 'volume-metadata' {{(pid=130586) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Apr 16 21:48:32.620783 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] No metadata type 'volume-metadata' available {{(pid=130586) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Apr 16 21:48:32.625998 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] Getting metadata type 'volume-glance-metadata' {{(pid=130586) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Apr 16 21:48:32.627280 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.driver [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] No metadata type 'volume-glance-metadata' available {{(pid=130586) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Apr 16 21:48:32.627839 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] Backing up metadata for volume 277643c8-e229-4a42-9dab-8ebf5d0634e2. {{(pid=130586) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} Apr 16 21:48:32.750188 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] Backup 'd25e3166-72cc-45f7-9af4-7f91b9cbcd3f' of volume 277643c8-e229-4a42-9dab-8ebf5d0634e2 finished. {{(pid=130586) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} Apr 16 21:48:32.752086 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-277643c8-e229-4a42-9dab-8ebf5d0634e2', 'hosts': ['10.222.0.20'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '277643c8-e229-4a42-9dab-8ebf5d0634e2', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Apr 16 21:48:33.704323 ncdv-master-1 cinder-backup[130586]: DEBUG os_brick.initiator.connectors.rbd [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] <== disconnect_volume: return (950ms) None {{(pid=130586) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Apr 16 21:48:33.882509 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-1e172218-4f9f-4338-bea6-1b90048f21e7 tempest-VolumesBackupsV39Test-2016061415 None] Create backup finished. backup: d25e3166-72cc-45f7-9af4-7f91b9cbcd3f. Apr 16 21:48:35.134241 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-0a03c49a-9fc0-449e-ac2e-e5512b95d92a tempest-VolumesBackupsV39Test-2016061415 None] Delete backup started, backup: d25e3166-72cc-45f7-9af4-7f91b9cbcd3f. Apr 16 21:48:35.136361 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-0a03c49a-9fc0-449e-ac2e-e5512b95d92a tempest-VolumesBackupsV39Test-2016061415 None] Delete started for backup=d25e3166-72cc-45f7-9af4-7f91b9cbcd3f {{(pid=130586) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} Apr 16 21:48:35.136882 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-0a03c49a-9fc0-449e-ac2e-e5512b95d92a tempest-VolumesBackupsV39Test-2016061415 None] Trying diff format basename='volume-277643c8-e229-4a42-9dab-8ebf5d0634e2.backup.d25e3166-72cc-45f7-9af4-7f91b9cbcd3f' for backup base image of volume 277643c8-e229-4a42-9dab-8ebf5d0634e2. {{(pid=130586) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} Apr 16 21:48:35.385074 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-0a03c49a-9fc0-449e-ac2e-e5512b95d92a tempest-VolumesBackupsV39Test-2016061415 None] Looking for snapshot of backup base 'volume-277643c8-e229-4a42-9dab-8ebf5d0634e2.backup.d25e3166-72cc-45f7-9af4-7f91b9cbcd3f' {{(pid=130586) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} Apr 16 21:48:35.390719 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-0a03c49a-9fc0-449e-ac2e-e5512b95d92a tempest-VolumesBackupsV39Test-2016061415 None] Found snapshot 'backup.d25e3166-72cc-45f7-9af4-7f91b9cbcd3f.snap.1650145706.9657145' {{(pid=130586) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} Apr 16 21:48:35.390719 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-0a03c49a-9fc0-449e-ac2e-e5512b95d92a tempest-VolumesBackupsV39Test-2016061415 None] Deleting backup snapshot='backup.d25e3166-72cc-45f7-9af4-7f91b9cbcd3f.snap.1650145706.9657145' {{(pid=130586) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}} Apr 16 21:48:38.608576 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.drivers.ceph [None req-0a03c49a-9fc0-449e-ac2e-e5512b95d92a tempest-VolumesBackupsV39Test-2016061415 None] Deleting backup base image='volume-277643c8-e229-4a42-9dab-8ebf5d0634e2.backup.d25e3166-72cc-45f7-9af4-7f91b9cbcd3f' of volume 277643c8-e229-4a42-9dab-8ebf5d0634e2. Apr 16 21:48:44.086789 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_db.sqlalchemy.engines [None req-61e8a934-990b-4a99-9b63-11946b0b3861 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=130586) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Apr 16 21:48:45.150944 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-0a03c49a-9fc0-449e-ac2e-e5512b95d92a tempest-VolumesBackupsV39Test-2016061415 None] Base backup image='volume-277643c8-e229-4a42-9dab-8ebf5d0634e2.backup.d25e3166-72cc-45f7-9af4-7f91b9cbcd3f' of volume 277643c8-e229-4a42-9dab-8ebf5d0634e2 deleted. {{(pid=130586) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} Apr 16 21:48:45.377461 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.backup.drivers.ceph [None req-0a03c49a-9fc0-449e-ac2e-e5512b95d92a tempest-VolumesBackupsV39Test-2016061415 None] Delete of backup 'd25e3166-72cc-45f7-9af4-7f91b9cbcd3f' for volume '277643c8-e229-4a42-9dab-8ebf5d0634e2' finished. {{(pid=130586) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} Apr 16 21:48:45.503850 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.quota [None req-0a03c49a-9fc0-449e-ac2e-e5512b95d92a tempest-VolumesBackupsV39Test-2016061415 None] Created reservations ['3ea86215-b535-4acf-8620-d0390526071a', 'e35598c3-563d-41cd-9912-35d37fbb106e'] {{(pid=130586) reserve /opt/stack/cinder/cinder/quota.py:755}} Apr 16 21:48:45.715635 ncdv-master-1 cinder-backup[130586]: INFO cinder.backup.manager [None req-0a03c49a-9fc0-449e-ac2e-e5512b95d92a tempest-VolumesBackupsV39Test-2016061415 None] Delete backup finished, backup d25e3166-72cc-45f7-9af4-7f91b9cbcd3f deleted. Apr 16 21:49:18.943343 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:49:18.944112 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:50:18.948940 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:50:18.950278 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:51:18.946301 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:51:18.946943 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:52:18.951096 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:52:18.951808 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:53:18.956374 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:53:18.957068 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:54:18.961814 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:54:18.962471 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:56:18.960916 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:56:18.961613 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:57:18.965602 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:57:18.966275 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:58:18.961780 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:58:18.962403 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 21:59:18.967048 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 21:59:18.967710 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:00:18.972661 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:00:18.973308 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:01:18.978724 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:01:18.979445 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:02:18.984060 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:02:18.984885 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:03:18.989410 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:03:18.990100 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:04:18.993887 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:04:18.994569 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:05:18.990243 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:05:18.990903 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:06:18.995018 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:06:18.995613 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:07:19.000409 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:07:19.000981 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:08:18.997338 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:08:18.997918 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:09:18.995950 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:09:18.996596 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:10:18.992964 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:10:18.993633 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:11:18.998138 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:11:18.998754 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:12:19.003149 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:12:19.003757 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:13:19.001034 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:13:19.001654 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:14:19.005751 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:14:19.006375 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:15:19.011566 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:15:19.012224 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:16:19.016630 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:16:19.017260 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:17:19.012963 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:17:19.013581 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:18:19.017569 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:18:19.018183 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:19:19.022350 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:19:19.022951 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:20:19.017730 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:20:19.018354 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:21:19.022551 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:21:19.023232 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:22:19.020763 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:22:19.021409 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:23:19.016962 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:23:19.017592 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:24:19.013542 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:24:19.014149 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:25:19.018743 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:25:19.019397 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:26:19.023825 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:26:19.024473 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:27:19.028420 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:27:19.029038 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:28:19.033233 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:28:19.033888 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:29:19.030357 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:29:19.031012 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:30:19.035807 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:30:19.036391 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:31:19.041584 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:31:19.042209 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:32:19.046154 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:32:19.046789 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:33:19.050963 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:33:19.051855 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:34:19.056094 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:34:19.056884 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:35:19.060650 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:35:19.061280 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:36:19.057525 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:36:19.058654 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Apr 16 22:37:19.062627 ncdv-master-1 cinder-backup[130586]: DEBUG oslo_service.periodic_task [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=130586) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Apr 16 22:37:19.063265 ncdv-master-1 cinder-backup[130586]: DEBUG cinder.manager [None req-fda8594d-3ddc-4f40-a1d9-ab2315b6dfed None None] Notifying Schedulers of capabilities ... {{(pid=130586) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}