-- Logs begin at Sat 2022-03-05 20:01:15 UTC, end at Sat 2022-03-05 22:06:02 UTC. -- Mar 05 20:39:35.795482 ncdv-master-1 systemd[1]: Started Devstack devstack@c-bak.service. Mar 05 20:39:39.488595 ncdv-master-1 cinder-backup[134217]: INFO cinder.cmd.backup [-] Backup running with 1 processes. Mar 05 20:39:39.490381 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_concurrency.lockutils [-] Acquired lock "singleton_lock" {{(pid=134217) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:294}} Mar 05 20:39:39.490483 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_concurrency.lockutils [-] Releasing lock "singleton_lock" {{(pid=134217) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:312}} Mar 05 20:39:39.674822 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_db.sqlalchemy.engines [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134217) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 05 20:39:39.891582 ncdv-master-1 cinder-backup[134217]: INFO cinder.rpc [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] Automatically selected cinder-backup objects version 1.39 as minimum service version. Mar 05 20:39:39.895436 ncdv-master-1 cinder-backup[134217]: INFO cinder.rpc [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] Automatically selected cinder-backup RPC version 2.3 as minimum service version. Mar 05 20:39:39.899412 ncdv-master-1 cinder-backup[134217]: INFO cinder.rpc [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] Automatically selected cinder-volume objects version 1.39 as minimum service version. Mar 05 20:39:39.903171 ncdv-master-1 cinder-backup[134217]: INFO cinder.rpc [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] Automatically selected cinder-volume RPC version 3.18 as minimum service version. Mar 05 20:39:39.919695 ncdv-master-1 cinder-backup[134217]: INFO cinder.rpc [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] Automatically selected cinder-scheduler objects version 1.39 as minimum service version. Mar 05 20:39:39.919695 ncdv-master-1 cinder-backup[134217]: INFO cinder.rpc [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] Automatically selected cinder-scheduler RPC version 3.12 as minimum service version. Mar 05 20:39:39.962118 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_db.sqlalchemy.engines [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134217) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 05 20:39:39.991695 ncdv-master-1 cinder-backup[134217]: INFO oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] Starting 1 workers Mar 05 20:39:39.991899 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] Started child 134663 {{(pid=134217) _start_child /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:575}} Mar 05 20:39:39.993092 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] Full set of CONF: {{(pid=134217) wait /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:649}} Mar 05 20:39:39.993647 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] ******************************************************************************** {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Mar 05 20:39:39.993961 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] Configuration options gathered from: {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Mar 05 20:39:39.994327 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] command line args: ['--config-file', '/etc/cinder/cinder.conf'] {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Mar 05 20:39:40.002174 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] config files: ['/etc/cinder/cinder.conf'] {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Mar 05 20:39:40.002174 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] ================================================================================ {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Mar 05 20:39:40.002174 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] allow_availability_zone_fallback = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.002174 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] allow_compression_on_image_upload = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.002174 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] allowed_direct_url_schemes = [] {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.002174 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] api_rate_limit = True {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.002751 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] auth_strategy = keystone {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.002751 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_availability_zone = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.002751 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backup_api_class = cinder.backup.api.API {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.002751 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backup_ceph_chunk_size = 134217728 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.003595 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backup_ceph_conf = /etc/ceph/ceph.conf {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.004068 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backup_ceph_image_journals = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.004468 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backup_ceph_pool = backups {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.004858 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backup_ceph_stripe_count = 0 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.005944 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backup_ceph_stripe_unit = 0 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.006419 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backup_ceph_user = cinder-bak {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.006808 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backup_driver = cinder.backup.drivers.ceph.CephBackupDriver {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.007191 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backup_driver_init_check_interval = 60 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.019715 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backup_driver_stats_polling_interval = 60 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.019715 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backup_manager = cinder.backup.manager.BackupManager {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.019715 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backup_max_operations = 15 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.019715 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backup_metadata_version = 2 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.019715 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backup_name_template = backup-%s {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.019715 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backup_native_threads_pool_size = 60 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.020286 ncdv-master-1 cinder-backup[134663]: INFO cinder.service [-] Starting cinder-backup node (version 19.1.0) Mar 05 20:39:40.021542 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backup_object_number_per_notification = 10 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.021542 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backup_service_inithost_offload = True {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.021542 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backup_timer_interval = 120 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.021542 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backup_use_same_host = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.021542 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backup_use_temp_snapshot = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.021542 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backup_workers = 1 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.021542 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] chap_password = **** {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.022121 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] chap_username = {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.022121 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.023830 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cinder_internal_tenant_project_id = 87292ea496944da4a7e67ecf8e0eaa0c {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.024287 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cinder_internal_tenant_user_id = 2cb0fb16af604f8cb64089e3a17a244b {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.025702 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] compression_format = gzip {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.026208 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] config_dir = [] {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.026539 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] config_file = ['/etc/cinder/cinder.conf'] {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.027899 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] config_source = [] {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.028288 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] consistencygroup_api_class = cinder.consistencygroup.api.API {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.035719 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] control_exchange = openstack {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.035719 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] debug = True {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.035719 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] default_availability_zone = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.035719 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] default_group_type = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.035719 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'swiftclient=WARN', 'botocore=WARN'] {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.036288 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] default_volume_type = ceph {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.036970 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] driver_client_cert = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.038249 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] driver_client_cert_key = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.039503 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] driver_data_namespace = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.040213 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] driver_ssl_cert_path = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.040606 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] driver_ssl_cert_verify = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.040993 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] driver_use_ssl = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.043271 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] enable_new_services = True {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.043700 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] enable_unsupported_driver = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.051836 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] enabled_backends = ['ceph'] {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.051836 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] enforce_multipath_for_image_xfer = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.051836 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] filter_function = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.051836 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] glance_api_insecure = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.051836 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] glance_api_servers = ['http://10.222.0.15/image'] {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.051836 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] glance_api_ssl_compression = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.052456 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] glance_ca_certificates_file = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.052456 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] glance_catalog_info = image:glance:publicURL {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.052456 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] glance_certfile = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.052456 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] glance_core_properties = ['checksum', 'container_format', 'disk_format', 'image_name', 'image_id', 'min_disk', 'min_ram', 'name', 'size'] {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.054577 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] glance_keyfile = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.054998 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] glance_num_retries = 3 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.055367 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] glance_request_timeout = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.062894 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] goodness_function = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.062894 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] graceful_shutdown_timeout = 5 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.062894 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] group_api_class = cinder.group.api.API {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.062894 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] host = ncdv-master-1 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.062894 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] iet_conf = /etc/iet/ietd.conf {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.062894 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] image_compress_on_upload = True {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.063544 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] image_conversion_address_space_limit = 1 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.063544 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] image_conversion_cpu_limit = 60 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.063544 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] image_conversion_dir = /opt/stack/data/cinder/conversion {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.063544 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] image_upload_use_cinder_backend = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.063544 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] image_upload_use_internal_tenant = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.063544 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] image_volume_cache_enabled = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.064025 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] image_volume_cache_max_count = 0 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.064453 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] image_volume_cache_max_size_gb = 0 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.064829 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] instance_format = [instance: %(uuid)s] {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.065197 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] instance_uuid_format = [instance: %(uuid)s] {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.065581 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] iscsi_iotype = fileio {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.066196 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] iscsi_secondary_ip_addresses = [] {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.079693 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] iscsi_target_flags = {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.079693 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] iscsi_write_cache = on {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.079693 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] iser_helper = tgtadm {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.079693 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] iser_ip_address = 10.222.0.15 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.080122 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.lockutils [None req-f2a0b876-efda-41dc-ae1a-b3429f4644ac None None] Lock "cleanup_incomplete_backups_134217" acquired by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: waited 0.003s {{(pid=134663) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:386}} Mar 05 20:39:40.080217 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] iser_port = 3260 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.080217 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.080217 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] log_config_append = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.080217 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.080217 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] log_dir = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.080217 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] log_file = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.080719 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] log_options = True {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.080719 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] log_rotate_interval = 1 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.080719 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] log_rotate_interval_type = days {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.080719 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] log_rotation_type = none {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.080719 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.080719 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.081183 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.081265 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-f2a0b876-efda-41dc-ae1a-b3429f4644ac None None] Cleaning up incomplete backup operations. Mar 05 20:39:40.084702 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.085034 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] logging_user_identity_format = %(user)s %(tenant)s %(domain)s %(user_domain)s %(project_domain)s {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.085436 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] max_age = 0 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.085798 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] max_logfile_count = 30 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.086156 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] max_logfile_size_mb = 200 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.086552 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] max_over_subscription_ratio = 20.0 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.086937 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] message_reap_interval = 86400 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.087310 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] message_ttl = 2592000 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.087656 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] monkey_patch = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.088046 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] monkey_patch_modules = [] {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.088337 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] my_ip = 10.222.0.15 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.088689 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] no_snapshot_gb_quota = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.099695 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] num_iser_scan_tries = 3 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.099695 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] num_shell_tries = 3 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.099695 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] num_volume_device_scan_tries = 3 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.099695 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] nvmet_ns_id = 10 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.099695 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] nvmet_port_id = 1 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.100183 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_db.sqlalchemy.engines [None req-f2a0b876-efda-41dc-ae1a-b3429f4644ac None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134663) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 05 20:39:40.100764 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] osapi_max_limit = 1000 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.101201 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] osapi_volume_ext_list = [] {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.101641 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] osapi_volume_extension = ['cinder.api.contrib.standard_extensions'] {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.102017 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] osapi_volume_listen = 0.0.0.0 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.102395 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] osapi_volume_listen_port = 8776 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.102747 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] osapi_volume_use_ssl = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.116123 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] osapi_volume_workers = 2 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.116613 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] per_volume_size_limit = -1 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.116988 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] periodic_fuzzy_delay = 60 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.117375 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] periodic_interval = 60 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.117802 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] project_id_regex = [0-9a-f\-]+ {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.118148 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] public_endpoint = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.118505 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] publish_errors = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.118853 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] quota_backup_gigabytes = 1000 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.119222 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] quota_backups = 10 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.119573 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] quota_consistencygroups = 10 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.134803 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] quota_driver = cinder.quota.DbQuotaDriver {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.135439 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] quota_gigabytes = 1000 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.135996 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] quota_groups = 10 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.136563 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] quota_snapshots = 10 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.141527 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] quota_volumes = 10 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.142056 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] rate_limit_burst = 0 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.142599 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] rate_limit_except_level = CRITICAL {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.143120 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] rate_limit_interval = 0 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.143653 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] replication_device = **** {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.144216 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] report_discard_supported = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.144823 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] report_interval = 10 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.145429 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] reservation_clean_interval = 86400 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.146408 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] reservation_expire = 86400 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.146985 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] reserved_percentage = 0 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.151698 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] resource_query_filters_file = /etc/cinder/resource_filters.json {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.152278 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] restore_discard_excess_bytes = True {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.152806 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] rootwrap_config = /etc/cinder/rootwrap.conf {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.153349 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] rpc_response_timeout = 120 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.153871 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] run_external_periodic_tasks = True {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.154396 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] scheduler_manager = cinder.scheduler.manager.SchedulerManager {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.155028 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] scst_target_driver = iscsi {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.155532 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] scst_target_iqn_name = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.156007 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] service_down_time = 60 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.156513 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] snapshot_name_template = snapshot-%s {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.159052 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] split_loggers = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.159530 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] state_path = /opt/stack/data/cinder {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.160096 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] storage_availability_zone = nova {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.160627 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] storage_protocol = iscsi {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.162540 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] syslog_log_facility = LOG_USER {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.165151 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] target_helper = lioadm {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.165818 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] target_ip_address = 10.222.0.15 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.166608 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] target_port = 3260 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.167137 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] target_prefix = iqn.2010-10.org.openstack: {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.171803 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] target_protocol = iscsi {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.172536 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] trace_flags = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.173085 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] transfer_api_class = cinder.transfer.api.API {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.173622 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] transport_url = **** {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.174139 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] until_refresh = 0 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.174640 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] use_chap_auth = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.175241 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] use_default_quota_class = True {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.175759 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] use_eventlog = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.176305 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] use_journal = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.176873 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] use_json = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.182015 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] use_multipath_for_image_xfer = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.182784 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] use_stderr = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.183308 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] use_syslog = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.183894 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] verify_glance_signatures = enabled {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.184432 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] volume_api_class = cinder.volume.api.API {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.185052 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] volume_backend_name = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.185582 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] volume_clear = zero {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.186077 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] volume_clear_ionice = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.186591 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] volume_clear_size = 0 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.187121 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.190395 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] volume_copy_bps_limit = 0 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.190972 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] volume_dd_blocksize = 1M {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.191693 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] volume_manager = cinder.volume.manager.VolumeManager {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.192248 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] volume_name_template = volume-%s {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.193001 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] volume_usage_audit_period = month {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.193786 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] volumes_dir = /opt/stack/data/cinder/volumes {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.194330 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] watch_log_file = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Mar 05 20:39:40.195038 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.195723 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.196303 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_policy.enforce_new_defaults = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.196867 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_policy.enforce_scope = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.197460 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_policy.policy_default_rule = default {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.198048 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_policy.policy_dirs = ['policy.d'] {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.198628 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_policy.policy_file = policy.yaml {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.199218 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.199794 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.200345 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.200889 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.201451 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.202111 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_concurrency.disable_process_locking = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.202897 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_concurrency.lock_path = /opt/stack/data/cinder {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.203533 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.admin_password = **** {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.204096 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.admin_tenant_name = admin {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.204623 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.admin_token = **** {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.205166 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.admin_user = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.205811 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.auth_admin_prefix = {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.206377 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.206926 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.auth_port = 35357 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.207622 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.auth_protocol = https {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.209130 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.auth_section = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.209692 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.auth_type = password {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.210257 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.auth_uri = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.210782 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.auth_version = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.211324 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.cache = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.211977 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.cafile = /opt/stack/data/ca-bundle.pem {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.212569 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.certfile = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.213297 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.delay_auth_decision = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.213871 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.enforce_token_bind = permissive {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.214566 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.http_connect_timeout = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.215153 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.http_request_max_retries = 3 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.215848 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.identity_uri = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.216411 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.include_service_catalog = True {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.216988 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.insecure = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.217561 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.interface = public {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.218136 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.keyfile = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.218695 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.219400 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.220013 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.220581 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.221153 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.221722 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.memcache_secret_key = **** {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.222391 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.memcache_security_strategy = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.223254 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.224441 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.memcached_servers = ['localhost:11211'] {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.225035 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.region_name = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.225715 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.service_token_roles = ['service'] {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.226386 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.service_token_roles_required = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.226949 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.service_type = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.227521 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.token_cache_time = 300 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.228138 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] keystone_authtoken.www_authenticate_uri = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.228851 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.backend = dogpile.cache.null {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.229450 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.backend_argument = **** {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.230027 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.config_prefix = cache.oslo {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.230645 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.dead_timeout = 60.0 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.231231 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.debug_cache_backend = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.231917 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.enable_retry_client = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.232579 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.enable_socket_keepalive = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.233258 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.enabled = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.233942 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.expiration_time = 600 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.234501 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.hashclient_retry_attempts = 2 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.235072 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.hashclient_retry_delay = 1.0 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.235929 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.memcache_dead_retry = 300 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.236494 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.memcache_pool_connection_get_timeout = 10 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.237066 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.memcache_pool_flush_on_reconnect = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.237648 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.memcache_pool_maxsize = 10 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.238203 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.memcache_pool_unused_timeout = 60 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.238834 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.memcache_servers = ['localhost:11211'] {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.239499 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.memcache_socket_timeout = 1.0 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.240167 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.proxies = [] {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.240749 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.retry_attempts = 2 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.241312 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.retry_delay = 0.0 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.241868 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.socket_keepalive_count = 1 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.242525 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.socket_keepalive_idle = 1 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.243109 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.socket_keepalive_interval = 1 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.243885 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.tls_allowed_ciphers = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.244440 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.tls_cafile = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.245028 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.tls_certfile = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.245584 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.tls_enabled = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.246249 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cache.tls_keyfile = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.246851 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] coordination.backend_url = etcd3+http://10.222.0.15:2379 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.247503 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] database.backend = sqlalchemy {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.248042 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] database.connection = **** {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.248645 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] database.connection_debug = 0 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.249214 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] database.connection_parameters = {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.249789 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] database.connection_recycle_time = 3600 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.250370 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] database.connection_trace = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.250929 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] database.db_inc_retry_interval = True {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.251493 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] database.db_max_retries = 20 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.252083 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] database.db_max_retry_interval = 10 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.252780 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] database.db_retry_interval = 1 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.253375 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] database.max_overflow = 50 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.254142 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] database.max_pool_size = 5 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.254710 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] database.max_retries = 10 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.255294 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] database.mysql_enable_ndb = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.256159 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] database.mysql_sql_mode = TRADITIONAL {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.256721 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] database.pool_timeout = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.257317 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] database.retry_interval = 10 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.257899 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] database.slave_connection = **** {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.258492 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] database.sqlite_synchronous = True {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.259062 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] database.use_db_reconnect = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.259717 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.260300 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.260868 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_metrics.metrics_process_name = {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.261433 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.262000 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.262714 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] profiler.connection_string = messaging:// {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.263229 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] profiler.enabled = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.264024 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] profiler.es_doc_type = notification {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.264603 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] profiler.es_scroll_size = 10000 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.265182 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] profiler.es_scroll_time = 2m {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.265743 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] profiler.filter_error_trace = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.266427 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] profiler.hmac_keys = SECRET_KEY {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.267014 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] profiler.sentinel_service_name = mymaster {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.267585 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] profiler.socket_timeout = 0.1 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.268315 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] profiler.trace_sqlalchemy = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.268954 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] privsep_osbrick.capabilities = [21] {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.269545 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] privsep_osbrick.group = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.270115 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] privsep_osbrick.helper_command = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.271245 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] privsep_osbrick.logger_name = oslo_privsep.daemon {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.271854 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] privsep_osbrick.thread_pool_size = 4 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.272509 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] privsep_osbrick.user = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.273209 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cinder_sys_admin.capabilities = [0, 1, 2, 3, 12, 21] {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.273803 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cinder_sys_admin.group = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.274502 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cinder_sys_admin.helper_command = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.275083 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cinder_sys_admin.logger_name = oslo_privsep.daemon {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.275643 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cinder_sys_admin.thread_pool_size = 4 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.276399 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] cinder_sys_admin.user = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.277017 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] service_user.auth_section = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.277590 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] service_user.auth_type = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.278251 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] service_user.cafile = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.278814 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] service_user.certfile = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.279410 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] service_user.collect_timing = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.280012 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] service_user.insecure = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.280636 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] service_user.keyfile = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.281210 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] service_user.send_service_user_token = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.281856 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] service_user.split_loggers = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.282428 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] service_user.timeout = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.283150 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.backend_availability_zone = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.283904 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.chap_password = **** {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.284589 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.chap_username = {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.285173 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.285735 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.deferred_deletion_delay = 0 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.286972 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.deferred_deletion_purge_interval = 60 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.287551 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.driver_client_cert = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.288284 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.driver_client_cert_key = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.288855 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.driver_data_namespace = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.289410 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.driver_ssl_cert_path = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.289965 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.driver_ssl_cert_verify = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.290524 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.driver_use_ssl = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.291090 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.enable_deferred_deletion = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.291946 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.enable_unsupported_driver = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.292511 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.enforce_multipath_for_image_xfer = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.293173 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.filter_function = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.293726 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.goodness_function = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.294500 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.iet_conf = /etc/iet/ietd.conf {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.295070 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.image_upload_use_cinder_backend = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.295641 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.image_upload_use_internal_tenant = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.296235 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.image_volume_cache_enabled = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.296800 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.image_volume_cache_max_count = 0 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.297445 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.image_volume_cache_max_size_gb = 0 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.298036 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.iscsi_iotype = fileio {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.298682 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.iscsi_secondary_ip_addresses = [] {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.299266 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.iscsi_target_flags = {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.299861 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.iscsi_write_cache = on {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.300429 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.iser_helper = tgtadm {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.301190 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.iser_ip_address = 10.222.0.15 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.301752 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.iser_port = 3260 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.302385 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.302969 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.max_over_subscription_ratio = 20.0 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.303610 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.num_iser_scan_tries = 3 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.304212 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.num_shell_tries = 3 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.304947 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.num_volume_device_scan_tries = 3 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.305506 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.nvmet_ns_id = 10 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.306078 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.nvmet_port_id = 1 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.306650 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.rados_connect_timeout = -1 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.307339 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.rados_connection_interval = 5 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.307951 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.rados_connection_retries = 3 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.308644 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.rbd_ceph_conf = {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.309219 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.rbd_cluster_name = ceph {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.309785 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.rbd_exclusive_cinder_pool = True {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.310348 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.rbd_flatten_volume_from_snapshot = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.310915 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.rbd_max_clone_depth = 5 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.311486 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.rbd_pool = rbd {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.312054 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.rbd_secret_uuid = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.312624 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.rbd_store_chunk_size = 4 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.313192 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.rbd_user = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.313847 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.replication_connect_timeout = 5 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.314454 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.replication_device = **** {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.315131 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.report_discard_supported = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.315765 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.report_dynamic_total_capacity = True {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.316322 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.reserved_percentage = 0 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.316905 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.scst_target_driver = iscsi {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.317710 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.scst_target_iqn_name = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.318285 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.storage_protocol = iscsi {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.318943 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.target_helper = tgtadm {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.319621 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.target_ip_address = 10.222.0.15 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.320234 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.target_port = 3260 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.320823 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.target_prefix = iqn.2010-10.org.openstack: {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.321415 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.target_protocol = iscsi {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.321974 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.trace_flags = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.322537 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.unique_fqdn_network = True {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.323104 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.use_chap_auth = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.323881 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.use_multipath_for_image_xfer = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.324440 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.volume_backend_name = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.325146 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.volume_clear = zero {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.325701 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.volume_clear_ionice = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.326262 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.volume_clear_size = 0 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.326828 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.327585 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.volume_copy_bps_limit = 0 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.328191 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.volume_dd_blocksize = 1M {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.328948 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] backend_defaults.volumes_dir = /opt/stack/data/cinder/volumes {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.329536 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_reports.file_event_handler = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.330141 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_reports.file_event_handler_interval = 1 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.330728 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_reports.log_dir = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.331483 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.332235 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.332850 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.334032 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.334674 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.335574 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.336204 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_rabbit.heartbeat_in_pthread = True {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.336827 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_rabbit.heartbeat_rate = 2 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.337395 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.338089 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.338709 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.339352 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.339985 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.340589 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.341146 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.341760 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.342320 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.342897 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.343481 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.344171 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.344768 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.345366 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.346073 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_rabbit.ssl = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.346661 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_rabbit.ssl_ca_file = {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.347247 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_rabbit.ssl_cert_file = {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.347950 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_rabbit.ssl_key_file = {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.348549 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_rabbit.ssl_version = {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.349372 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_notifications.driver = [] {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.349985 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_notifications.retry = -1 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.350578 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.351155 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] oslo_messaging_notifications.transport_url = **** {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.351845 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] key_manager.backend = cinder.keymgr.conf_key_mgr.ConfKeyManager {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.352484 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] barbican.auth_endpoint = http://localhost/identity/v3 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.353062 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] barbican.barbican_api_version = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.353617 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] barbican.barbican_endpoint = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.354333 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] barbican.barbican_endpoint_type = public {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.354906 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] barbican.barbican_region_name = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.355456 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] barbican.cafile = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.356150 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] barbican.certfile = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.356726 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] barbican.collect_timing = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.357281 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] barbican.insecure = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.357957 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] barbican.keyfile = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.358531 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] barbican.number_of_retries = 60 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.359085 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] barbican.retry_delay = 1 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.359783 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] barbican.send_service_user_token = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.360348 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] barbican.split_loggers = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.360911 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] barbican.timeout = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.361476 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] barbican.verify_ssl = True {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.362024 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] barbican.verify_ssl_path = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.362583 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] barbican_service_user.auth_section = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.363161 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] barbican_service_user.auth_type = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.363742 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] barbican_service_user.cafile = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.365014 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] barbican_service_user.certfile = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.365582 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] barbican_service_user.collect_timing = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.366287 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] barbican_service_user.insecure = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.366837 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] barbican_service_user.keyfile = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.367400 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] barbican_service_user.split_loggers = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.368102 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] barbican_service_user.timeout = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.368696 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] vault.approle_role_id = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.369257 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] vault.approle_secret_id = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.369897 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] vault.cafile = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.370464 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] vault.certfile = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.371035 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] vault.collect_timing = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.371609 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] vault.insecure = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.372199 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] vault.keyfile = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.372778 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] vault.kv_mountpoint = secret {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.373346 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] vault.kv_version = 2 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.373895 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] vault.namespace = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.374557 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] vault.root_token_id = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.375129 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] vault.split_loggers = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.375717 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] vault.ssl_ca_crt_file = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.376402 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] vault.timeout = None {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.376979 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] vault.use_ssl = False {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.377561 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] vault.vault_url = http://127.0.0.1:8200 {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Mar 05 20:39:40.378065 ncdv-master-1 cinder-backup[134217]: DEBUG oslo_service.service [None req-966f0f89-3cc8-4264-850b-665aa9187757 None None] ******************************************************************************** {{(pid=134217) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Mar 05 20:39:40.399564 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.lockutils [None req-f2a0b876-efda-41dc-ae1a-b3429f4644ac None None] Lock "cleanup_incomplete_backups_134217" "released" by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: held 0.324s {{(pid=134663) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:400}} Mar 05 20:39:40.405401 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_db.sqlalchemy.engines [None req-f2a0b876-efda-41dc-ae1a-b3429f4644ac None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134663) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 05 20:39:40.412417 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-f2a0b876-efda-41dc-ae1a-b3429f4644ac None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 20:39:40.445784 ncdv-master-1 cinder-backup[134663]: INFO cinder.keymgr.migration [None req-f28a9af3-65e7-4c55-8830-a229d8803448 None None] Not migrating encryption keys because the ConfKeyManager is still in use. Mar 05 20:39:40.463379 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.service [None req-f2a0b876-efda-41dc-ae1a-b3429f4644ac None None] Creating RPC server for service cinder-backup {{(pid=134663) start /opt/stack/cinder/cinder/service.py:231}} Mar 05 20:39:40.477818 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.service [None req-f2a0b876-efda-41dc-ae1a-b3429f4644ac None None] Pinning object versions for RPC server serializer to 1.39 {{(pid=134663) start /opt/stack/cinder/cinder/service.py:237}} Mar 05 20:40:49.517232 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 20:40:49.517984 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 20:41:49.521881 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 20:41:49.522574 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 20:42:49.527174 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 20:42:49.528284 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 20:43:49.523166 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 20:43:49.523909 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 20:44:49.528157 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 20:44:49.528858 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 20:45:49.533078 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 20:45:49.533783 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 20:46:49.538441 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 20:46:49.539270 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 20:47:49.534250 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 20:47:49.535005 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 20:48:49.539184 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 20:48:49.539818 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 20:49:49.538539 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 20:49:49.539852 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 20:50:49.537743 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 20:50:49.538915 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 20:52:49.538651 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 20:52:49.539445 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 20:53:49.535974 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 20:53:49.536565 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 20:54:49.538837 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 20:54:49.539511 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 20:55:49.540523 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 20:55:49.541254 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 20:56:49.546838 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 20:56:49.547491 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 20:57:49.542884 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 20:57:49.544437 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 20:58:28.783868 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] Create backup started, backup: c9ae053c-27a0-41db-88dd-5d0400936bd6 volume: a7055011-640a-4cb4-b12b-077a721ff496. Mar 05 20:58:28.803754 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-03-05T20:58:28Z,data_timestamp=2022-03-05T20:58:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsV39Test-Backup-834020033',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=c9ae053c-27a0-41db-88dd-5d0400936bd6,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='0ff38e7d62dc4f5daa04cd84693c2e90',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='196f6dfa309e48088c9be46210001e7b',volume_id=a7055011-640a-4cb4-b12b-077a721ff496) Mar 05 20:58:28.870611 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.utils [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.15', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 20:58:28.872967 ncdv-master-1 cinder-backup[134663]: INFO oslo.privsep.daemon [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] Running privsep helper: ['sudo', 'cinder-rootwrap', '/etc/cinder/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/cinder/cinder.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmprejwipx2/privsep.sock'] Mar 05 20:58:28.921845 ncdv-master-1 sudo[151003]: ubuntu : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf privsep-helper --config-file /etc/cinder/cinder.conf --privsep_context os_brick.privileged.default --privsep_sock_path /tmp/tmprejwipx2/privsep.sock Mar 05 20:58:28.922700 ncdv-master-1 sudo[151003]: pam_unix(sudo:session): session opened for user root by (uid=0) Mar 05 20:58:30.666552 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_db.sqlalchemy.engines [None req-ca790f2e-02b0-4b56-8816-8cf551d2bc00 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134663) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 05 20:58:32.539227 ncdv-master-1 sudo[151003]: pam_unix(sudo:session): session closed for user root Mar 05 20:58:32.549690 ncdv-master-1 cinder-backup[134663]: INFO oslo.privsep.daemon [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] Spawned new privsep daemon via rootwrap Mar 05 20:58:32.551343 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] Accepted privsep connection to /tmp/tmprejwipx2/privsep.sock {{(pid=134663) __init__ /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:362}} Mar 05 20:58:32.556686 ncdv-master-1 cinder-backup[134663]: INFO oslo.privsep.daemon [-] privsep daemon starting Mar 05 20:58:32.556864 ncdv-master-1 cinder-backup[134663]: INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 Mar 05 20:58:32.557485 ncdv-master-1 cinder-backup[134663]: INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none Mar 05 20:58:32.558437 ncdv-master-1 cinder-backup[134663]: INFO oslo.privsep.daemon [-] privsep daemon running as pid 151032 Mar 05 20:58:32.561727 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[416234b1-9c95-428d-878a-56abdf046db8]: (2,) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 20:58:33.122915 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 20:58:33.141301 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.020s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 20:58:33.141783 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[074a9e17-b3d9-454d-a532-e81f05a74d20]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ab336d69f798\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 20:58:33.143444 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 20:58:33.157466 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 05 20:58:33.190535 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[f62918e3-08f9-4112-b524-1c95c2e09262]: [Errno 2] No such file or directory: 'systool' {{(pid=151032) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 05 20:58:33.190535 ncdv-master-1 cinder-backup[134663]: Traceback (most recent call last): Mar 05 20:58:33.190535 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 05 20:58:33.190535 ncdv-master-1 cinder-backup[134663]: ret = func(*f_args, **f_kwargs) Mar 05 20:58:33.190535 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 05 20:58:33.190535 ncdv-master-1 cinder-backup[134663]: return func(*args, **kwargs) Mar 05 20:58:33.190535 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 05 20:58:33.190535 ncdv-master-1 cinder-backup[134663]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 05 20:58:33.190535 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 05 20:58:33.190535 ncdv-master-1 cinder-backup[134663]: return putils.execute(on_execute=on_execute, Mar 05 20:58:33.190535 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 05 20:58:33.190535 ncdv-master-1 cinder-backup[134663]: obj = subprocess.Popen(cmd, Mar 05 20:58:33.190535 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 05 20:58:33.190535 ncdv-master-1 cinder-backup[134663]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 05 20:58:33.190535 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 05 20:58:33.190535 ncdv-master-1 cinder-backup[134663]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 05 20:58:33.190535 ncdv-master-1 cinder-backup[134663]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 05 20:58:33.192432 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[f62918e3-08f9-4112-b524-1c95c2e09262]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 20:58:33.193633 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 20:58:33.201511 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 05 20:58:33.202497 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[7864a0f7-1e5d-4117-8bc3-817787b3b25b]: [Errno 2] No such file or directory: 'systool' {{(pid=151032) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 05 20:58:33.202497 ncdv-master-1 cinder-backup[134663]: Traceback (most recent call last): Mar 05 20:58:33.202497 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 05 20:58:33.202497 ncdv-master-1 cinder-backup[134663]: ret = func(*f_args, **f_kwargs) Mar 05 20:58:33.202497 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 05 20:58:33.202497 ncdv-master-1 cinder-backup[134663]: return func(*args, **kwargs) Mar 05 20:58:33.202497 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 05 20:58:33.202497 ncdv-master-1 cinder-backup[134663]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 05 20:58:33.202497 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 05 20:58:33.202497 ncdv-master-1 cinder-backup[134663]: return putils.execute(on_execute=on_execute, Mar 05 20:58:33.202497 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 05 20:58:33.202497 ncdv-master-1 cinder-backup[134663]: obj = subprocess.Popen(cmd, Mar 05 20:58:33.202497 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 05 20:58:33.202497 ncdv-master-1 cinder-backup[134663]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 05 20:58:33.202497 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 05 20:58:33.202497 ncdv-master-1 cinder-backup[134663]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 05 20:58:33.202497 ncdv-master-1 cinder-backup[134663]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 05 20:58:33.203904 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[7864a0f7-1e5d-4117-8bc3-817787b3b25b]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 20:58:33.206352 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 20:58:33.258704 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.052s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 20:58:33.259020 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[6e3edc1b-5b58-4e82-8f79-082b0202940f]: (4, ('/dev/sda1\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 20:58:33.260734 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 20:58:33.325514 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.064s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 20:58:33.326131 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[d595253b-c1e0-45b2-9767-b2e24688bfbc]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 20:58:33.328028 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 20:58:33.350882 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.022s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 20:58:33.351148 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[d108c191-849c-4afa-93b6-f67ae66c84df]: (4, ('8e294445-14da-a449-a0d4-86f2c95f64cd\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 20:58:33.352607 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] Running cmd (subprocess): nvme version {{(pid=134663) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 20:58:33.408940 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] CMD "nvme version" returned: 0 in 0.056s {{(pid=134663) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 20:58:33.418120 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134663) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 05 20:58:33.418488 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134663) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 05 20:58:33.419103 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:71cf25bc-00fa-4fd3-8614-fee15d100887 dsc: {{(pid=134663) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 05 20:58:33.420332 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.utils [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] <== get_connector_properties: return (4548ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.15', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ab336d69f798', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '8e294445-14da-a449-a0d4-86f2c95f64cd', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:71cf25bc-00fa-4fd3-8614-fee15d100887', 'found_dsc': ''} {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 20:58:34.524790 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connector [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] Factory for rbd on None {{(pid=134663) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 05 20:58:34.528493 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-a7055011-640a-4cb4-b12b-077a721ff496', 'hosts': ['10.222.0.15'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'a7055011-640a-4cb4-b12b-077a721ff496', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 20:58:34.531034 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.linuxrbd [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] opening connection to ceph cluster (timeout=-1). {{(pid=134663) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 05 20:58:34.647187 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] <== connect_volume: return (116ms) {'path': } {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 20:58:34.714686 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] Starting backup of volume='a7055011-640a-4cb4-b12b-077a721ff496'. {{(pid=134663) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Mar 05 20:58:34.718906 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] Volume file is RBD: attempting optimized backup {{(pid=134663) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Mar 05 20:58:34.821942 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] Creating base image 'volume-a7055011-640a-4cb4-b12b-077a721ff496.backup.c9ae053c-27a0-41db-88dd-5d0400936bd6' {{(pid=134663) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Mar 05 20:58:34.822374 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134663) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Mar 05 20:58:36.162148 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] Using --from-snap 'None' for incremental backup of volume a7055011-640a-4cb4-b12b-077a721ff496. {{(pid=134663) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Mar 05 20:58:36.164063 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] Creating backup snapshot='backup.c9ae053c-27a0-41db-88dd-5d0400936bd6.snap.1646513916.1629133' {{(pid=134663) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Mar 05 20:58:37.135321 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] Performing differential transfer from 'volume-a7055011-640a-4cb4-b12b-077a721ff496' to 'volume-a7055011-640a-4cb4-b12b-077a721ff496.backup.c9ae053c-27a0-41db-88dd-5d0400936bd6' {{(pid=134663) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Mar 05 20:58:37.135960 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_5mmmm1zd --pool volumes volumes/volume-a7055011-640a-4cb4-b12b-077a721ff496@backup.c9ae053c-27a0-41db-88dd-5d0400936bd6.snap.1646513916.1629133 -' into... {{(pid=134663) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Mar 05 20:58:37.136731 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-a7055011-640a-4cb4-b12b-077a721ff496.backup.c9ae053c-27a0-41db-88dd-5d0400936bd6' {{(pid=134663) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Mar 05 20:58:38.197255 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] Differential backup transfer completed in 1.0609s {{(pid=134663) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Mar 05 20:58:38.197905 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] Getting metadata type 'volume-base-metadata' {{(pid=134663) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Mar 05 20:58:38.231824 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134663) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Mar 05 20:58:38.234256 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] Getting metadata type 'volume-metadata' {{(pid=134663) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Mar 05 20:58:38.274998 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] No metadata type 'volume-metadata' available {{(pid=134663) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Mar 05 20:58:38.277535 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] Getting metadata type 'volume-glance-metadata' {{(pid=134663) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Mar 05 20:58:38.307764 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] No metadata type 'volume-glance-metadata' available {{(pid=134663) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Mar 05 20:58:38.308614 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] Backing up metadata for volume a7055011-640a-4cb4-b12b-077a721ff496. {{(pid=134663) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Mar 05 20:58:38.405064 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] Backup 'c9ae053c-27a0-41db-88dd-5d0400936bd6' of volume a7055011-640a-4cb4-b12b-077a721ff496 finished. {{(pid=134663) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Mar 05 20:58:38.408759 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-a7055011-640a-4cb4-b12b-077a721ff496', 'hosts': ['10.222.0.15'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'a7055011-640a-4cb4-b12b-077a721ff496', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 20:58:38.450349 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] <== disconnect_volume: return (40ms) None {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 20:58:38.824030 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-d24df752-a049-4460-9051-eb58314dfef9 tempest-VolumesBackupsV39Test-610288572 None] Create backup finished. backup: c9ae053c-27a0-41db-88dd-5d0400936bd6. Mar 05 20:58:39.358246 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-bf0a5a81-b5d6-4359-9528-218cc5db457b tempest-VolumesBackupsV39Test-610288572 None] Delete backup started, backup: c9ae053c-27a0-41db-88dd-5d0400936bd6. Mar 05 20:58:39.360866 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-bf0a5a81-b5d6-4359-9528-218cc5db457b tempest-VolumesBackupsV39Test-610288572 None] Delete started for backup=c9ae053c-27a0-41db-88dd-5d0400936bd6 {{(pid=134663) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Mar 05 20:58:39.361294 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-bf0a5a81-b5d6-4359-9528-218cc5db457b tempest-VolumesBackupsV39Test-610288572 None] Trying diff format basename='volume-a7055011-640a-4cb4-b12b-077a721ff496.backup.c9ae053c-27a0-41db-88dd-5d0400936bd6' for backup base image of volume a7055011-640a-4cb4-b12b-077a721ff496. {{(pid=134663) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Mar 05 20:58:39.451995 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-bf0a5a81-b5d6-4359-9528-218cc5db457b tempest-VolumesBackupsV39Test-610288572 None] Looking for snapshot of backup base 'volume-a7055011-640a-4cb4-b12b-077a721ff496.backup.c9ae053c-27a0-41db-88dd-5d0400936bd6' {{(pid=134663) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 05 20:58:39.451995 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-bf0a5a81-b5d6-4359-9528-218cc5db457b tempest-VolumesBackupsV39Test-610288572 None] Found snapshot 'backup.c9ae053c-27a0-41db-88dd-5d0400936bd6.snap.1646513916.1629133' {{(pid=134663) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Mar 05 20:58:39.454834 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-bf0a5a81-b5d6-4359-9528-218cc5db457b tempest-VolumesBackupsV39Test-610288572 None] Deleting backup snapshot='backup.c9ae053c-27a0-41db-88dd-5d0400936bd6.snap.1646513916.1629133' {{(pid=134663) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Mar 05 20:58:40.234271 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.drivers.ceph [None req-bf0a5a81-b5d6-4359-9528-218cc5db457b tempest-VolumesBackupsV39Test-610288572 None] Deleting backup base image='volume-a7055011-640a-4cb4-b12b-077a721ff496.backup.c9ae053c-27a0-41db-88dd-5d0400936bd6' of volume a7055011-640a-4cb4-b12b-077a721ff496. Mar 05 20:58:40.584021 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-bf0a5a81-b5d6-4359-9528-218cc5db457b tempest-VolumesBackupsV39Test-610288572 None] Base backup image='volume-a7055011-640a-4cb4-b12b-077a721ff496.backup.c9ae053c-27a0-41db-88dd-5d0400936bd6' of volume a7055011-640a-4cb4-b12b-077a721ff496 deleted. {{(pid=134663) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Mar 05 20:58:40.669064 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_db.sqlalchemy.engines [None req-ca790f2e-02b0-4b56-8816-8cf551d2bc00 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134663) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 05 20:58:40.690255 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-bf0a5a81-b5d6-4359-9528-218cc5db457b tempest-VolumesBackupsV39Test-610288572 None] Delete of backup 'c9ae053c-27a0-41db-88dd-5d0400936bd6' for volume 'a7055011-640a-4cb4-b12b-077a721ff496' finished. {{(pid=134663) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Mar 05 20:58:40.741704 ncdv-master-1 cinder-backup[134663]: WARNING cinder.quota [None req-bf0a5a81-b5d6-4359-9528-218cc5db457b tempest-VolumesBackupsV39Test-610288572 None] Deprecated: Default quota for resource: backups is set by the default quota flag: quota_backups, it is now deprecated. Please use the default quota class for default quota. Mar 05 20:58:40.743201 ncdv-master-1 cinder-backup[134663]: WARNING cinder.quota [None req-bf0a5a81-b5d6-4359-9528-218cc5db457b tempest-VolumesBackupsV39Test-610288572 None] Deprecated: Default quota for resource: backup_gigabytes is set by the default quota flag: quota_backup_gigabytes, it is now deprecated. Please use the default quota class for default quota. Mar 05 20:58:40.768474 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.quota [None req-bf0a5a81-b5d6-4359-9528-218cc5db457b tempest-VolumesBackupsV39Test-610288572 None] Created reservations ['ba419c7a-a246-4eb0-9c40-ad98bf79e3bb', '29c3e5aa-5689-488a-9807-55f62a022804'] {{(pid=134663) reserve /opt/stack/cinder/cinder/quota.py:755}} Mar 05 20:58:40.849398 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-bf0a5a81-b5d6-4359-9528-218cc5db457b tempest-VolumesBackupsV39Test-610288572 None] Delete backup finished, backup c9ae053c-27a0-41db-88dd-5d0400936bd6 deleted. Mar 05 20:58:49.541389 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 20:58:49.542575 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 20:59:49.539688 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 20:59:49.541362 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:00:49.537441 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:00:49.538497 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:01:49.533981 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:01:49.534945 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:02:49.531638 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:02:49.532377 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:03:49.536283 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:03:49.536986 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:04:49.539254 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:04:49.541379 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:05:49.544198 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:05:49.549185 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:06:49.548493 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:06:49.549254 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:07:49.546774 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:07:49.547397 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:08:13.706168 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] Create backup started, backup: e6dd2589-6441-4a5d-8b21-aca7be541bc9 volume: b22e1911-a4e8-47cf-863e-0a7a04265855. Mar 05 21:08:13.738354 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-03-05T21:08:13Z,data_timestamp=2022-03-05T21:08:13Z,deleted=False,deleted_at=None,display_description=None,display_name=None,encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=e6dd2589-6441-4a5d-8b21-aca7be541bc9,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='aa71bdd8c3394c1ca9fb656a112d9349',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='d0f84866b0f34e6da2d051b5b3583e13',volume_id=b22e1911-a4e8-47cf-863e-0a7a04265855) Mar 05 21:08:13.851996 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.utils [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.15', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 21:08:13.857164 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:08:13.925593 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.070s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:08:13.926117 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[a30a12ba-3dd0-4153-b5f1-428fe967146f]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ab336d69f798\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:08:13.929989 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:08:13.980357 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 05 21:08:13.981923 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[5f0e6d25-65d3-40bb-981d-4eaf6691be2d]: [Errno 2] No such file or directory: 'systool' {{(pid=151032) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 05 21:08:13.981923 ncdv-master-1 cinder-backup[134663]: Traceback (most recent call last): Mar 05 21:08:13.981923 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 05 21:08:13.981923 ncdv-master-1 cinder-backup[134663]: ret = func(*f_args, **f_kwargs) Mar 05 21:08:13.981923 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 05 21:08:13.981923 ncdv-master-1 cinder-backup[134663]: return func(*args, **kwargs) Mar 05 21:08:13.981923 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 05 21:08:13.981923 ncdv-master-1 cinder-backup[134663]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 05 21:08:13.981923 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 05 21:08:13.981923 ncdv-master-1 cinder-backup[134663]: return putils.execute(on_execute=on_execute, Mar 05 21:08:13.981923 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 05 21:08:13.981923 ncdv-master-1 cinder-backup[134663]: obj = subprocess.Popen(cmd, Mar 05 21:08:13.981923 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 05 21:08:13.981923 ncdv-master-1 cinder-backup[134663]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 05 21:08:13.981923 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 05 21:08:13.981923 ncdv-master-1 cinder-backup[134663]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 05 21:08:13.981923 ncdv-master-1 cinder-backup[134663]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 05 21:08:13.983202 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[5f0e6d25-65d3-40bb-981d-4eaf6691be2d]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:08:13.985615 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:08:14.012068 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 05 21:08:14.012657 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[1bf65acc-1929-4f69-9e04-cbda1ca4a539]: [Errno 2] No such file or directory: 'systool' {{(pid=151032) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 05 21:08:14.012657 ncdv-master-1 cinder-backup[134663]: Traceback (most recent call last): Mar 05 21:08:14.012657 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 05 21:08:14.012657 ncdv-master-1 cinder-backup[134663]: ret = func(*f_args, **f_kwargs) Mar 05 21:08:14.012657 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 05 21:08:14.012657 ncdv-master-1 cinder-backup[134663]: return func(*args, **kwargs) Mar 05 21:08:14.012657 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 05 21:08:14.012657 ncdv-master-1 cinder-backup[134663]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 05 21:08:14.012657 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 05 21:08:14.012657 ncdv-master-1 cinder-backup[134663]: return putils.execute(on_execute=on_execute, Mar 05 21:08:14.012657 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 05 21:08:14.012657 ncdv-master-1 cinder-backup[134663]: obj = subprocess.Popen(cmd, Mar 05 21:08:14.012657 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 05 21:08:14.012657 ncdv-master-1 cinder-backup[134663]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 05 21:08:14.012657 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 05 21:08:14.012657 ncdv-master-1 cinder-backup[134663]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 05 21:08:14.012657 ncdv-master-1 cinder-backup[134663]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 05 21:08:14.013752 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[1bf65acc-1929-4f69-9e04-cbda1ca4a539]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:08:14.016086 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:08:14.061548 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.045s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:08:14.061839 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[c02665a1-6c6e-4c8f-8a32-5ad130800e39]: (4, ('/dev/sda1\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:08:14.066635 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:08:14.129246 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.064s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:08:14.129886 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[1df1faee-b39d-479e-abdb-7576fd2d33bd]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:08:14.133270 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:08:14.167825 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.036s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:08:14.168366 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[65ffb9a1-8968-430b-bfc3-fdd76eb517f2]: (4, ('8e294445-14da-a449-a0d4-86f2c95f64cd\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:08:14.169445 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] Running cmd (subprocess): nvme version {{(pid=134663) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:08:14.214081 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] CMD "nvme version" returned: 0 in 0.044s {{(pid=134663) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:08:14.216511 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134663) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 05 21:08:14.217130 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134663) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 05 21:08:14.217667 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:71cf25bc-00fa-4fd3-8614-fee15d100887 dsc: {{(pid=134663) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 05 21:08:14.218513 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.utils [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] <== get_connector_properties: return (365ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.15', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ab336d69f798', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '8e294445-14da-a449-a0d4-86f2c95f64cd', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:71cf25bc-00fa-4fd3-8614-fee15d100887', 'found_dsc': ''} {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 21:08:15.390813 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connector [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] Factory for rbd on None {{(pid=134663) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 05 21:08:15.392408 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-b22e1911-a4e8-47cf-863e-0a7a04265855', 'hosts': ['10.222.0.15'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'b22e1911-a4e8-47cf-863e-0a7a04265855', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 21:08:15.393638 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.linuxrbd [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] opening connection to ceph cluster (timeout=-1). {{(pid=134663) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 05 21:08:15.503218 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] <== connect_volume: return (110ms) {'path': } {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 21:08:15.521509 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_db.sqlalchemy.engines [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134663) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 05 21:08:15.559061 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] Starting backup of volume='b22e1911-a4e8-47cf-863e-0a7a04265855'. {{(pid=134663) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Mar 05 21:08:15.561703 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] Volume file is RBD: attempting optimized backup {{(pid=134663) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Mar 05 21:08:15.630261 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] Creating base image 'volume-b22e1911-a4e8-47cf-863e-0a7a04265855.backup.e6dd2589-6441-4a5d-8b21-aca7be541bc9' {{(pid=134663) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Mar 05 21:08:15.630988 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134663) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Mar 05 21:08:15.696722 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] Using --from-snap 'None' for incremental backup of volume b22e1911-a4e8-47cf-863e-0a7a04265855. {{(pid=134663) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Mar 05 21:08:15.697281 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] Creating backup snapshot='backup.e6dd2589-6441-4a5d-8b21-aca7be541bc9.snap.1646514495.696748' {{(pid=134663) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Mar 05 21:08:16.123309 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] Performing differential transfer from 'volume-b22e1911-a4e8-47cf-863e-0a7a04265855' to 'volume-b22e1911-a4e8-47cf-863e-0a7a04265855.backup.e6dd2589-6441-4a5d-8b21-aca7be541bc9' {{(pid=134663) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Mar 05 21:08:16.123747 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_whvln5tw --pool volumes volumes/volume-b22e1911-a4e8-47cf-863e-0a7a04265855@backup.e6dd2589-6441-4a5d-8b21-aca7be541bc9.snap.1646514495.696748 -' into... {{(pid=134663) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Mar 05 21:08:16.124641 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-b22e1911-a4e8-47cf-863e-0a7a04265855.backup.e6dd2589-6441-4a5d-8b21-aca7be541bc9' {{(pid=134663) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Mar 05 21:08:24.432240 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] Differential backup transfer completed in 8.3087s {{(pid=134663) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Mar 05 21:08:24.433541 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] Getting metadata type 'volume-base-metadata' {{(pid=134663) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Mar 05 21:08:24.451905 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134663) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Mar 05 21:08:24.452319 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] Getting metadata type 'volume-metadata' {{(pid=134663) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Mar 05 21:08:24.460115 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] No metadata type 'volume-metadata' available {{(pid=134663) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Mar 05 21:08:24.460494 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] Getting metadata type 'volume-glance-metadata' {{(pid=134663) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Mar 05 21:08:24.467823 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=134663) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}} Mar 05 21:08:24.468473 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] Backing up metadata for volume b22e1911-a4e8-47cf-863e-0a7a04265855. {{(pid=134663) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Mar 05 21:08:24.514574 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] Backup 'e6dd2589-6441-4a5d-8b21-aca7be541bc9' of volume b22e1911-a4e8-47cf-863e-0a7a04265855 finished. {{(pid=134663) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Mar 05 21:08:24.516900 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-b22e1911-a4e8-47cf-863e-0a7a04265855', 'hosts': ['10.222.0.15'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'b22e1911-a4e8-47cf-863e-0a7a04265855', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 21:08:24.551708 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] <== disconnect_volume: return (33ms) None {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 21:08:24.656460 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-f48a0563-41c4-4631-b675-dab6edc26e20 tempest-TestVolumeBackupRestore-590870803 None] Create backup finished. backup: e6dd2589-6441-4a5d-8b21-aca7be541bc9. Mar 05 21:08:26.684015 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] Restore backup started, backup: e6dd2589-6441-4a5d-8b21-aca7be541bc9 volume: 58a73208-a97d-4148-854a-a7d6520bb60b. Mar 05 21:08:26.709428 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.utils [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.15', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 21:08:26.717965 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:08:26.745078 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.031s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:08:26.745416 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[bca73799-c12c-43f8-9e0c-be2f0ee479ec]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ab336d69f798\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:08:26.746890 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:08:26.756880 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 05 21:08:26.758042 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[d12d8f76-e1dc-4cf3-a738-05b38b7ac442]: [Errno 2] No such file or directory: 'systool' {{(pid=151032) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 05 21:08:26.758042 ncdv-master-1 cinder-backup[134663]: Traceback (most recent call last): Mar 05 21:08:26.758042 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 05 21:08:26.758042 ncdv-master-1 cinder-backup[134663]: ret = func(*f_args, **f_kwargs) Mar 05 21:08:26.758042 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 05 21:08:26.758042 ncdv-master-1 cinder-backup[134663]: return func(*args, **kwargs) Mar 05 21:08:26.758042 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 05 21:08:26.758042 ncdv-master-1 cinder-backup[134663]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 05 21:08:26.758042 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 05 21:08:26.758042 ncdv-master-1 cinder-backup[134663]: return putils.execute(on_execute=on_execute, Mar 05 21:08:26.758042 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 05 21:08:26.758042 ncdv-master-1 cinder-backup[134663]: obj = subprocess.Popen(cmd, Mar 05 21:08:26.758042 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 05 21:08:26.758042 ncdv-master-1 cinder-backup[134663]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 05 21:08:26.758042 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 05 21:08:26.758042 ncdv-master-1 cinder-backup[134663]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 05 21:08:26.758042 ncdv-master-1 cinder-backup[134663]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 05 21:08:26.760214 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[d12d8f76-e1dc-4cf3-a738-05b38b7ac442]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:08:26.761954 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:08:26.774949 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 05 21:08:26.775658 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[9a5fc5cc-8b53-4b7c-b465-088bc0bdd9af]: [Errno 2] No such file or directory: 'systool' {{(pid=151032) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 05 21:08:26.775658 ncdv-master-1 cinder-backup[134663]: Traceback (most recent call last): Mar 05 21:08:26.775658 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 05 21:08:26.775658 ncdv-master-1 cinder-backup[134663]: ret = func(*f_args, **f_kwargs) Mar 05 21:08:26.775658 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 05 21:08:26.775658 ncdv-master-1 cinder-backup[134663]: return func(*args, **kwargs) Mar 05 21:08:26.775658 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 05 21:08:26.775658 ncdv-master-1 cinder-backup[134663]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 05 21:08:26.775658 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 05 21:08:26.775658 ncdv-master-1 cinder-backup[134663]: return putils.execute(on_execute=on_execute, Mar 05 21:08:26.775658 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 05 21:08:26.775658 ncdv-master-1 cinder-backup[134663]: obj = subprocess.Popen(cmd, Mar 05 21:08:26.775658 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 05 21:08:26.775658 ncdv-master-1 cinder-backup[134663]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 05 21:08:26.775658 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 05 21:08:26.775658 ncdv-master-1 cinder-backup[134663]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 05 21:08:26.775658 ncdv-master-1 cinder-backup[134663]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 05 21:08:26.776683 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[9a5fc5cc-8b53-4b7c-b465-088bc0bdd9af]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:08:26.777970 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:08:26.793056 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.015s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:08:26.793547 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[b5d214cf-6fcc-4214-b985-ecb3edd1fcf2]: (4, ('/dev/sda1\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:08:26.794920 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:08:26.826639 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.029s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:08:26.828274 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[733fbf6b-7858-40b4-b77c-df8de73af739]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:08:26.829659 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:08:26.841024 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.011s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:08:26.841427 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[3009bc86-f475-4bd8-b98e-71b0eb21110d]: (4, ('8e294445-14da-a449-a0d4-86f2c95f64cd\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:08:26.843836 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] Running cmd (subprocess): nvme version {{(pid=134663) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:08:26.871987 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] CMD "nvme version" returned: 0 in 0.028s {{(pid=134663) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:08:26.874032 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134663) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 05 21:08:26.874520 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134663) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 05 21:08:26.875703 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:71cf25bc-00fa-4fd3-8614-fee15d100887 dsc: {{(pid=134663) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 05 21:08:26.876435 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.utils [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] <== get_connector_properties: return (166ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.15', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ab336d69f798', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '8e294445-14da-a449-a0d4-86f2c95f64cd', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:71cf25bc-00fa-4fd3-8614-fee15d100887', 'found_dsc': ''} {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 21:08:27.946114 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connector [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] Factory for rbd on None {{(pid=134663) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 05 21:08:27.947837 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-58a73208-a97d-4148-854a-a7d6520bb60b', 'hosts': ['10.222.0.15'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '58a73208-a97d-4148-854a-a7d6520bb60b', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 21:08:27.949499 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.linuxrbd [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] opening connection to ceph cluster (timeout=-1). {{(pid=134663) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 05 21:08:28.060789 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] <== connect_volume: return (112ms) {'path': } {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 21:08:28.074510 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_db.sqlalchemy.engines [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134663) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 05 21:08:28.091584 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] Starting restore from Ceph backup=e6dd2589-6441-4a5d-8b21-aca7be541bc9 to volume=volume-58a73208-a97d-4148-854a-a7d6520bb60b {{(pid=134663) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1274}} Mar 05 21:08:28.197243 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] Looking for snapshot of backup base 'volume-b22e1911-a4e8-47cf-863e-0a7a04265855.backup.e6dd2589-6441-4a5d-8b21-aca7be541bc9' {{(pid=134663) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 05 21:08:28.197681 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] Found snapshot 'backup.e6dd2589-6441-4a5d-8b21-aca7be541bc9.snap.1646514495.696748' {{(pid=134663) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Mar 05 21:08:28.203698 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] Volume file is RBD. {{(pid=134663) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1184}} Mar 05 21:08:28.262378 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] Attempting differential restore. {{(pid=134663) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1234}} Mar 05 21:08:28.263182 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] Attempting incremental restore from base='volume-b22e1911-a4e8-47cf-863e-0a7a04265855.backup.e6dd2589-6441-4a5d-8b21-aca7be541bc9' snap='backup.e6dd2589-6441-4a5d-8b21-aca7be541bc9.snap.1646514495.696748' {{(pid=134663) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1089}} Mar 05 21:08:28.263806 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] Performing differential transfer from 'volume-b22e1911-a4e8-47cf-863e-0a7a04265855.backup.e6dd2589-6441-4a5d-8b21-aca7be541bc9' to 'volume-58a73208-a97d-4148-854a-a7d6520bb60b' {{(pid=134663) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Mar 05 21:08:28.264445 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-b22e1911-a4e8-47cf-863e-0a7a04265855.backup.e6dd2589-6441-4a5d-8b21-aca7be541bc9@backup.e6dd2589-6441-4a5d-8b21-aca7be541bc9.snap.1646514495.696748 -' into... {{(pid=134663) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Mar 05 21:08:28.265017 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_atcl7y5h --pool volumes - volumes/volume-58a73208-a97d-4148-854a-a7d6520bb60b' {{(pid=134663) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Mar 05 21:08:34.323447 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] Restore transfer completed in 6.0586s {{(pid=134663) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1111}} Mar 05 21:08:34.324505 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=134663) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1285}} Mar 05 21:08:34.388725 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] Restoring volume base metadata {{(pid=134663) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} Mar 05 21:08:34.418886 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] No metadata of type 'volume-metadata' to restore {{(pid=134663) put /opt/stack/cinder/cinder/backup/driver.py:345}} Mar 05 21:08:34.419553 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] Restoring volume glance metadata {{(pid=134663) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}} Mar 05 21:08:34.737047 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] Restore to volume 58a73208-a97d-4148-854a-a7d6520bb60b finished successfully. {{(pid=134663) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1292}} Mar 05 21:08:34.739527 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-58a73208-a97d-4148-854a-a7d6520bb60b', 'hosts': ['10.222.0.15'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '58a73208-a97d-4148-854a-a7d6520bb60b', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 21:08:34.773893 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] <== disconnect_volume: return (33ms) None {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 21:08:34.832445 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.manager [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] Encryption key ID for volume 58a73208-a97d-4148-854a-a7d6520bb60b already matches encryption key ID in backup e6dd2589-6441-4a5d-8b21-aca7be541bc9. {{(pid=134663) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} Mar 05 21:08:34.888591 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-4f85034d-c0a9-4fc3-810e-5d4bc8b2b9ce tempest-TestVolumeBackupRestore-590870803 None] Finished restoring backup e6dd2589-6441-4a5d-8b21-aca7be541bc9 to volume 58a73208-a97d-4148-854a-a7d6520bb60b. Mar 05 21:08:49.548551 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:08:49.548551 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:09:30.097787 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] Create backup started, backup: abb3d0ba-0786-4b6f-90fa-1ca659bb2e19 volume: 7415431b-6312-4edb-9a69-a3a621dd7ad5. Mar 05 21:09:30.123573 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-03-05T21:09:29Z,data_timestamp=2022-03-05T21:09:29Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsAdminTest-Backup-1008800290',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=abb3d0ba-0786-4b6f-90fa-1ca659bb2e19,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='0f4d92586d654a0ab7a99a2f8764bf96',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='47d19c4547d842da92fb1d989788c632',volume_id=7415431b-6312-4edb-9a69-a3a621dd7ad5) Mar 05 21:09:30.269692 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.utils [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.15', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 21:09:30.275318 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:09:30.301449 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.027s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:09:30.302087 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[fb71f8f0-7121-47e5-897e-cc1e65629d31]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ab336d69f798\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:09:30.306762 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:09:30.325995 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 05 21:09:30.326672 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[a1583b09-c1a4-4b15-a420-53cf205290da]: [Errno 2] No such file or directory: 'systool' {{(pid=151032) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 05 21:09:30.326672 ncdv-master-1 cinder-backup[134663]: Traceback (most recent call last): Mar 05 21:09:30.326672 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 05 21:09:30.326672 ncdv-master-1 cinder-backup[134663]: ret = func(*f_args, **f_kwargs) Mar 05 21:09:30.326672 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 05 21:09:30.326672 ncdv-master-1 cinder-backup[134663]: return func(*args, **kwargs) Mar 05 21:09:30.326672 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 05 21:09:30.326672 ncdv-master-1 cinder-backup[134663]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 05 21:09:30.326672 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 05 21:09:30.326672 ncdv-master-1 cinder-backup[134663]: return putils.execute(on_execute=on_execute, Mar 05 21:09:30.326672 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 05 21:09:30.326672 ncdv-master-1 cinder-backup[134663]: obj = subprocess.Popen(cmd, Mar 05 21:09:30.326672 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 05 21:09:30.326672 ncdv-master-1 cinder-backup[134663]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 05 21:09:30.326672 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 05 21:09:30.326672 ncdv-master-1 cinder-backup[134663]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 05 21:09:30.326672 ncdv-master-1 cinder-backup[134663]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 05 21:09:30.329109 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[a1583b09-c1a4-4b15-a420-53cf205290da]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:09:30.332307 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:09:30.364848 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 05 21:09:30.365693 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[b93a414b-5cdc-44a2-93b1-008876e77b8b]: [Errno 2] No such file or directory: 'systool' {{(pid=151032) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 05 21:09:30.365693 ncdv-master-1 cinder-backup[134663]: Traceback (most recent call last): Mar 05 21:09:30.365693 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 05 21:09:30.365693 ncdv-master-1 cinder-backup[134663]: ret = func(*f_args, **f_kwargs) Mar 05 21:09:30.365693 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 05 21:09:30.365693 ncdv-master-1 cinder-backup[134663]: return func(*args, **kwargs) Mar 05 21:09:30.365693 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 05 21:09:30.365693 ncdv-master-1 cinder-backup[134663]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 05 21:09:30.365693 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 05 21:09:30.365693 ncdv-master-1 cinder-backup[134663]: return putils.execute(on_execute=on_execute, Mar 05 21:09:30.365693 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 05 21:09:30.365693 ncdv-master-1 cinder-backup[134663]: obj = subprocess.Popen(cmd, Mar 05 21:09:30.365693 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 05 21:09:30.365693 ncdv-master-1 cinder-backup[134663]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 05 21:09:30.365693 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 05 21:09:30.365693 ncdv-master-1 cinder-backup[134663]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 05 21:09:30.365693 ncdv-master-1 cinder-backup[134663]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 05 21:09:30.367506 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[b93a414b-5cdc-44a2-93b1-008876e77b8b]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:09:30.369991 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:09:30.407936 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.037s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:09:30.408595 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[ed2c76a7-83fb-46e9-8d65-ca28c180656c]: (4, ('/dev/sda1\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:09:30.409928 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:09:30.449985 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.039s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:09:30.450405 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[06c9f6ae-6497-462c-80d6-dab01aea14db]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:09:30.454644 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:09:30.491478 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.039s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:09:30.492120 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[59e47170-6be5-40cd-ab0f-300f488c2575]: (4, ('8e294445-14da-a449-a0d4-86f2c95f64cd\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:09:30.493197 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] Running cmd (subprocess): nvme version {{(pid=134663) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:09:30.568360 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] CMD "nvme version" returned: 0 in 0.074s {{(pid=134663) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:09:30.570548 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134663) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 05 21:09:30.571116 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134663) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 05 21:09:30.571947 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:71cf25bc-00fa-4fd3-8614-fee15d100887 dsc: {{(pid=134663) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 05 21:09:30.572850 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.utils [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] <== get_connector_properties: return (299ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.15', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ab336d69f798', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '8e294445-14da-a449-a0d4-86f2c95f64cd', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:71cf25bc-00fa-4fd3-8614-fee15d100887', 'found_dsc': ''} {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 21:09:30.736274 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_db.sqlalchemy.engines [None req-ca790f2e-02b0-4b56-8816-8cf551d2bc00 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134663) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 05 21:09:32.209171 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connector [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] Factory for rbd on None {{(pid=134663) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 05 21:09:32.211493 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-7415431b-6312-4edb-9a69-a3a621dd7ad5', 'hosts': ['10.222.0.15'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '7415431b-6312-4edb-9a69-a3a621dd7ad5', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 21:09:32.218508 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.linuxrbd [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] opening connection to ceph cluster (timeout=-1). {{(pid=134663) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 05 21:09:32.405039 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] <== connect_volume: return (192ms) {'path': } {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 21:09:32.465867 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] Starting backup of volume='7415431b-6312-4edb-9a69-a3a621dd7ad5'. {{(pid=134663) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Mar 05 21:09:32.472217 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] Volume file is RBD: attempting optimized backup {{(pid=134663) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Mar 05 21:09:32.545197 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] Creating base image 'volume-7415431b-6312-4edb-9a69-a3a621dd7ad5.backup.abb3d0ba-0786-4b6f-90fa-1ca659bb2e19' {{(pid=134663) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Mar 05 21:09:32.545686 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134663) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Mar 05 21:09:32.634028 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] Using --from-snap 'None' for incremental backup of volume 7415431b-6312-4edb-9a69-a3a621dd7ad5. {{(pid=134663) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Mar 05 21:09:32.634760 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] Creating backup snapshot='backup.abb3d0ba-0786-4b6f-90fa-1ca659bb2e19.snap.1646514572.6343534' {{(pid=134663) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Mar 05 21:09:33.273016 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] Performing differential transfer from 'volume-7415431b-6312-4edb-9a69-a3a621dd7ad5' to 'volume-7415431b-6312-4edb-9a69-a3a621dd7ad5.backup.abb3d0ba-0786-4b6f-90fa-1ca659bb2e19' {{(pid=134663) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Mar 05 21:09:33.274775 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_1ncx99yk --pool volumes volumes/volume-7415431b-6312-4edb-9a69-a3a621dd7ad5@backup.abb3d0ba-0786-4b6f-90fa-1ca659bb2e19.snap.1646514572.6343534 -' into... {{(pid=134663) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Mar 05 21:09:33.275359 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-7415431b-6312-4edb-9a69-a3a621dd7ad5.backup.abb3d0ba-0786-4b6f-90fa-1ca659bb2e19' {{(pid=134663) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Mar 05 21:09:34.447809 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] Differential backup transfer completed in 1.1743s {{(pid=134663) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Mar 05 21:09:34.448636 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] Getting metadata type 'volume-base-metadata' {{(pid=134663) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Mar 05 21:09:34.474833 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134663) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Mar 05 21:09:34.481131 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] Getting metadata type 'volume-metadata' {{(pid=134663) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Mar 05 21:09:34.497052 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] No metadata type 'volume-metadata' available {{(pid=134663) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Mar 05 21:09:34.497700 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] Getting metadata type 'volume-glance-metadata' {{(pid=134663) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Mar 05 21:09:34.515648 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] No metadata type 'volume-glance-metadata' available {{(pid=134663) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Mar 05 21:09:34.516813 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] Backing up metadata for volume 7415431b-6312-4edb-9a69-a3a621dd7ad5. {{(pid=134663) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Mar 05 21:09:34.607328 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] Backup 'abb3d0ba-0786-4b6f-90fa-1ca659bb2e19' of volume 7415431b-6312-4edb-9a69-a3a621dd7ad5 finished. {{(pid=134663) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Mar 05 21:09:34.610609 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-7415431b-6312-4edb-9a69-a3a621dd7ad5', 'hosts': ['10.222.0.15'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '7415431b-6312-4edb-9a69-a3a621dd7ad5', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 21:09:34.681429 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] <== disconnect_volume: return (69ms) None {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 21:09:34.800528 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-338271a6-50fa-4ce9-ab70-39e1703ebf7d tempest-VolumesBackupsAdminTest-1154175086 None] Create backup finished. backup: abb3d0ba-0786-4b6f-90fa-1ca659bb2e19. Mar 05 21:09:36.023266 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-95a6a34a-e780-4c85-85e1-7f61c4619ac6 tempest-VolumesBackupsAdminTest-1017618328 None] Export record started, backup: abb3d0ba-0786-4b6f-90fa-1ca659bb2e19. Mar 05 21:09:36.032611 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-95a6a34a-e780-4c85-85e1-7f61c4619ac6 tempest-VolumesBackupsAdminTest-1017618328 None] Export record finished, backup abb3d0ba-0786-4b6f-90fa-1ca659bb2e19 exported. Mar 05 21:09:36.338684 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-04ca2ed2-e3a2-4d72-9a24-fe819fcaa7dc tempest-VolumesBackupsAdminTest-1017618328 None] Import record started, backup_url: eyJkcml2ZXJfaW5mbyI6IHt9LCAiaWQiOiAiNzljY2ZlNzQtNGIzOS00OTllLWJhMDktZDcxNDJlYmQ4MDA0IiwgInVzZXJfaWQiOiAiNDdkMTljNDU0N2Q4NDJkYTkyZmIxZDk4OTc4OGM2MzIiLCAicHJvamVjdF9pZCI6ICIwZjRkOTI1ODZkNjU0YTBhYjdhOTlhMmY4NzY0YmY5NiIsICJ2b2x1bWVfaWQiOiAiNzQxNTQzMWItNjMxMi00ZWRiLTlhNjktYTNhNjIxZGQ3YWQ1IiwgImhvc3QiOiAibmNkdi1tYXN0ZXItMSIsICJhdmFpbGFiaWxpdHlfem9uZSI6IG51bGwsICJjb250YWluZXIiOiAiYmFja3VwcyIsICJwYXJlbnRfaWQiOiBudWxsLCAic3RhdHVzIjogImF2YWlsYWJsZSIsICJmYWlsX3JlYXNvbiI6IG51bGwsICJzaXplIjogMSwgImRpc3BsYXlfbmFtZSI6ICJ0ZW1wZXN0LVZvbHVtZXNCYWNrdXBzQWRtaW5UZXN0LUJhY2t1cC0xMDA4ODAwMjkwIiwgImRpc3BsYXlfZGVzY3JpcHRpb24iOiBudWxsLCAic2VydmljZV9tZXRhZGF0YSI6ICJ7XCJiYXNlXCI6IFwidm9sdW1lLTc0MTU0MzFiLTYzMTItNGVkYi05YTY5LWEzYTYyMWRkN2FkNS5iYWNrdXAuYWJiM2QwYmEtMDc4Ni00YjZmLTkwZmEtMWNhNjU5YmIyZTE5XCJ9IiwgInNlcnZpY2UiOiAiY2luZGVyLmJhY2t1cC5kcml2ZXJzLmNlcGguQ2VwaEJhY2t1cERyaXZlciIsICJvYmplY3RfY291bnQiOiAwLCAidGVtcF92b2x1bWVfaWQiOiBudWxsLCAidGVtcF9zbmFwc2hvdF9pZCI6IG51bGwsICJudW1fZGVwZW5kZW50X2JhY2t1cHMiOiAwLCAic25hcHNob3RfaWQiOiBudWxsLCAiZGF0YV90aW1lc3RhbXAiOiAiMjAyMi0wMy0wNVQyMTowOTozMFoiLCAicmVzdG9yZV92b2x1bWVfaWQiOiBudWxsLCAibWV0YWRhdGEiOiB7fSwgImVuY3J5cHRpb25fa2V5X2lkIjogbnVsbCwgImNyZWF0ZWRfYXQiOiAiMjAyMi0wMy0wNVQyMTowOTozMFoiLCAidXBkYXRlZF9hdCI6ICIyMDIyLTAzLTA1VDIxOjA5OjM1WiIsICJkZWxldGVkX2F0IjogbnVsbCwgImRlbGV0ZWQiOiBmYWxzZX0=. Mar 05 21:09:36.399257 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_db.sqlalchemy.engines [None req-04ca2ed2-e3a2-4d72-9a24-fe819fcaa7dc tempest-VolumesBackupsAdminTest-1017618328 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134663) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 05 21:09:36.495946 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-04ca2ed2-e3a2-4d72-9a24-fe819fcaa7dc tempest-VolumesBackupsAdminTest-1017618328 None] Import record id 79ccfe74-4b39-499e-ba09-d7142ebd8004 metadata from driver finished. Mar 05 21:09:39.093197 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] Restore backup started, backup: abb3d0ba-0786-4b6f-90fa-1ca659bb2e19 volume: 010285b6-3707-4982-bf57-a118b6f26b0c. Mar 05 21:09:39.102774 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.utils [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.15', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 21:09:39.104261 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:09:39.130122 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.025s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:09:39.130652 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[ab9fd92d-706c-42ca-97d7-712e603132a1]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ab336d69f798\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:09:39.133262 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:09:39.149050 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 05 21:09:39.149671 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[04f8fe7a-d8da-449b-b15a-f8a3f09a3aca]: [Errno 2] No such file or directory: 'systool' {{(pid=151032) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 05 21:09:39.149671 ncdv-master-1 cinder-backup[134663]: Traceback (most recent call last): Mar 05 21:09:39.149671 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 05 21:09:39.149671 ncdv-master-1 cinder-backup[134663]: ret = func(*f_args, **f_kwargs) Mar 05 21:09:39.149671 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 05 21:09:39.149671 ncdv-master-1 cinder-backup[134663]: return func(*args, **kwargs) Mar 05 21:09:39.149671 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 05 21:09:39.149671 ncdv-master-1 cinder-backup[134663]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 05 21:09:39.149671 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 05 21:09:39.149671 ncdv-master-1 cinder-backup[134663]: return putils.execute(on_execute=on_execute, Mar 05 21:09:39.149671 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 05 21:09:39.149671 ncdv-master-1 cinder-backup[134663]: obj = subprocess.Popen(cmd, Mar 05 21:09:39.149671 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 05 21:09:39.149671 ncdv-master-1 cinder-backup[134663]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 05 21:09:39.149671 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 05 21:09:39.149671 ncdv-master-1 cinder-backup[134663]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 05 21:09:39.149671 ncdv-master-1 cinder-backup[134663]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 05 21:09:39.156261 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[04f8fe7a-d8da-449b-b15a-f8a3f09a3aca]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:09:39.158677 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:09:39.185025 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 05 21:09:39.185325 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[139be929-aa66-419e-98e6-cdf7f13a205b]: [Errno 2] No such file or directory: 'systool' {{(pid=151032) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 05 21:09:39.185325 ncdv-master-1 cinder-backup[134663]: Traceback (most recent call last): Mar 05 21:09:39.185325 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 05 21:09:39.185325 ncdv-master-1 cinder-backup[134663]: ret = func(*f_args, **f_kwargs) Mar 05 21:09:39.185325 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 05 21:09:39.185325 ncdv-master-1 cinder-backup[134663]: return func(*args, **kwargs) Mar 05 21:09:39.185325 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 05 21:09:39.185325 ncdv-master-1 cinder-backup[134663]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 05 21:09:39.185325 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 05 21:09:39.185325 ncdv-master-1 cinder-backup[134663]: return putils.execute(on_execute=on_execute, Mar 05 21:09:39.185325 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 05 21:09:39.185325 ncdv-master-1 cinder-backup[134663]: obj = subprocess.Popen(cmd, Mar 05 21:09:39.185325 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 05 21:09:39.185325 ncdv-master-1 cinder-backup[134663]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 05 21:09:39.185325 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 05 21:09:39.185325 ncdv-master-1 cinder-backup[134663]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 05 21:09:39.185325 ncdv-master-1 cinder-backup[134663]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 05 21:09:39.189060 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[139be929-aa66-419e-98e6-cdf7f13a205b]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:09:39.192125 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:09:39.205692 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.014s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:09:39.206221 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[48d719a5-2a21-4ced-87da-1ae4c806aa94]: (4, ('/dev/sda1\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:09:39.210129 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:09:39.249148 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.036s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:09:39.250518 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[7d23d692-e0e4-44b6-82fb-1d03a9abe0f6]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:09:39.252399 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:09:39.274992 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.021s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:09:39.275784 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[e8a3cf84-b801-4224-8d97-2c4926e42a7e]: (4, ('8e294445-14da-a449-a0d4-86f2c95f64cd\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:09:39.276916 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] Running cmd (subprocess): nvme version {{(pid=134663) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:09:39.302757 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] CMD "nvme version" returned: 0 in 0.025s {{(pid=134663) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:09:39.305225 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134663) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 05 21:09:39.306853 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134663) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 05 21:09:39.307304 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:71cf25bc-00fa-4fd3-8614-fee15d100887 dsc: {{(pid=134663) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 05 21:09:39.308038 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.utils [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] <== get_connector_properties: return (204ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.15', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ab336d69f798', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '8e294445-14da-a449-a0d4-86f2c95f64cd', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:71cf25bc-00fa-4fd3-8614-fee15d100887', 'found_dsc': ''} {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 21:09:40.252753 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connector [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] Factory for rbd on None {{(pid=134663) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 05 21:09:40.255031 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-010285b6-3707-4982-bf57-a118b6f26b0c', 'hosts': ['10.222.0.15'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '010285b6-3707-4982-bf57-a118b6f26b0c', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 21:09:40.263015 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.linuxrbd [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] opening connection to ceph cluster (timeout=-1). {{(pid=134663) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 05 21:09:40.365420 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] <== connect_volume: return (109ms) {'path': } {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 21:09:40.376427 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_db.sqlalchemy.engines [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134663) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 05 21:09:40.428623 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] Starting restore from Ceph backup=abb3d0ba-0786-4b6f-90fa-1ca659bb2e19 to volume=volume-010285b6-3707-4982-bf57-a118b6f26b0c {{(pid=134663) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1274}} Mar 05 21:09:40.638070 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] Looking for snapshot of backup base 'volume-7415431b-6312-4edb-9a69-a3a621dd7ad5.backup.abb3d0ba-0786-4b6f-90fa-1ca659bb2e19' {{(pid=134663) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 05 21:09:40.639380 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] Found snapshot 'backup.abb3d0ba-0786-4b6f-90fa-1ca659bb2e19.snap.1646514572.6343534' {{(pid=134663) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Mar 05 21:09:40.652913 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] Volume file is RBD. {{(pid=134663) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1184}} Mar 05 21:09:40.755265 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] Attempting differential restore. {{(pid=134663) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1234}} Mar 05 21:09:40.755926 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] Attempting incremental restore from base='volume-7415431b-6312-4edb-9a69-a3a621dd7ad5.backup.abb3d0ba-0786-4b6f-90fa-1ca659bb2e19' snap='backup.abb3d0ba-0786-4b6f-90fa-1ca659bb2e19.snap.1646514572.6343534' {{(pid=134663) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1089}} Mar 05 21:09:40.757055 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] Performing differential transfer from 'volume-7415431b-6312-4edb-9a69-a3a621dd7ad5.backup.abb3d0ba-0786-4b6f-90fa-1ca659bb2e19' to 'volume-010285b6-3707-4982-bf57-a118b6f26b0c' {{(pid=134663) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Mar 05 21:09:40.758590 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-7415431b-6312-4edb-9a69-a3a621dd7ad5.backup.abb3d0ba-0786-4b6f-90fa-1ca659bb2e19@backup.abb3d0ba-0786-4b6f-90fa-1ca659bb2e19.snap.1646514572.6343534 -' into... {{(pid=134663) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Mar 05 21:09:40.759202 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_z8vrn5gb --pool volumes - volumes/volume-010285b6-3707-4982-bf57-a118b6f26b0c' {{(pid=134663) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Mar 05 21:09:41.407901 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] Restore transfer completed in 0.6508s {{(pid=134663) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1111}} Mar 05 21:09:41.409195 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=134663) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1285}} Mar 05 21:09:41.477791 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] Restoring volume base metadata {{(pid=134663) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} Mar 05 21:09:41.510891 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] No metadata of type 'volume-metadata' to restore {{(pid=134663) put /opt/stack/cinder/cinder/backup/driver.py:345}} Mar 05 21:09:41.511536 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] No metadata of type 'volume-glance-metadata' to restore {{(pid=134663) put /opt/stack/cinder/cinder/backup/driver.py:345}} Mar 05 21:09:41.522245 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] Restore to volume 010285b6-3707-4982-bf57-a118b6f26b0c finished successfully. {{(pid=134663) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1292}} Mar 05 21:09:41.525538 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-010285b6-3707-4982-bf57-a118b6f26b0c', 'hosts': ['10.222.0.15'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '010285b6-3707-4982-bf57-a118b6f26b0c', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 21:09:41.554975 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] <== disconnect_volume: return (28ms) None {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 21:09:41.655049 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.manager [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] Encryption key ID for volume 010285b6-3707-4982-bf57-a118b6f26b0c already matches encryption key ID in backup abb3d0ba-0786-4b6f-90fa-1ca659bb2e19. {{(pid=134663) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} Mar 05 21:09:41.788529 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-8b06cde2-c7ab-4397-b5bc-dedda5124ce8 tempest-VolumesBackupsAdminTest-1154175086 None] Finished restoring backup abb3d0ba-0786-4b6f-90fa-1ca659bb2e19 to volume 010285b6-3707-4982-bf57-a118b6f26b0c. Mar 05 21:09:42.606001 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-130fe4e3-032c-4e08-b739-02f54a557bf2 tempest-VolumesBackupsAdminTest-1017618328 None] Delete backup started, backup: 79ccfe74-4b39-499e-ba09-d7142ebd8004. Mar 05 21:09:42.612984 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-130fe4e3-032c-4e08-b739-02f54a557bf2 tempest-VolumesBackupsAdminTest-1017618328 None] Delete started for backup=79ccfe74-4b39-499e-ba09-d7142ebd8004 {{(pid=134663) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Mar 05 21:09:42.617496 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-130fe4e3-032c-4e08-b739-02f54a557bf2 tempest-VolumesBackupsAdminTest-1017618328 None] Trying diff format basename='volume-7415431b-6312-4edb-9a69-a3a621dd7ad5.backup.abb3d0ba-0786-4b6f-90fa-1ca659bb2e19' for backup base image of volume 7415431b-6312-4edb-9a69-a3a621dd7ad5. {{(pid=134663) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Mar 05 21:09:42.755534 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-130fe4e3-032c-4e08-b739-02f54a557bf2 tempest-VolumesBackupsAdminTest-1017618328 None] Looking for snapshot of backup base 'volume-7415431b-6312-4edb-9a69-a3a621dd7ad5.backup.abb3d0ba-0786-4b6f-90fa-1ca659bb2e19' {{(pid=134663) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 05 21:09:42.756148 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-130fe4e3-032c-4e08-b739-02f54a557bf2 tempest-VolumesBackupsAdminTest-1017618328 None] Backup '79ccfe74-4b39-499e-ba09-d7142ebd8004' has no snapshot {{(pid=134663) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:915}} Mar 05 21:09:42.756692 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-130fe4e3-032c-4e08-b739-02f54a557bf2 tempest-VolumesBackupsAdminTest-1017618328 None] No backup snapshot to delete {{(pid=134663) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:476}} Mar 05 21:09:42.776570 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.drivers.ceph [None req-130fe4e3-032c-4e08-b739-02f54a557bf2 tempest-VolumesBackupsAdminTest-1017618328 None] Backup base image of volume 7415431b-6312-4edb-9a69-a3a621dd7ad5 still has 1 snapshots so skipping base image delete. Mar 05 21:09:42.880673 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-130fe4e3-032c-4e08-b739-02f54a557bf2 tempest-VolumesBackupsAdminTest-1017618328 None] Metadata backup object 'backup.79ccfe74-4b39-499e-ba09-d7142ebd8004.meta' not found - ignoring {{(pid=134663) remove_if_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:160}} Mar 05 21:09:42.884194 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-130fe4e3-032c-4e08-b739-02f54a557bf2 tempest-VolumesBackupsAdminTest-1017618328 None] Delete of backup '79ccfe74-4b39-499e-ba09-d7142ebd8004' for volume '7415431b-6312-4edb-9a69-a3a621dd7ad5' finished. {{(pid=134663) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Mar 05 21:09:42.889859 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_db.sqlalchemy.engines [None req-130fe4e3-032c-4e08-b739-02f54a557bf2 tempest-VolumesBackupsAdminTest-1017618328 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134663) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 05 21:09:42.938467 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.quota [None req-130fe4e3-032c-4e08-b739-02f54a557bf2 tempest-VolumesBackupsAdminTest-1017618328 None] Created reservations ['bdb43691-2461-420b-aa9e-ab6c5cbed6bc', '697ebf51-6b0b-43d4-8f6c-89f6af3785b9'] {{(pid=134663) reserve /opt/stack/cinder/cinder/quota.py:755}} Mar 05 21:09:42.982101 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-130fe4e3-032c-4e08-b739-02f54a557bf2 tempest-VolumesBackupsAdminTest-1017618328 None] Delete backup finished, backup 79ccfe74-4b39-499e-ba09-d7142ebd8004 deleted. Mar 05 21:09:43.861469 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-7758e005-7068-4dbe-912d-1386d8e00679 tempest-VolumesBackupsAdminTest-1154175086 None] Delete backup started, backup: abb3d0ba-0786-4b6f-90fa-1ca659bb2e19. Mar 05 21:09:43.864622 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-7758e005-7068-4dbe-912d-1386d8e00679 tempest-VolumesBackupsAdminTest-1154175086 None] Delete started for backup=abb3d0ba-0786-4b6f-90fa-1ca659bb2e19 {{(pid=134663) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Mar 05 21:09:43.865060 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-7758e005-7068-4dbe-912d-1386d8e00679 tempest-VolumesBackupsAdminTest-1154175086 None] Trying diff format basename='volume-7415431b-6312-4edb-9a69-a3a621dd7ad5.backup.abb3d0ba-0786-4b6f-90fa-1ca659bb2e19' for backup base image of volume 7415431b-6312-4edb-9a69-a3a621dd7ad5. {{(pid=134663) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Mar 05 21:09:43.980385 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-7758e005-7068-4dbe-912d-1386d8e00679 tempest-VolumesBackupsAdminTest-1154175086 None] Looking for snapshot of backup base 'volume-7415431b-6312-4edb-9a69-a3a621dd7ad5.backup.abb3d0ba-0786-4b6f-90fa-1ca659bb2e19' {{(pid=134663) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 05 21:09:43.981035 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-7758e005-7068-4dbe-912d-1386d8e00679 tempest-VolumesBackupsAdminTest-1154175086 None] Found snapshot 'backup.abb3d0ba-0786-4b6f-90fa-1ca659bb2e19.snap.1646514572.6343534' {{(pid=134663) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Mar 05 21:09:43.981582 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-7758e005-7068-4dbe-912d-1386d8e00679 tempest-VolumesBackupsAdminTest-1154175086 None] Deleting backup snapshot='backup.abb3d0ba-0786-4b6f-90fa-1ca659bb2e19.snap.1646514572.6343534' {{(pid=134663) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Mar 05 21:09:44.429546 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.drivers.ceph [None req-7758e005-7068-4dbe-912d-1386d8e00679 tempest-VolumesBackupsAdminTest-1154175086 None] Deleting backup base image='volume-7415431b-6312-4edb-9a69-a3a621dd7ad5.backup.abb3d0ba-0786-4b6f-90fa-1ca659bb2e19' of volume 7415431b-6312-4edb-9a69-a3a621dd7ad5. Mar 05 21:09:44.985869 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-7758e005-7068-4dbe-912d-1386d8e00679 tempest-VolumesBackupsAdminTest-1154175086 None] Base backup image='volume-7415431b-6312-4edb-9a69-a3a621dd7ad5.backup.abb3d0ba-0786-4b6f-90fa-1ca659bb2e19' of volume 7415431b-6312-4edb-9a69-a3a621dd7ad5 deleted. {{(pid=134663) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Mar 05 21:09:45.087549 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-7758e005-7068-4dbe-912d-1386d8e00679 tempest-VolumesBackupsAdminTest-1154175086 None] Delete of backup 'abb3d0ba-0786-4b6f-90fa-1ca659bb2e19' for volume '7415431b-6312-4edb-9a69-a3a621dd7ad5' finished. {{(pid=134663) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Mar 05 21:09:45.096445 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_db.sqlalchemy.engines [None req-7758e005-7068-4dbe-912d-1386d8e00679 tempest-VolumesBackupsAdminTest-1154175086 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134663) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 05 21:09:45.142021 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.quota [None req-7758e005-7068-4dbe-912d-1386d8e00679 tempest-VolumesBackupsAdminTest-1154175086 None] Created reservations ['dc633d8d-89ea-4a94-ba8e-397f90ab2623', 'e75ce865-661e-474b-9632-e335312542a5'] {{(pid=134663) reserve /opt/stack/cinder/cinder/quota.py:755}} Mar 05 21:09:45.196165 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-7758e005-7068-4dbe-912d-1386d8e00679 tempest-VolumesBackupsAdminTest-1154175086 None] Delete backup finished, backup abb3d0ba-0786-4b6f-90fa-1ca659bb2e19 deleted. Mar 05 21:09:47.793155 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] Create backup started, backup: 2a50b36e-fb78-4798-a0f2-627dd997003e volume: d62e8383-bc33-4341-9525-48f0aef7a585. Mar 05 21:09:47.809931 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-03-05T21:09:47Z,data_timestamp=2022-03-05T21:09:47Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsAdminTest-Backup-656813056',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=2a50b36e-fb78-4798-a0f2-627dd997003e,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='0f4d92586d654a0ab7a99a2f8764bf96',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='47d19c4547d842da92fb1d989788c632',volume_id=d62e8383-bc33-4341-9525-48f0aef7a585) Mar 05 21:09:47.948781 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.utils [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.15', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 21:09:47.958498 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:09:48.001863 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.044s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:09:48.003061 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[b5d6e75f-8173-46d4-b888-d457e3bf3cbb]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ab336d69f798\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:09:48.006671 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:09:48.044233 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 05 21:09:48.044719 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[7c0c8724-b62d-47a5-ad48-403229226d9a]: [Errno 2] No such file or directory: 'systool' {{(pid=151032) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 05 21:09:48.044719 ncdv-master-1 cinder-backup[134663]: Traceback (most recent call last): Mar 05 21:09:48.044719 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 05 21:09:48.044719 ncdv-master-1 cinder-backup[134663]: ret = func(*f_args, **f_kwargs) Mar 05 21:09:48.044719 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 05 21:09:48.044719 ncdv-master-1 cinder-backup[134663]: return func(*args, **kwargs) Mar 05 21:09:48.044719 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 05 21:09:48.044719 ncdv-master-1 cinder-backup[134663]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 05 21:09:48.044719 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 05 21:09:48.044719 ncdv-master-1 cinder-backup[134663]: return putils.execute(on_execute=on_execute, Mar 05 21:09:48.044719 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 05 21:09:48.044719 ncdv-master-1 cinder-backup[134663]: obj = subprocess.Popen(cmd, Mar 05 21:09:48.044719 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 05 21:09:48.044719 ncdv-master-1 cinder-backup[134663]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 05 21:09:48.044719 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 05 21:09:48.044719 ncdv-master-1 cinder-backup[134663]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 05 21:09:48.044719 ncdv-master-1 cinder-backup[134663]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 05 21:09:48.047341 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[7c0c8724-b62d-47a5-ad48-403229226d9a]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:09:48.051241 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:09:48.069496 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 05 21:09:48.069933 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[904461de-3fca-4872-be4c-e5c64b4bc184]: [Errno 2] No such file or directory: 'systool' {{(pid=151032) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 05 21:09:48.069933 ncdv-master-1 cinder-backup[134663]: Traceback (most recent call last): Mar 05 21:09:48.069933 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 05 21:09:48.069933 ncdv-master-1 cinder-backup[134663]: ret = func(*f_args, **f_kwargs) Mar 05 21:09:48.069933 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 05 21:09:48.069933 ncdv-master-1 cinder-backup[134663]: return func(*args, **kwargs) Mar 05 21:09:48.069933 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 05 21:09:48.069933 ncdv-master-1 cinder-backup[134663]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 05 21:09:48.069933 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 05 21:09:48.069933 ncdv-master-1 cinder-backup[134663]: return putils.execute(on_execute=on_execute, Mar 05 21:09:48.069933 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 05 21:09:48.069933 ncdv-master-1 cinder-backup[134663]: obj = subprocess.Popen(cmd, Mar 05 21:09:48.069933 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 05 21:09:48.069933 ncdv-master-1 cinder-backup[134663]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 05 21:09:48.069933 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 05 21:09:48.069933 ncdv-master-1 cinder-backup[134663]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 05 21:09:48.069933 ncdv-master-1 cinder-backup[134663]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 05 21:09:48.071300 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[904461de-3fca-4872-be4c-e5c64b4bc184]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:09:48.074666 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:09:48.108423 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.035s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:09:48.109690 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[cd4dbb6e-8242-4577-91ec-1182a7a70216]: (4, ('/dev/sda1\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:09:48.111908 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:09:48.162826 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.051s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:09:48.163209 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[d949ad86-e853-4ae2-8f55-2372a554dbf5]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:09:48.164553 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:09:48.198266 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.033s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:09:48.198639 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[adfdda38-8a8c-41ec-964a-6bb9e2d8593b]: (4, ('8e294445-14da-a449-a0d4-86f2c95f64cd\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:09:48.199539 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] Running cmd (subprocess): nvme version {{(pid=134663) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:09:48.261161 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] CMD "nvme version" returned: 0 in 0.061s {{(pid=134663) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:09:48.263919 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134663) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 05 21:09:48.264545 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134663) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 05 21:09:48.265802 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:71cf25bc-00fa-4fd3-8614-fee15d100887 dsc: {{(pid=134663) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 05 21:09:48.266819 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.utils [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] <== get_connector_properties: return (314ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.15', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ab336d69f798', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '8e294445-14da-a449-a0d4-86f2c95f64cd', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:71cf25bc-00fa-4fd3-8614-fee15d100887', 'found_dsc': ''} {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 21:09:49.307059 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connector [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] Factory for rbd on None {{(pid=134663) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 05 21:09:49.308811 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-d62e8383-bc33-4341-9525-48f0aef7a585', 'hosts': ['10.222.0.15'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'd62e8383-bc33-4341-9525-48f0aef7a585', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 21:09:49.309884 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.linuxrbd [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] opening connection to ceph cluster (timeout=-1). {{(pid=134663) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 05 21:09:49.384699 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] <== connect_volume: return (75ms) {'path': } {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 21:09:49.394324 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_db.sqlalchemy.engines [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134663) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 05 21:09:49.428994 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] Starting backup of volume='d62e8383-bc33-4341-9525-48f0aef7a585'. {{(pid=134663) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Mar 05 21:09:49.435745 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] Volume file is RBD: attempting optimized backup {{(pid=134663) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Mar 05 21:09:49.482039 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] Creating base image 'volume-d62e8383-bc33-4341-9525-48f0aef7a585.backup.2a50b36e-fb78-4798-a0f2-627dd997003e' {{(pid=134663) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Mar 05 21:09:49.483207 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134663) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Mar 05 21:09:49.548463 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:09:49.548822 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:09:49.580717 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] Using --from-snap 'None' for incremental backup of volume d62e8383-bc33-4341-9525-48f0aef7a585. {{(pid=134663) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Mar 05 21:09:49.581941 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] Creating backup snapshot='backup.2a50b36e-fb78-4798-a0f2-627dd997003e.snap.1646514589.5815482' {{(pid=134663) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Mar 05 21:09:50.016611 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] Performing differential transfer from 'volume-d62e8383-bc33-4341-9525-48f0aef7a585' to 'volume-d62e8383-bc33-4341-9525-48f0aef7a585.backup.2a50b36e-fb78-4798-a0f2-627dd997003e' {{(pid=134663) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Mar 05 21:09:50.017257 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_mx93mjc2 --pool volumes volumes/volume-d62e8383-bc33-4341-9525-48f0aef7a585@backup.2a50b36e-fb78-4798-a0f2-627dd997003e.snap.1646514589.5815482 -' into... {{(pid=134663) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Mar 05 21:09:50.018058 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-d62e8383-bc33-4341-9525-48f0aef7a585.backup.2a50b36e-fb78-4798-a0f2-627dd997003e' {{(pid=134663) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Mar 05 21:09:51.022208 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] Differential backup transfer completed in 1.0055s {{(pid=134663) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Mar 05 21:09:51.022836 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] Getting metadata type 'volume-base-metadata' {{(pid=134663) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Mar 05 21:09:51.037506 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134663) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Mar 05 21:09:51.037919 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] Getting metadata type 'volume-metadata' {{(pid=134663) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Mar 05 21:09:51.048957 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] No metadata type 'volume-metadata' available {{(pid=134663) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Mar 05 21:09:51.049831 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] Getting metadata type 'volume-glance-metadata' {{(pid=134663) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Mar 05 21:09:51.060987 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] No metadata type 'volume-glance-metadata' available {{(pid=134663) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Mar 05 21:09:51.062486 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] Backing up metadata for volume d62e8383-bc33-4341-9525-48f0aef7a585. {{(pid=134663) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Mar 05 21:09:51.163400 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] Backup '2a50b36e-fb78-4798-a0f2-627dd997003e' of volume d62e8383-bc33-4341-9525-48f0aef7a585 finished. {{(pid=134663) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Mar 05 21:09:51.165011 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-d62e8383-bc33-4341-9525-48f0aef7a585', 'hosts': ['10.222.0.15'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'd62e8383-bc33-4341-9525-48f0aef7a585', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 21:09:51.213925 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] <== disconnect_volume: return (46ms) None {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 21:09:51.307258 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-914b4cc8-619e-4e2c-bf74-a6438ec87881 tempest-VolumesBackupsAdminTest-1154175086 None] Create backup finished. backup: 2a50b36e-fb78-4798-a0f2-627dd997003e. Mar 05 21:09:52.270310 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-b2bad0d5-c55b-4fbd-a069-c716d40f5a23 tempest-VolumesBackupsAdminTest-1017618328 None] Reset backup status started, backup_id: 2a50b36e-fb78-4798-a0f2-627dd997003e, status: error. Mar 05 21:09:52.271419 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-b2bad0d5-c55b-4fbd-a069-c716d40f5a23 tempest-VolumesBackupsAdminTest-1017618328 None] Backup service: cinder.backup.drivers.ceph.CephBackupDriver. Mar 05 21:09:53.496063 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-609fb557-43d8-4958-adce-2a79b6b5150b tempest-VolumesBackupsAdminTest-1154175086 None] Delete backup started, backup: 2a50b36e-fb78-4798-a0f2-627dd997003e. Mar 05 21:09:53.504892 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-609fb557-43d8-4958-adce-2a79b6b5150b tempest-VolumesBackupsAdminTest-1154175086 None] Delete started for backup=2a50b36e-fb78-4798-a0f2-627dd997003e {{(pid=134663) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Mar 05 21:09:53.507159 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-609fb557-43d8-4958-adce-2a79b6b5150b tempest-VolumesBackupsAdminTest-1154175086 None] Trying diff format basename='volume-d62e8383-bc33-4341-9525-48f0aef7a585.backup.2a50b36e-fb78-4798-a0f2-627dd997003e' for backup base image of volume d62e8383-bc33-4341-9525-48f0aef7a585. {{(pid=134663) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Mar 05 21:09:53.613405 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-609fb557-43d8-4958-adce-2a79b6b5150b tempest-VolumesBackupsAdminTest-1154175086 None] Looking for snapshot of backup base 'volume-d62e8383-bc33-4341-9525-48f0aef7a585.backup.2a50b36e-fb78-4798-a0f2-627dd997003e' {{(pid=134663) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 05 21:09:53.614281 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-609fb557-43d8-4958-adce-2a79b6b5150b tempest-VolumesBackupsAdminTest-1154175086 None] Found snapshot 'backup.2a50b36e-fb78-4798-a0f2-627dd997003e.snap.1646514589.5815482' {{(pid=134663) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Mar 05 21:09:53.614852 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-609fb557-43d8-4958-adce-2a79b6b5150b tempest-VolumesBackupsAdminTest-1154175086 None] Deleting backup snapshot='backup.2a50b36e-fb78-4798-a0f2-627dd997003e.snap.1646514589.5815482' {{(pid=134663) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Mar 05 21:09:54.109967 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.drivers.ceph [None req-609fb557-43d8-4958-adce-2a79b6b5150b tempest-VolumesBackupsAdminTest-1154175086 None] Deleting backup base image='volume-d62e8383-bc33-4341-9525-48f0aef7a585.backup.2a50b36e-fb78-4798-a0f2-627dd997003e' of volume d62e8383-bc33-4341-9525-48f0aef7a585. Mar 05 21:09:54.726844 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-609fb557-43d8-4958-adce-2a79b6b5150b tempest-VolumesBackupsAdminTest-1154175086 None] Base backup image='volume-d62e8383-bc33-4341-9525-48f0aef7a585.backup.2a50b36e-fb78-4798-a0f2-627dd997003e' of volume d62e8383-bc33-4341-9525-48f0aef7a585 deleted. {{(pid=134663) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Mar 05 21:09:54.805146 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-609fb557-43d8-4958-adce-2a79b6b5150b tempest-VolumesBackupsAdminTest-1154175086 None] Delete of backup '2a50b36e-fb78-4798-a0f2-627dd997003e' for volume 'd62e8383-bc33-4341-9525-48f0aef7a585' finished. {{(pid=134663) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Mar 05 21:09:54.812107 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_db.sqlalchemy.engines [None req-609fb557-43d8-4958-adce-2a79b6b5150b tempest-VolumesBackupsAdminTest-1154175086 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134663) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 05 21:09:54.844144 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.quota [None req-609fb557-43d8-4958-adce-2a79b6b5150b tempest-VolumesBackupsAdminTest-1154175086 None] Created reservations ['47eb523f-4e71-4f5c-af88-9540b339636b', '25440c53-04fb-4080-b987-879d7c8c21ff'] {{(pid=134663) reserve /opt/stack/cinder/cinder/quota.py:755}} Mar 05 21:09:54.928603 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-609fb557-43d8-4958-adce-2a79b6b5150b tempest-VolumesBackupsAdminTest-1154175086 None] Delete backup finished, backup 2a50b36e-fb78-4798-a0f2-627dd997003e deleted. Mar 05 21:10:24.761734 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-280bf6cb-5fbe-4dca-aa31-64aac6b24ace tempest-TestVolumeBackupRestore-590870803 None] Delete backup started, backup: e6dd2589-6441-4a5d-8b21-aca7be541bc9. Mar 05 21:10:24.765126 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-280bf6cb-5fbe-4dca-aa31-64aac6b24ace tempest-TestVolumeBackupRestore-590870803 None] Delete started for backup=e6dd2589-6441-4a5d-8b21-aca7be541bc9 {{(pid=134663) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Mar 05 21:10:24.765577 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-280bf6cb-5fbe-4dca-aa31-64aac6b24ace tempest-TestVolumeBackupRestore-590870803 None] Trying diff format basename='volume-b22e1911-a4e8-47cf-863e-0a7a04265855.backup.e6dd2589-6441-4a5d-8b21-aca7be541bc9' for backup base image of volume b22e1911-a4e8-47cf-863e-0a7a04265855. {{(pid=134663) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Mar 05 21:10:24.888538 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-280bf6cb-5fbe-4dca-aa31-64aac6b24ace tempest-TestVolumeBackupRestore-590870803 None] Looking for snapshot of backup base 'volume-b22e1911-a4e8-47cf-863e-0a7a04265855.backup.e6dd2589-6441-4a5d-8b21-aca7be541bc9' {{(pid=134663) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 05 21:10:24.890067 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-280bf6cb-5fbe-4dca-aa31-64aac6b24ace tempest-TestVolumeBackupRestore-590870803 None] Found snapshot 'backup.e6dd2589-6441-4a5d-8b21-aca7be541bc9.snap.1646514495.696748' {{(pid=134663) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Mar 05 21:10:24.890687 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-280bf6cb-5fbe-4dca-aa31-64aac6b24ace tempest-TestVolumeBackupRestore-590870803 None] Deleting backup snapshot='backup.e6dd2589-6441-4a5d-8b21-aca7be541bc9.snap.1646514495.696748' {{(pid=134663) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Mar 05 21:10:25.470344 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.drivers.ceph [None req-280bf6cb-5fbe-4dca-aa31-64aac6b24ace tempest-TestVolumeBackupRestore-590870803 None] Deleting backup base image='volume-b22e1911-a4e8-47cf-863e-0a7a04265855.backup.e6dd2589-6441-4a5d-8b21-aca7be541bc9' of volume b22e1911-a4e8-47cf-863e-0a7a04265855. Mar 05 21:10:26.909704 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-280bf6cb-5fbe-4dca-aa31-64aac6b24ace tempest-TestVolumeBackupRestore-590870803 None] Base backup image='volume-b22e1911-a4e8-47cf-863e-0a7a04265855.backup.e6dd2589-6441-4a5d-8b21-aca7be541bc9' of volume b22e1911-a4e8-47cf-863e-0a7a04265855 deleted. {{(pid=134663) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Mar 05 21:10:27.026244 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-280bf6cb-5fbe-4dca-aa31-64aac6b24ace tempest-TestVolumeBackupRestore-590870803 None] Delete of backup 'e6dd2589-6441-4a5d-8b21-aca7be541bc9' for volume 'b22e1911-a4e8-47cf-863e-0a7a04265855' finished. {{(pid=134663) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Mar 05 21:10:27.035316 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_db.sqlalchemy.engines [None req-280bf6cb-5fbe-4dca-aa31-64aac6b24ace tempest-TestVolumeBackupRestore-590870803 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134663) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 05 21:10:27.165716 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.quota [None req-280bf6cb-5fbe-4dca-aa31-64aac6b24ace tempest-TestVolumeBackupRestore-590870803 None] Created reservations ['6844cd11-6faf-48b5-8895-a21eb7edc62a', 'c002badd-1822-4b50-ba6c-0b68395b0022'] {{(pid=134663) reserve /opt/stack/cinder/cinder/quota.py:755}} Mar 05 21:10:27.226241 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-280bf6cb-5fbe-4dca-aa31-64aac6b24ace tempest-TestVolumeBackupRestore-590870803 None] Delete backup finished, backup e6dd2589-6441-4a5d-8b21-aca7be541bc9 deleted. Mar 05 21:10:49.545044 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:10:49.545621 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:11:49.550539 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:11:49.551275 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:12:49.552976 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:12:49.554235 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:13:49.555380 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:13:49.558565 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:14:49.557755 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:14:49.558919 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:15:03.996642 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] Create backup started, backup: a273ab6d-b020-47de-876e-e182bf0c50d3 volume: 2de60bd3-fd9c-4cf2-a463-7d2086df54a0. Mar 05 21:15:04.015177 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-03-05T21:15:03Z,data_timestamp=2022-03-05T21:15:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsTest-Backup-1452719834',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=a273ab6d-b020-47de-876e-e182bf0c50d3,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='0bc2945ca7204c80907515ca0c271337',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='f82efe872057495292d8e41f987c48a4',volume_id=2de60bd3-fd9c-4cf2-a463-7d2086df54a0) Mar 05 21:15:04.092455 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.utils [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.15', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 21:15:04.104499 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:15:04.135687 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.040s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:15:04.137840 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[efd5225c-8f51-43a1-ba0e-345d8b31e133]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ab336d69f798\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:15:04.144117 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:15:04.165941 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 05 21:15:04.166785 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[7318748c-de5e-433a-aecd-8f1e12e36694]: [Errno 2] No such file or directory: 'systool' {{(pid=151032) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 05 21:15:04.166785 ncdv-master-1 cinder-backup[134663]: Traceback (most recent call last): Mar 05 21:15:04.166785 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 05 21:15:04.166785 ncdv-master-1 cinder-backup[134663]: ret = func(*f_args, **f_kwargs) Mar 05 21:15:04.166785 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 05 21:15:04.166785 ncdv-master-1 cinder-backup[134663]: return func(*args, **kwargs) Mar 05 21:15:04.166785 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 05 21:15:04.166785 ncdv-master-1 cinder-backup[134663]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 05 21:15:04.166785 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 05 21:15:04.166785 ncdv-master-1 cinder-backup[134663]: return putils.execute(on_execute=on_execute, Mar 05 21:15:04.166785 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 05 21:15:04.166785 ncdv-master-1 cinder-backup[134663]: obj = subprocess.Popen(cmd, Mar 05 21:15:04.166785 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 05 21:15:04.166785 ncdv-master-1 cinder-backup[134663]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 05 21:15:04.166785 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 05 21:15:04.166785 ncdv-master-1 cinder-backup[134663]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 05 21:15:04.166785 ncdv-master-1 cinder-backup[134663]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 05 21:15:04.168730 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[7318748c-de5e-433a-aecd-8f1e12e36694]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:15:04.172617 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:15:04.184880 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 05 21:15:04.185410 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[25561d48-5679-4f62-97b8-0569ab126e4f]: [Errno 2] No such file or directory: 'systool' {{(pid=151032) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 05 21:15:04.185410 ncdv-master-1 cinder-backup[134663]: Traceback (most recent call last): Mar 05 21:15:04.185410 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 05 21:15:04.185410 ncdv-master-1 cinder-backup[134663]: ret = func(*f_args, **f_kwargs) Mar 05 21:15:04.185410 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 05 21:15:04.185410 ncdv-master-1 cinder-backup[134663]: return func(*args, **kwargs) Mar 05 21:15:04.185410 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 05 21:15:04.185410 ncdv-master-1 cinder-backup[134663]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 05 21:15:04.185410 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 05 21:15:04.185410 ncdv-master-1 cinder-backup[134663]: return putils.execute(on_execute=on_execute, Mar 05 21:15:04.185410 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 05 21:15:04.185410 ncdv-master-1 cinder-backup[134663]: obj = subprocess.Popen(cmd, Mar 05 21:15:04.185410 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 05 21:15:04.185410 ncdv-master-1 cinder-backup[134663]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 05 21:15:04.185410 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 05 21:15:04.185410 ncdv-master-1 cinder-backup[134663]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 05 21:15:04.185410 ncdv-master-1 cinder-backup[134663]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 05 21:15:04.187186 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[25561d48-5679-4f62-97b8-0569ab126e4f]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:15:04.191738 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:15:04.212543 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.021s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:15:04.213114 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[d269e59f-ebd8-441e-9a0b-d639160dc586]: (4, ('/dev/sda1\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:15:04.217486 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:15:04.231598 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.015s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:15:04.232616 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[5865d948-7234-4da5-bef9-38e1f1e4f9f2]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:15:04.235993 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:15:04.257650 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.023s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:15:04.257949 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[4035511f-ab82-4e17-b035-bfd054d59ef0]: (4, ('8e294445-14da-a449-a0d4-86f2c95f64cd\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:15:04.258718 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] Running cmd (subprocess): nvme version {{(pid=134663) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:15:04.298169 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] CMD "nvme version" returned: 0 in 0.039s {{(pid=134663) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:15:04.301292 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134663) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 05 21:15:04.302006 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134663) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 05 21:15:04.303227 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:71cf25bc-00fa-4fd3-8614-fee15d100887 dsc: {{(pid=134663) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 05 21:15:04.304115 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.utils [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] <== get_connector_properties: return (210ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.15', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ab336d69f798', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '8e294445-14da-a449-a0d4-86f2c95f64cd', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:71cf25bc-00fa-4fd3-8614-fee15d100887', 'found_dsc': ''} {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 21:15:05.338063 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connector [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] Factory for rbd on None {{(pid=134663) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 05 21:15:05.339648 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-2de60bd3-fd9c-4cf2-a463-7d2086df54a0', 'hosts': ['10.222.0.15'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '2de60bd3-fd9c-4cf2-a463-7d2086df54a0', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 21:15:05.340848 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.linuxrbd [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] opening connection to ceph cluster (timeout=-1). {{(pid=134663) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 05 21:15:05.420046 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] <== connect_volume: return (79ms) {'path': } {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 21:15:05.427863 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_db.sqlalchemy.engines [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134663) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 05 21:15:05.451658 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] Starting backup of volume='2de60bd3-fd9c-4cf2-a463-7d2086df54a0'. {{(pid=134663) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Mar 05 21:15:05.452693 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] Volume file is RBD: attempting optimized backup {{(pid=134663) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Mar 05 21:15:05.502182 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] Creating base image 'volume-2de60bd3-fd9c-4cf2-a463-7d2086df54a0.backup.a273ab6d-b020-47de-876e-e182bf0c50d3' {{(pid=134663) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Mar 05 21:15:05.502706 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134663) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Mar 05 21:15:05.545799 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] Using --from-snap 'None' for incremental backup of volume 2de60bd3-fd9c-4cf2-a463-7d2086df54a0. {{(pid=134663) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Mar 05 21:15:05.546393 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] Creating backup snapshot='backup.a273ab6d-b020-47de-876e-e182bf0c50d3.snap.1646514905.5460045' {{(pid=134663) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Mar 05 21:15:06.381782 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] Performing differential transfer from 'volume-2de60bd3-fd9c-4cf2-a463-7d2086df54a0' to 'volume-2de60bd3-fd9c-4cf2-a463-7d2086df54a0.backup.a273ab6d-b020-47de-876e-e182bf0c50d3' {{(pid=134663) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Mar 05 21:15:06.382376 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd__sq5kc9s --pool volumes volumes/volume-2de60bd3-fd9c-4cf2-a463-7d2086df54a0@backup.a273ab6d-b020-47de-876e-e182bf0c50d3.snap.1646514905.5460045 -' into... {{(pid=134663) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Mar 05 21:15:06.382572 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-2de60bd3-fd9c-4cf2-a463-7d2086df54a0.backup.a273ab6d-b020-47de-876e-e182bf0c50d3' {{(pid=134663) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Mar 05 21:15:07.432857 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] Differential backup transfer completed in 1.0507s {{(pid=134663) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Mar 05 21:15:07.434576 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] Getting metadata type 'volume-base-metadata' {{(pid=134663) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Mar 05 21:15:07.452119 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134663) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Mar 05 21:15:07.452657 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] Getting metadata type 'volume-metadata' {{(pid=134663) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Mar 05 21:15:07.461011 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] No metadata type 'volume-metadata' available {{(pid=134663) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Mar 05 21:15:07.461797 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] Getting metadata type 'volume-glance-metadata' {{(pid=134663) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Mar 05 21:15:07.470010 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] No metadata type 'volume-glance-metadata' available {{(pid=134663) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Mar 05 21:15:07.470629 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] Backing up metadata for volume 2de60bd3-fd9c-4cf2-a463-7d2086df54a0. {{(pid=134663) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Mar 05 21:15:07.523217 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] Backup 'a273ab6d-b020-47de-876e-e182bf0c50d3' of volume 2de60bd3-fd9c-4cf2-a463-7d2086df54a0 finished. {{(pid=134663) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Mar 05 21:15:07.525785 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-2de60bd3-fd9c-4cf2-a463-7d2086df54a0', 'hosts': ['10.222.0.15'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '2de60bd3-fd9c-4cf2-a463-7d2086df54a0', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 21:15:07.545477 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] <== disconnect_volume: return (18ms) None {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 21:15:07.632816 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-8288ddf2-c579-4db3-be85-5c9798d066e5 tempest-VolumesBackupsTest-1617152067 None] Create backup finished. backup: a273ab6d-b020-47de-876e-e182bf0c50d3. Mar 05 21:15:08.231850 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-9755c401-9191-4993-bc76-f5fcd8026262 tempest-VolumesBackupsTest-1617152067 None] Delete backup started, backup: a273ab6d-b020-47de-876e-e182bf0c50d3. Mar 05 21:15:08.234057 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-9755c401-9191-4993-bc76-f5fcd8026262 tempest-VolumesBackupsTest-1617152067 None] Delete started for backup=a273ab6d-b020-47de-876e-e182bf0c50d3 {{(pid=134663) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Mar 05 21:15:08.234494 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-9755c401-9191-4993-bc76-f5fcd8026262 tempest-VolumesBackupsTest-1617152067 None] Trying diff format basename='volume-2de60bd3-fd9c-4cf2-a463-7d2086df54a0.backup.a273ab6d-b020-47de-876e-e182bf0c50d3' for backup base image of volume 2de60bd3-fd9c-4cf2-a463-7d2086df54a0. {{(pid=134663) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Mar 05 21:15:08.313114 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-9755c401-9191-4993-bc76-f5fcd8026262 tempest-VolumesBackupsTest-1617152067 None] Looking for snapshot of backup base 'volume-2de60bd3-fd9c-4cf2-a463-7d2086df54a0.backup.a273ab6d-b020-47de-876e-e182bf0c50d3' {{(pid=134663) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 05 21:15:08.313458 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-9755c401-9191-4993-bc76-f5fcd8026262 tempest-VolumesBackupsTest-1617152067 None] Found snapshot 'backup.a273ab6d-b020-47de-876e-e182bf0c50d3.snap.1646514905.5460045' {{(pid=134663) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Mar 05 21:15:08.314073 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-9755c401-9191-4993-bc76-f5fcd8026262 tempest-VolumesBackupsTest-1617152067 None] Deleting backup snapshot='backup.a273ab6d-b020-47de-876e-e182bf0c50d3.snap.1646514905.5460045' {{(pid=134663) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Mar 05 21:15:08.852580 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.drivers.ceph [None req-9755c401-9191-4993-bc76-f5fcd8026262 tempest-VolumesBackupsTest-1617152067 None] Deleting backup base image='volume-2de60bd3-fd9c-4cf2-a463-7d2086df54a0.backup.a273ab6d-b020-47de-876e-e182bf0c50d3' of volume 2de60bd3-fd9c-4cf2-a463-7d2086df54a0. Mar 05 21:15:09.136972 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-9755c401-9191-4993-bc76-f5fcd8026262 tempest-VolumesBackupsTest-1617152067 None] Base backup image='volume-2de60bd3-fd9c-4cf2-a463-7d2086df54a0.backup.a273ab6d-b020-47de-876e-e182bf0c50d3' of volume 2de60bd3-fd9c-4cf2-a463-7d2086df54a0 deleted. {{(pid=134663) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Mar 05 21:15:09.236476 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-9755c401-9191-4993-bc76-f5fcd8026262 tempest-VolumesBackupsTest-1617152067 None] Delete of backup 'a273ab6d-b020-47de-876e-e182bf0c50d3' for volume '2de60bd3-fd9c-4cf2-a463-7d2086df54a0' finished. {{(pid=134663) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Mar 05 21:15:09.243781 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_db.sqlalchemy.engines [None req-9755c401-9191-4993-bc76-f5fcd8026262 tempest-VolumesBackupsTest-1617152067 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134663) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 05 21:15:09.298504 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.quota [None req-9755c401-9191-4993-bc76-f5fcd8026262 tempest-VolumesBackupsTest-1617152067 None] Created reservations ['b1e05264-97b2-44c6-9f9f-d465d8fe5592', '1fbca227-f638-4daf-b23f-00ab7aa8a959'] {{(pid=134663) reserve /opt/stack/cinder/cinder/quota.py:755}} Mar 05 21:15:09.344823 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-9755c401-9191-4993-bc76-f5fcd8026262 tempest-VolumesBackupsTest-1617152067 None] Delete backup finished, backup a273ab6d-b020-47de-876e-e182bf0c50d3 deleted. Mar 05 21:15:19.834969 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] Create backup started, backup: b48ce6f8-b6c1-47fa-8378-5b7bd8130650 volume: 47b94a70-47d6-4b47-9cf2-6c025f612fb4. Mar 05 21:15:19.842914 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-03-05T21:15:19Z,data_timestamp=2022-03-05T21:15:19Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsTest-Backup-1981703880',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=b48ce6f8-b6c1-47fa-8378-5b7bd8130650,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='0bc2945ca7204c80907515ca0c271337',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='f82efe872057495292d8e41f987c48a4',volume_id=47b94a70-47d6-4b47-9cf2-6c025f612fb4) Mar 05 21:15:19.891887 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.utils [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.15', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 21:15:19.894652 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:15:19.901176 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.008s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:15:19.901503 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[25b11764-8872-4b81-b2ff-edb810cd4ea6]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ab336d69f798\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:15:19.902790 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:15:19.910179 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 05 21:15:19.911137 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[322e000d-a824-4e5c-9d04-a9af41fc1f48]: [Errno 2] No such file or directory: 'systool' {{(pid=151032) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 05 21:15:19.911137 ncdv-master-1 cinder-backup[134663]: Traceback (most recent call last): Mar 05 21:15:19.911137 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 05 21:15:19.911137 ncdv-master-1 cinder-backup[134663]: ret = func(*f_args, **f_kwargs) Mar 05 21:15:19.911137 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 05 21:15:19.911137 ncdv-master-1 cinder-backup[134663]: return func(*args, **kwargs) Mar 05 21:15:19.911137 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 05 21:15:19.911137 ncdv-master-1 cinder-backup[134663]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 05 21:15:19.911137 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 05 21:15:19.911137 ncdv-master-1 cinder-backup[134663]: return putils.execute(on_execute=on_execute, Mar 05 21:15:19.911137 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 05 21:15:19.911137 ncdv-master-1 cinder-backup[134663]: obj = subprocess.Popen(cmd, Mar 05 21:15:19.911137 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 05 21:15:19.911137 ncdv-master-1 cinder-backup[134663]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 05 21:15:19.911137 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 05 21:15:19.911137 ncdv-master-1 cinder-backup[134663]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 05 21:15:19.911137 ncdv-master-1 cinder-backup[134663]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 05 21:15:19.913314 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[322e000d-a824-4e5c-9d04-a9af41fc1f48]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:15:19.914494 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:15:19.921311 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 05 21:15:19.921836 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[e8a3c433-04d0-45ed-8216-f05189d5571f]: [Errno 2] No such file or directory: 'systool' {{(pid=151032) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 05 21:15:19.921836 ncdv-master-1 cinder-backup[134663]: Traceback (most recent call last): Mar 05 21:15:19.921836 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 05 21:15:19.921836 ncdv-master-1 cinder-backup[134663]: ret = func(*f_args, **f_kwargs) Mar 05 21:15:19.921836 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 05 21:15:19.921836 ncdv-master-1 cinder-backup[134663]: return func(*args, **kwargs) Mar 05 21:15:19.921836 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 05 21:15:19.921836 ncdv-master-1 cinder-backup[134663]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 05 21:15:19.921836 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 05 21:15:19.921836 ncdv-master-1 cinder-backup[134663]: return putils.execute(on_execute=on_execute, Mar 05 21:15:19.921836 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 05 21:15:19.921836 ncdv-master-1 cinder-backup[134663]: obj = subprocess.Popen(cmd, Mar 05 21:15:19.921836 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 05 21:15:19.921836 ncdv-master-1 cinder-backup[134663]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 05 21:15:19.921836 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 05 21:15:19.921836 ncdv-master-1 cinder-backup[134663]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 05 21:15:19.921836 ncdv-master-1 cinder-backup[134663]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 05 21:15:19.923232 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[e8a3c433-04d0-45ed-8216-f05189d5571f]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:15:19.924782 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:15:19.937224 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.012s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:15:19.937940 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[820ace86-b23b-4ef5-9bbf-b21f822cdd7e]: (4, ('/dev/sda1\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:15:19.939030 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:15:19.948655 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.009s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:15:19.948991 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[d1c7b456-8fd5-4a09-bbaf-a65f3d919e30]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:15:19.950643 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:15:19.959896 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.009s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:15:19.960223 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[b5390971-afd3-497c-90c0-ab568fcf9c3a]: (4, ('8e294445-14da-a449-a0d4-86f2c95f64cd\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:15:19.960972 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] Running cmd (subprocess): nvme version {{(pid=134663) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:15:19.979883 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] CMD "nvme version" returned: 0 in 0.019s {{(pid=134663) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:15:19.982146 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134663) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 05 21:15:19.983111 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134663) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 05 21:15:19.983496 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:71cf25bc-00fa-4fd3-8614-fee15d100887 dsc: {{(pid=134663) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 05 21:15:19.984186 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.utils [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] <== get_connector_properties: return (92ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.15', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ab336d69f798', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '8e294445-14da-a449-a0d4-86f2c95f64cd', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:71cf25bc-00fa-4fd3-8614-fee15d100887', 'found_dsc': ''} {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 21:15:20.691449 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connector [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] Factory for rbd on None {{(pid=134663) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 05 21:15:20.693508 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-47b94a70-47d6-4b47-9cf2-6c025f612fb4', 'hosts': ['10.222.0.15'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '47b94a70-47d6-4b47-9cf2-6c025f612fb4', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 21:15:20.694577 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.linuxrbd [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] opening connection to ceph cluster (timeout=-1). {{(pid=134663) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 05 21:15:20.764145 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] <== connect_volume: return (69ms) {'path': } {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 21:15:20.772163 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_db.sqlalchemy.engines [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134663) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 05 21:15:20.790628 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] Starting backup of volume='47b94a70-47d6-4b47-9cf2-6c025f612fb4'. {{(pid=134663) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Mar 05 21:15:20.792088 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] Volume file is RBD: attempting optimized backup {{(pid=134663) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Mar 05 21:15:20.817110 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] Creating base image 'volume-47b94a70-47d6-4b47-9cf2-6c025f612fb4.backup.b48ce6f8-b6c1-47fa-8378-5b7bd8130650' {{(pid=134663) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Mar 05 21:15:20.817635 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134663) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Mar 05 21:15:20.849105 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] Using --from-snap 'None' for incremental backup of volume 47b94a70-47d6-4b47-9cf2-6c025f612fb4. {{(pid=134663) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Mar 05 21:15:20.850831 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] Creating backup snapshot='backup.b48ce6f8-b6c1-47fa-8378-5b7bd8130650.snap.1646514920.8491316' {{(pid=134663) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Mar 05 21:15:21.138234 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] Performing differential transfer from 'volume-47b94a70-47d6-4b47-9cf2-6c025f612fb4' to 'volume-47b94a70-47d6-4b47-9cf2-6c025f612fb4.backup.b48ce6f8-b6c1-47fa-8378-5b7bd8130650' {{(pid=134663) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Mar 05 21:15:21.138732 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_m5zacjzu --pool volumes volumes/volume-47b94a70-47d6-4b47-9cf2-6c025f612fb4@backup.b48ce6f8-b6c1-47fa-8378-5b7bd8130650.snap.1646514920.8491316 -' into... {{(pid=134663) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Mar 05 21:15:21.138998 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-47b94a70-47d6-4b47-9cf2-6c025f612fb4.backup.b48ce6f8-b6c1-47fa-8378-5b7bd8130650' {{(pid=134663) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Mar 05 21:15:26.960780 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] Differential backup transfer completed in 5.8222s {{(pid=134663) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Mar 05 21:15:26.961514 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] Getting metadata type 'volume-base-metadata' {{(pid=134663) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Mar 05 21:15:26.976793 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134663) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Mar 05 21:15:26.977268 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] Getting metadata type 'volume-metadata' {{(pid=134663) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Mar 05 21:15:26.983916 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] No metadata type 'volume-metadata' available {{(pid=134663) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Mar 05 21:15:26.984373 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] Getting metadata type 'volume-glance-metadata' {{(pid=134663) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Mar 05 21:15:26.991887 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=134663) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}} Mar 05 21:15:26.992544 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] Backing up metadata for volume 47b94a70-47d6-4b47-9cf2-6c025f612fb4. {{(pid=134663) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Mar 05 21:15:27.027614 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] Backup 'b48ce6f8-b6c1-47fa-8378-5b7bd8130650' of volume 47b94a70-47d6-4b47-9cf2-6c025f612fb4 finished. {{(pid=134663) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Mar 05 21:15:27.029769 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-47b94a70-47d6-4b47-9cf2-6c025f612fb4', 'hosts': ['10.222.0.15'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '47b94a70-47d6-4b47-9cf2-6c025f612fb4', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 21:15:27.056815 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] <== disconnect_volume: return (26ms) None {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 21:15:27.120942 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-fa18d988-6da9-4493-9143-1cb8e811efa9 tempest-VolumesBackupsTest-1617152067 None] Create backup finished. backup: b48ce6f8-b6c1-47fa-8378-5b7bd8130650. Mar 05 21:15:29.656768 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] Restore backup started, backup: b48ce6f8-b6c1-47fa-8378-5b7bd8130650 volume: 64e64fb0-06b0-49f8-a57b-8d1b70369c59. Mar 05 21:15:29.681093 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.utils [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.15', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 21:15:29.684627 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:15:29.690585 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.008s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:15:29.691151 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[22a58df3-efe9-407c-9b6e-848e8ddb2788]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ab336d69f798\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:15:29.692926 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:15:29.699074 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 05 21:15:29.700072 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[1b194c1e-58b8-4ab7-bf50-e3a0624a4ca6]: [Errno 2] No such file or directory: 'systool' {{(pid=151032) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 05 21:15:29.700072 ncdv-master-1 cinder-backup[134663]: Traceback (most recent call last): Mar 05 21:15:29.700072 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 05 21:15:29.700072 ncdv-master-1 cinder-backup[134663]: ret = func(*f_args, **f_kwargs) Mar 05 21:15:29.700072 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 05 21:15:29.700072 ncdv-master-1 cinder-backup[134663]: return func(*args, **kwargs) Mar 05 21:15:29.700072 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 05 21:15:29.700072 ncdv-master-1 cinder-backup[134663]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 05 21:15:29.700072 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 05 21:15:29.700072 ncdv-master-1 cinder-backup[134663]: return putils.execute(on_execute=on_execute, Mar 05 21:15:29.700072 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 05 21:15:29.700072 ncdv-master-1 cinder-backup[134663]: obj = subprocess.Popen(cmd, Mar 05 21:15:29.700072 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 05 21:15:29.700072 ncdv-master-1 cinder-backup[134663]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 05 21:15:29.700072 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 05 21:15:29.700072 ncdv-master-1 cinder-backup[134663]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 05 21:15:29.700072 ncdv-master-1 cinder-backup[134663]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 05 21:15:29.701576 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[1b194c1e-58b8-4ab7-bf50-e3a0624a4ca6]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:15:29.701744 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:15:29.707490 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 05 21:15:29.708415 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[be2bd56b-0b26-4233-9691-e99bcfe5dfd2]: [Errno 2] No such file or directory: 'systool' {{(pid=151032) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 05 21:15:29.708415 ncdv-master-1 cinder-backup[134663]: Traceback (most recent call last): Mar 05 21:15:29.708415 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 05 21:15:29.708415 ncdv-master-1 cinder-backup[134663]: ret = func(*f_args, **f_kwargs) Mar 05 21:15:29.708415 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 05 21:15:29.708415 ncdv-master-1 cinder-backup[134663]: return func(*args, **kwargs) Mar 05 21:15:29.708415 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 05 21:15:29.708415 ncdv-master-1 cinder-backup[134663]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 05 21:15:29.708415 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 05 21:15:29.708415 ncdv-master-1 cinder-backup[134663]: return putils.execute(on_execute=on_execute, Mar 05 21:15:29.708415 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 05 21:15:29.708415 ncdv-master-1 cinder-backup[134663]: obj = subprocess.Popen(cmd, Mar 05 21:15:29.708415 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 05 21:15:29.708415 ncdv-master-1 cinder-backup[134663]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 05 21:15:29.708415 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 05 21:15:29.708415 ncdv-master-1 cinder-backup[134663]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 05 21:15:29.708415 ncdv-master-1 cinder-backup[134663]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 05 21:15:29.709851 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[be2bd56b-0b26-4233-9691-e99bcfe5dfd2]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:15:29.711372 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:15:29.725633 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.014s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:15:29.726039 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[b6b757a7-f332-4192-b0b2-5e1b9f712211]: (4, ('/dev/sda1\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:15:29.727605 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:15:29.760316 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.031s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:15:29.760758 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[f5e224d7-261a-40e4-9fdd-ab1cbc65d7cc]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:15:29.764760 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:15:29.774976 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.012s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:15:29.776233 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[bd7060ea-318e-47d7-8d3c-1bc5837ffde1]: (4, ('8e294445-14da-a449-a0d4-86f2c95f64cd\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:15:29.777733 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] Running cmd (subprocess): nvme version {{(pid=134663) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:15:29.801857 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] CMD "nvme version" returned: 0 in 0.023s {{(pid=134663) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:15:29.804804 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134663) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 05 21:15:29.805300 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134663) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 05 21:15:29.805746 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:71cf25bc-00fa-4fd3-8614-fee15d100887 dsc: {{(pid=134663) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 05 21:15:29.809354 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.utils [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] <== get_connector_properties: return (124ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.15', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ab336d69f798', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '8e294445-14da-a449-a0d4-86f2c95f64cd', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:71cf25bc-00fa-4fd3-8614-fee15d100887', 'found_dsc': ''} {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 21:15:30.621980 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connector [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] Factory for rbd on None {{(pid=134663) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 05 21:15:30.623438 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-64e64fb0-06b0-49f8-a57b-8d1b70369c59', 'hosts': ['10.222.0.15'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '64e64fb0-06b0-49f8-a57b-8d1b70369c59', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 21:15:30.624521 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.linuxrbd [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] opening connection to ceph cluster (timeout=-1). {{(pid=134663) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 05 21:15:30.700470 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] <== connect_volume: return (76ms) {'path': } {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 21:15:30.715453 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_db.sqlalchemy.engines [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134663) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 05 21:15:30.731728 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] Starting restore from Ceph backup=b48ce6f8-b6c1-47fa-8378-5b7bd8130650 to volume=volume-64e64fb0-06b0-49f8-a57b-8d1b70369c59 {{(pid=134663) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1274}} Mar 05 21:15:30.856196 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] Looking for snapshot of backup base 'volume-47b94a70-47d6-4b47-9cf2-6c025f612fb4.backup.b48ce6f8-b6c1-47fa-8378-5b7bd8130650' {{(pid=134663) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 05 21:15:30.856619 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] Found snapshot 'backup.b48ce6f8-b6c1-47fa-8378-5b7bd8130650.snap.1646514920.8491316' {{(pid=134663) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Mar 05 21:15:30.861290 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] Volume file is RBD. {{(pid=134663) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1184}} Mar 05 21:15:30.918032 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] Attempting differential restore. {{(pid=134663) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1234}} Mar 05 21:15:30.918565 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] Attempting incremental restore from base='volume-47b94a70-47d6-4b47-9cf2-6c025f612fb4.backup.b48ce6f8-b6c1-47fa-8378-5b7bd8130650' snap='backup.b48ce6f8-b6c1-47fa-8378-5b7bd8130650.snap.1646514920.8491316' {{(pid=134663) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1089}} Mar 05 21:15:30.919849 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] Performing differential transfer from 'volume-47b94a70-47d6-4b47-9cf2-6c025f612fb4.backup.b48ce6f8-b6c1-47fa-8378-5b7bd8130650' to 'volume-64e64fb0-06b0-49f8-a57b-8d1b70369c59' {{(pid=134663) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Mar 05 21:15:30.920397 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-47b94a70-47d6-4b47-9cf2-6c025f612fb4.backup.b48ce6f8-b6c1-47fa-8378-5b7bd8130650@backup.b48ce6f8-b6c1-47fa-8378-5b7bd8130650.snap.1646514920.8491316 -' into... {{(pid=134663) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Mar 05 21:15:30.921014 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_pllnbu0e --pool volumes - volumes/volume-64e64fb0-06b0-49f8-a57b-8d1b70369c59' {{(pid=134663) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Mar 05 21:15:36.320705 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] Restore transfer completed in 5.4010s {{(pid=134663) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1111}} Mar 05 21:15:36.321943 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=134663) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1285}} Mar 05 21:15:36.374238 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] Restoring volume base metadata {{(pid=134663) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} Mar 05 21:15:36.402834 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] No metadata of type 'volume-metadata' to restore {{(pid=134663) put /opt/stack/cinder/cinder/backup/driver.py:345}} Mar 05 21:15:36.403244 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] Restoring volume glance metadata {{(pid=134663) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}} Mar 05 21:15:36.566188 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] Restore to volume 64e64fb0-06b0-49f8-a57b-8d1b70369c59 finished successfully. {{(pid=134663) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1292}} Mar 05 21:15:36.567946 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-64e64fb0-06b0-49f8-a57b-8d1b70369c59', 'hosts': ['10.222.0.15'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '64e64fb0-06b0-49f8-a57b-8d1b70369c59', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 21:15:36.580484 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] <== disconnect_volume: return (12ms) None {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 21:15:36.661437 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.manager [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] Encryption key ID for volume 64e64fb0-06b0-49f8-a57b-8d1b70369c59 already matches encryption key ID in backup b48ce6f8-b6c1-47fa-8378-5b7bd8130650. {{(pid=134663) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} Mar 05 21:15:36.692909 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-8bc7f593-575b-41a9-b496-25b2aa3666ad tempest-VolumesBackupsTest-1617152067 None] Finished restoring backup b48ce6f8-b6c1-47fa-8378-5b7bd8130650 to volume 64e64fb0-06b0-49f8-a57b-8d1b70369c59. Mar 05 21:15:39.360027 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-db5d1ecf-b968-4eb3-a8a0-7838586f72f0 tempest-VolumesBackupsTest-1617152067 None] Delete backup started, backup: b48ce6f8-b6c1-47fa-8378-5b7bd8130650. Mar 05 21:15:39.362567 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-db5d1ecf-b968-4eb3-a8a0-7838586f72f0 tempest-VolumesBackupsTest-1617152067 None] Delete started for backup=b48ce6f8-b6c1-47fa-8378-5b7bd8130650 {{(pid=134663) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Mar 05 21:15:39.363758 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-db5d1ecf-b968-4eb3-a8a0-7838586f72f0 tempest-VolumesBackupsTest-1617152067 None] Trying diff format basename='volume-47b94a70-47d6-4b47-9cf2-6c025f612fb4.backup.b48ce6f8-b6c1-47fa-8378-5b7bd8130650' for backup base image of volume 47b94a70-47d6-4b47-9cf2-6c025f612fb4. {{(pid=134663) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Mar 05 21:15:39.430187 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-db5d1ecf-b968-4eb3-a8a0-7838586f72f0 tempest-VolumesBackupsTest-1617152067 None] Looking for snapshot of backup base 'volume-47b94a70-47d6-4b47-9cf2-6c025f612fb4.backup.b48ce6f8-b6c1-47fa-8378-5b7bd8130650' {{(pid=134663) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 05 21:15:39.430479 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-db5d1ecf-b968-4eb3-a8a0-7838586f72f0 tempest-VolumesBackupsTest-1617152067 None] Found snapshot 'backup.b48ce6f8-b6c1-47fa-8378-5b7bd8130650.snap.1646514920.8491316' {{(pid=134663) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Mar 05 21:15:39.430885 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-db5d1ecf-b968-4eb3-a8a0-7838586f72f0 tempest-VolumesBackupsTest-1617152067 None] Deleting backup snapshot='backup.b48ce6f8-b6c1-47fa-8378-5b7bd8130650.snap.1646514920.8491316' {{(pid=134663) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Mar 05 21:15:40.257994 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.drivers.ceph [None req-db5d1ecf-b968-4eb3-a8a0-7838586f72f0 tempest-VolumesBackupsTest-1617152067 None] Deleting backup base image='volume-47b94a70-47d6-4b47-9cf2-6c025f612fb4.backup.b48ce6f8-b6c1-47fa-8378-5b7bd8130650' of volume 47b94a70-47d6-4b47-9cf2-6c025f612fb4. Mar 05 21:15:40.654642 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-db5d1ecf-b968-4eb3-a8a0-7838586f72f0 tempest-VolumesBackupsTest-1617152067 None] Base backup image='volume-47b94a70-47d6-4b47-9cf2-6c025f612fb4.backup.b48ce6f8-b6c1-47fa-8378-5b7bd8130650' of volume 47b94a70-47d6-4b47-9cf2-6c025f612fb4 deleted. {{(pid=134663) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Mar 05 21:15:40.689168 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-db5d1ecf-b968-4eb3-a8a0-7838586f72f0 tempest-VolumesBackupsTest-1617152067 None] Delete of backup 'b48ce6f8-b6c1-47fa-8378-5b7bd8130650' for volume '47b94a70-47d6-4b47-9cf2-6c025f612fb4' finished. {{(pid=134663) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Mar 05 21:15:40.695100 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_db.sqlalchemy.engines [None req-db5d1ecf-b968-4eb3-a8a0-7838586f72f0 tempest-VolumesBackupsTest-1617152067 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134663) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 05 21:15:40.718158 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.quota [None req-db5d1ecf-b968-4eb3-a8a0-7838586f72f0 tempest-VolumesBackupsTest-1617152067 None] Created reservations ['8a561b3b-3492-49e4-8fa9-c00dc1e8a871', '1a343558-c547-4ddb-9d3f-dac641377289'] {{(pid=134663) reserve /opt/stack/cinder/cinder/quota.py:755}} Mar 05 21:15:40.743554 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-db5d1ecf-b968-4eb3-a8a0-7838586f72f0 tempest-VolumesBackupsTest-1617152067 None] Delete backup finished, backup b48ce6f8-b6c1-47fa-8378-5b7bd8130650 deleted. Mar 05 21:15:49.562781 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:15:49.563478 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:16:00.781895 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] Create backup started, backup: 0deae2ce-eba9-444d-bec9-6d109df62c50 volume: 2b235070-e46f-4802-b5ac-7531e3c2899c snapshot: 721f128b-3d36-425b-b691-ec50a945edf6. Mar 05 21:16:00.790955 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-03-05T21:16:00Z,data_timestamp=2022-03-05T21:15:58Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Backup-938448643',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=0deae2ce-eba9-444d-bec9-6d109df62c50,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='46aa322abe044eb7baa1f99f151b28cc',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id='721f128b-3d36-425b-b691-ec50a945edf6',status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='2b581b22102f425daee098c1318c506e',volume_id=2b235070-e46f-4802-b5ac-7531e3c2899c) Mar 05 21:16:01.168819 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.utils [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.15', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 21:16:01.173111 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:16:01.186911 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.011s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:16:01.187346 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[38f84272-1db6-403a-8a38-126c32fa6ab7]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:ab336d69f798\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:16:01.189372 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:16:01.198218 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 05 21:16:01.199132 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[c237c405-14e7-4389-b3c1-5dcbf17d930b]: [Errno 2] No such file or directory: 'systool' {{(pid=151032) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 05 21:16:01.199132 ncdv-master-1 cinder-backup[134663]: Traceback (most recent call last): Mar 05 21:16:01.199132 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 05 21:16:01.199132 ncdv-master-1 cinder-backup[134663]: ret = func(*f_args, **f_kwargs) Mar 05 21:16:01.199132 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 05 21:16:01.199132 ncdv-master-1 cinder-backup[134663]: return func(*args, **kwargs) Mar 05 21:16:01.199132 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 05 21:16:01.199132 ncdv-master-1 cinder-backup[134663]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 05 21:16:01.199132 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 05 21:16:01.199132 ncdv-master-1 cinder-backup[134663]: return putils.execute(on_execute=on_execute, Mar 05 21:16:01.199132 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 05 21:16:01.199132 ncdv-master-1 cinder-backup[134663]: obj = subprocess.Popen(cmd, Mar 05 21:16:01.199132 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 05 21:16:01.199132 ncdv-master-1 cinder-backup[134663]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 05 21:16:01.199132 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 05 21:16:01.199132 ncdv-master-1 cinder-backup[134663]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 05 21:16:01.199132 ncdv-master-1 cinder-backup[134663]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 05 21:16:01.201054 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[c237c405-14e7-4389-b3c1-5dcbf17d930b]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:16:01.202488 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:16:01.210038 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Mar 05 21:16:01.210624 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[a10871ce-4bb0-491c-affe-d88c25b1a9d2]: [Errno 2] No such file or directory: 'systool' {{(pid=151032) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Mar 05 21:16:01.210624 ncdv-master-1 cinder-backup[134663]: Traceback (most recent call last): Mar 05 21:16:01.210624 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Mar 05 21:16:01.210624 ncdv-master-1 cinder-backup[134663]: ret = func(*f_args, **f_kwargs) Mar 05 21:16:01.210624 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Mar 05 21:16:01.210624 ncdv-master-1 cinder-backup[134663]: return func(*args, **kwargs) Mar 05 21:16:01.210624 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Mar 05 21:16:01.210624 ncdv-master-1 cinder-backup[134663]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Mar 05 21:16:01.210624 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Mar 05 21:16:01.210624 ncdv-master-1 cinder-backup[134663]: return putils.execute(on_execute=on_execute, Mar 05 21:16:01.210624 ncdv-master-1 cinder-backup[134663]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Mar 05 21:16:01.210624 ncdv-master-1 cinder-backup[134663]: obj = subprocess.Popen(cmd, Mar 05 21:16:01.210624 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Mar 05 21:16:01.210624 ncdv-master-1 cinder-backup[134663]: self._execute_child(args, executable, preexec_fn, close_fds, Mar 05 21:16:01.210624 ncdv-master-1 cinder-backup[134663]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Mar 05 21:16:01.210624 ncdv-master-1 cinder-backup[134663]: raise child_exception_type(errno_num, err_msg, err_filename) Mar 05 21:16:01.210624 ncdv-master-1 cinder-backup[134663]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Mar 05 21:16:01.211861 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[a10871ce-4bb0-491c-affe-d88c25b1a9d2]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:16:01.213403 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:16:01.230788 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.017s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:16:01.231207 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[20aaca73-0d5e-42e3-8421-726b17e3fa33]: (4, ('/dev/sda1\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:16:01.233305 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:16:01.245397 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.012s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:16:01.245868 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[b056fb2c-ff11-404e-8b02-d30b4dc8b281]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:16:01.248159 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:16:01.256898 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.009s {{(pid=151032) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:16:01.257913 ncdv-master-1 cinder-backup[134663]: DEBUG oslo.privsep.daemon [-] privsep: reply[bc44069f-3e1c-4e89-b11c-25f20979dbd6]: (4, ('8e294445-14da-a449-a0d4-86f2c95f64cd\n', '')) {{(pid=151032) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Mar 05 21:16:01.258278 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] Running cmd (subprocess): nvme version {{(pid=134663) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Mar 05 21:16:01.292533 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_concurrency.processutils [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] CMD "nvme version" returned: 0 in 0.034s {{(pid=134663) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Mar 05 21:16:01.294532 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=134663) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Mar 05 21:16:01.294928 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=134663) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Mar 05 21:16:01.295292 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.lightos [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:71cf25bc-00fa-4fd3-8614-fee15d100887 dsc: {{(pid=134663) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Mar 05 21:16:01.295981 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.utils [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] <== get_connector_properties: return (126ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.15', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:ab336d69f798', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '8e294445-14da-a449-a0d4-86f2c95f64cd', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:71cf25bc-00fa-4fd3-8614-fee15d100887', 'found_dsc': ''} {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 21:16:02.171907 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connector [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] Factory for rbd on None {{(pid=134663) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Mar 05 21:16:02.173791 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-b4332c86-ff99-4ed6-b274-653113259fdb', 'hosts': ['10.222.0.15'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'b4332c86-ff99-4ed6-b274-653113259fdb', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 21:16:02.176298 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.linuxrbd [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] opening connection to ceph cluster (timeout=-1). {{(pid=134663) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Mar 05 21:16:02.232427 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] <== connect_volume: return (58ms) {'path': } {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 21:16:02.241822 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_db.sqlalchemy.engines [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134663) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 05 21:16:02.262104 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] Starting backup of volume='2b235070-e46f-4802-b5ac-7531e3c2899c'. {{(pid=134663) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Mar 05 21:16:02.286517 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=134663) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Mar 05 21:16:02.286912 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] Creating backup base image='volume-2b235070-e46f-4802-b5ac-7531e3c2899c.backup.base' for volume 2b235070-e46f-4802-b5ac-7531e3c2899c. {{(pid=134663) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:830}} Mar 05 21:16:02.339318 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] Copying data from volume 2b235070-e46f-4802-b5ac-7531e3c2899c. {{(pid=134663) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:842}} Mar 05 21:16:02.360114 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] Transferring data between 'volume-2b235070-e46f-4802-b5ac-7531e3c2899c' and 'volume-2b235070-e46f-4802-b5ac-7531e3c2899c.backup.base' {{(pid=134663) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:399}} Mar 05 21:16:02.360635 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] 8 chunks of 134217728 bytes to be transferred {{(pid=134663) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:403}} Mar 05 21:16:03.583854 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] Transferred chunk 1 of 8 (107536K/s) {{(pid=134663) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Mar 05 21:16:05.132884 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] Transferred chunk 2 of 8 (84718K/s) {{(pid=134663) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Mar 05 21:16:06.427058 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] Transferred chunk 3 of 8 (101343K/s) {{(pid=134663) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Mar 05 21:16:08.136916 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] Transferred chunk 4 of 8 (76735K/s) {{(pid=134663) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Mar 05 21:16:09.894041 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] Transferred chunk 5 of 8 (74623K/s) {{(pid=134663) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Mar 05 21:16:11.756689 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] Transferred chunk 6 of 8 (70438K/s) {{(pid=134663) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Mar 05 21:16:13.559257 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] Transferred chunk 7 of 8 (72796K/s) {{(pid=134663) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Mar 05 21:16:15.381405 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] Transferred chunk 8 of 8 (71991K/s) {{(pid=134663) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Mar 05 21:16:15.472923 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] Getting metadata type 'volume-base-metadata' {{(pid=134663) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Mar 05 21:16:15.509464 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=134663) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Mar 05 21:16:15.511064 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] Getting metadata type 'volume-metadata' {{(pid=134663) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Mar 05 21:16:15.523963 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] No metadata type 'volume-metadata' available {{(pid=134663) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Mar 05 21:16:15.525009 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] Getting metadata type 'volume-glance-metadata' {{(pid=134663) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Mar 05 21:16:15.539341 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.driver [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] No metadata type 'volume-glance-metadata' available {{(pid=134663) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Mar 05 21:16:15.540401 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] Backing up metadata for volume 2b235070-e46f-4802-b5ac-7531e3c2899c. {{(pid=134663) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Mar 05 21:16:15.604331 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] Backup '0deae2ce-eba9-444d-bec9-6d109df62c50' of volume 2b235070-e46f-4802-b5ac-7531e3c2899c finished. {{(pid=134663) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Mar 05 21:16:15.606380 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-b4332c86-ff99-4ed6-b274-653113259fdb', 'hosts': ['10.222.0.15'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'b4332c86-ff99-4ed6-b274-653113259fdb', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Mar 05 21:16:15.622106 ncdv-master-1 cinder-backup[134663]: DEBUG os_brick.initiator.connectors.rbd [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] <== disconnect_volume: return (14ms) None {{(pid=134663) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Mar 05 21:16:15.787824 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-617d9854-f3b5-4080-a304-617252073d6b tempest-VolumesSnapshotTestJSON-163973597 None] Create backup finished. backup: 0deae2ce-eba9-444d-bec9-6d109df62c50. Mar 05 21:16:16.759402 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-30ecc1f2-57fb-42d3-bacb-a17bb24e8215 tempest-VolumesSnapshotTestJSON-163973597 None] Delete backup started, backup: 0deae2ce-eba9-444d-bec9-6d109df62c50. Mar 05 21:16:16.762021 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-30ecc1f2-57fb-42d3-bacb-a17bb24e8215 tempest-VolumesSnapshotTestJSON-163973597 None] Delete started for backup=0deae2ce-eba9-444d-bec9-6d109df62c50 {{(pid=134663) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Mar 05 21:16:16.762593 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-30ecc1f2-57fb-42d3-bacb-a17bb24e8215 tempest-VolumesSnapshotTestJSON-163973597 None] Trying diff format basename='volume-2b235070-e46f-4802-b5ac-7531e3c2899c.backup.0deae2ce-eba9-444d-bec9-6d109df62c50' for backup base image of volume 2b235070-e46f-4802-b5ac-7531e3c2899c. {{(pid=134663) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Mar 05 21:16:16.838518 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-30ecc1f2-57fb-42d3-bacb-a17bb24e8215 tempest-VolumesSnapshotTestJSON-163973597 None] Image 'volume-2b235070-e46f-4802-b5ac-7531e3c2899c.backup.0deae2ce-eba9-444d-bec9-6d109df62c50' not found - trying diff format name {{(pid=134663) _rbd_image_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:661}} Mar 05 21:16:16.911111 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-30ecc1f2-57fb-42d3-bacb-a17bb24e8215 tempest-VolumesSnapshotTestJSON-163973597 None] Looking for snapshot of backup base 'volume-2b235070-e46f-4802-b5ac-7531e3c2899c.backup.base' {{(pid=134663) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Mar 05 21:16:16.911453 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-30ecc1f2-57fb-42d3-bacb-a17bb24e8215 tempest-VolumesSnapshotTestJSON-163973597 None] Backup base 'volume-2b235070-e46f-4802-b5ac-7531e3c2899c.backup.base' has no snapshots {{(pid=134663) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:908}} Mar 05 21:16:16.911845 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-30ecc1f2-57fb-42d3-bacb-a17bb24e8215 tempest-VolumesSnapshotTestJSON-163973597 None] No backup snapshot to delete {{(pid=134663) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:476}} Mar 05 21:16:16.927537 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.drivers.ceph [None req-30ecc1f2-57fb-42d3-bacb-a17bb24e8215 tempest-VolumesSnapshotTestJSON-163973597 None] Deleting backup base image='volume-2b235070-e46f-4802-b5ac-7531e3c2899c.backup.base' of volume 2b235070-e46f-4802-b5ac-7531e3c2899c. Mar 05 21:16:18.148890 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-30ecc1f2-57fb-42d3-bacb-a17bb24e8215 tempest-VolumesSnapshotTestJSON-163973597 None] Base backup image='volume-2b235070-e46f-4802-b5ac-7531e3c2899c.backup.base' of volume 2b235070-e46f-4802-b5ac-7531e3c2899c deleted. {{(pid=134663) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Mar 05 21:16:18.246078 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.backup.drivers.ceph [None req-30ecc1f2-57fb-42d3-bacb-a17bb24e8215 tempest-VolumesSnapshotTestJSON-163973597 None] Delete of backup '0deae2ce-eba9-444d-bec9-6d109df62c50' for volume '2b235070-e46f-4802-b5ac-7531e3c2899c' finished. {{(pid=134663) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Mar 05 21:16:18.253924 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_db.sqlalchemy.engines [None req-30ecc1f2-57fb-42d3-bacb-a17bb24e8215 tempest-VolumesSnapshotTestJSON-163973597 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=134663) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Mar 05 21:16:18.321884 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.quota [None req-30ecc1f2-57fb-42d3-bacb-a17bb24e8215 tempest-VolumesSnapshotTestJSON-163973597 None] Created reservations ['1808a5b8-4e95-4a43-9007-a406a5cf484d', 'b7035ae3-2339-4b04-9a4b-73455cf0c230'] {{(pid=134663) reserve /opt/stack/cinder/cinder/quota.py:755}} Mar 05 21:16:18.375410 ncdv-master-1 cinder-backup[134663]: INFO cinder.backup.manager [None req-30ecc1f2-57fb-42d3-bacb-a17bb24e8215 tempest-VolumesSnapshotTestJSON-163973597 None] Delete backup finished, backup 0deae2ce-eba9-444d-bec9-6d109df62c50 deleted. Mar 05 21:16:49.560407 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:16:49.561115 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:17:49.556413 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:17:49.557034 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:18:49.555290 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:18:49.556002 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:19:49.550949 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:19:49.551562 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:20:49.556749 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:20:49.557437 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:21:49.560144 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:21:49.561018 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:22:49.566428 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:22:49.566428 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:23:49.570499 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:23:49.571990 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:24:49.572065 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:24:49.572731 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:25:49.577596 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:25:49.578865 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:26:49.574840 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:26:49.575518 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:27:49.570389 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:27:49.571059 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:28:49.574812 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:28:49.575466 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:29:49.579889 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:29:49.580419 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:30:49.585141 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:30:49.585711 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:31:49.590126 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:31:49.590786 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:32:49.595155 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:32:49.595881 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:33:49.600884 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:33:49.601562 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:34:49.597588 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:34:49.598179 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:35:49.597454 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:35:49.598124 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:36:49.602481 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:36:49.603154 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:37:49.598352 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:37:49.599030 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:38:49.602802 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:38:49.603473 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:39:49.607511 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:39:49.608756 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:40:49.603315 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:40:49.603988 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:41:49.599162 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:41:49.599825 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:42:49.604526 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:42:49.605100 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:43:49.609075 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:43:49.609642 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:44:49.605349 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:44:49.605942 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:45:49.601575 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:45:49.602262 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:46:49.607394 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:46:49.608105 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:47:49.613121 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:47:49.613883 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:49:49.610105 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:49:49.610802 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:50:49.606020 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:50:49.606711 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:51:49.611172 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:51:49.611939 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:52:49.617065 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:52:49.617757 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:53:49.622450 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:53:49.623153 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:54:49.627430 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:54:49.628185 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:55:49.632539 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:55:49.633201 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:56:49.629175 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:56:49.629904 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:57:49.624620 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:57:49.625229 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:58:49.630035 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:58:49.630734 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 21:59:49.635504 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 21:59:49.636419 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 22:00:49.640978 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 22:00:49.641693 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 22:01:49.636926 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 22:01:49.637685 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 22:02:49.641944 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 22:02:49.642635 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 22:03:49.646518 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 22:03:49.647347 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 22:04:49.650997 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 22:04:49.651817 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Mar 05 22:05:49.647211 ncdv-master-1 cinder-backup[134663]: DEBUG oslo_service.periodic_task [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=134663) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Mar 05 22:05:49.647956 ncdv-master-1 cinder-backup[134663]: DEBUG cinder.manager [None req-b0077623-2caf-4d96-b685-b248943cf3f1 None None] Notifying Schedulers of capabilities ... {{(pid=134663) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}