-- Logs begin at Sat 2022-02-19 20:01:15 UTC, end at Sat 2022-02-19 22:17:24 UTC. -- Feb 19 20:40:59.140964 ncdv-master-1 systemd[1]: Started Devstack devstack@c-bak.service. Feb 19 20:41:02.434139 ncdv-master-1 cinder-backup[128784]: INFO cinder.cmd.backup [-] Backup running with 1 processes. Feb 19 20:41:02.435511 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_concurrency.lockutils [-] Acquired lock "singleton_lock" {{(pid=128784) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:294}} Feb 19 20:41:02.436151 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_concurrency.lockutils [-] Releasing lock "singleton_lock" {{(pid=128784) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:312}} Feb 19 20:41:02.599600 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_db.sqlalchemy.engines [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=128784) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Feb 19 20:41:02.806850 ncdv-master-1 cinder-backup[128784]: INFO cinder.rpc [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] Automatically selected cinder-backup objects version 1.39 as minimum service version. Feb 19 20:41:02.811226 ncdv-master-1 cinder-backup[128784]: INFO cinder.rpc [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] Automatically selected cinder-backup RPC version 2.3 as minimum service version. Feb 19 20:41:02.815063 ncdv-master-1 cinder-backup[128784]: INFO cinder.rpc [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] Automatically selected cinder-volume objects version 1.39 as minimum service version. Feb 19 20:41:02.818617 ncdv-master-1 cinder-backup[128784]: INFO cinder.rpc [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] Automatically selected cinder-volume RPC version 3.17 as minimum service version. Feb 19 20:41:02.822849 ncdv-master-1 cinder-backup[128784]: INFO cinder.rpc [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] Automatically selected cinder-scheduler objects version 1.39 as minimum service version. Feb 19 20:41:02.827291 ncdv-master-1 cinder-backup[128784]: INFO cinder.rpc [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] Automatically selected cinder-scheduler RPC version 3.12 as minimum service version. Feb 19 20:41:02.891961 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_db.sqlalchemy.engines [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=128784) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Feb 19 20:41:02.912143 ncdv-master-1 cinder-backup[128784]: INFO oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] Starting 1 workers Feb 19 20:41:02.919311 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] Started child 129230 {{(pid=128784) _start_child /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:575}} Feb 19 20:41:02.920798 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] Full set of CONF: {{(pid=128784) wait /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:649}} Feb 19 20:41:02.921405 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] ******************************************************************************** {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Feb 19 20:41:02.921880 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] Configuration options gathered from: {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Feb 19 20:41:02.922482 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] command line args: ['--config-file', '/etc/cinder/cinder.conf'] {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Feb 19 20:41:02.923340 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] config files: ['/etc/cinder/cinder.conf'] {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Feb 19 20:41:02.923843 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] ================================================================================ {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Feb 19 20:41:02.924836 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] allow_availability_zone_fallback = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.925414 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] allow_compression_on_image_upload = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.926125 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] allowed_direct_url_schemes = [] {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.926759 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] api_rate_limit = True {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.927433 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] auth_strategy = keystone {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.928042 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_availability_zone = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.928642 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backup_api_class = cinder.backup.api.API {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.929429 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backup_ceph_chunk_size = 134217728 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.930159 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backup_ceph_conf = /etc/ceph/ceph.conf {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.930773 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backup_ceph_image_journals = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.931529 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backup_ceph_pool = backups {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.932138 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backup_ceph_stripe_count = 0 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.933294 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backup_ceph_stripe_unit = 0 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.933920 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backup_ceph_user = cinder-bak {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.934550 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backup_driver = cinder.backup.drivers.ceph.CephBackupDriver {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.935121 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backup_driver_init_check_interval = 60 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.935681 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backup_driver_stats_polling_interval = 60 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.936202 ncdv-master-1 cinder-backup[129230]: INFO cinder.service [-] Starting cinder-backup node (version 19.1.0) Feb 19 20:41:02.936954 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backup_manager = cinder.backup.manager.BackupManager {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.937568 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backup_max_operations = 15 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.938151 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backup_metadata_version = 2 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.938808 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backup_name_template = backup-%s {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.939362 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backup_native_threads_pool_size = 60 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.940127 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backup_object_number_per_notification = 10 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.940730 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backup_service_inithost_offload = True {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.941330 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backup_timer_interval = 120 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.942053 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backup_use_same_host = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.942622 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backup_use_temp_snapshot = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.943213 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backup_workers = 1 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.943801 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] chap_password = **** {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.955524 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] chap_username = {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.956169 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.958211 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cinder_internal_tenant_project_id = 676e84132178466da2cf0c656f99ebd2 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.959295 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cinder_internal_tenant_user_id = 9398233d060c43048d49d758695153a3 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.960969 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] compression_format = gzip {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.961860 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] config_dir = [] {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.962718 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] config_file = ['/etc/cinder/cinder.conf'] {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.963029 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] config_source = [] {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.963638 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] consistencygroup_api_class = cinder.consistencygroup.api.API {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.964255 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] control_exchange = openstack {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.964858 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] debug = True {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.965396 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] default_availability_zone = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.965840 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.lockutils [None req-9f40f1c8-9ce6-4cbf-9fab-b5785891bfaf None None] Lock "cleanup_incomplete_backups_128784" acquired by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: waited 0.001s {{(pid=129230) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:386}} Feb 19 20:41:02.966211 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-9f40f1c8-9ce6-4cbf-9fab-b5785891bfaf None None] Cleaning up incomplete backup operations. Feb 19 20:41:02.966499 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] default_group_type = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.967404 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'swiftclient=WARN', 'botocore=WARN'] {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.968319 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] default_volume_type = ceph {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.969542 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] driver_client_cert = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.976412 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] driver_client_cert_key = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.976936 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] driver_data_namespace = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.977324 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] driver_ssl_cert_path = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.977706 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] driver_ssl_cert_verify = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.978120 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] driver_use_ssl = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.978474 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] enable_new_services = True {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.978852 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] enable_unsupported_driver = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.979295 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] enabled_backends = ['ceph'] {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.979719 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] enforce_multipath_for_image_xfer = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.980090 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] filter_function = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.984703 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] glance_api_insecure = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.985226 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] glance_api_servers = ['http://10.222.0.30/image'] {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.985519 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] glance_api_ssl_compression = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.985888 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] glance_ca_certificates_file = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.986289 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] glance_catalog_info = image:glance:publicURL {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.986787 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] glance_certfile = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.987180 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] glance_core_properties = ['checksum', 'container_format', 'disk_format', 'image_name', 'image_id', 'min_disk', 'min_ram', 'name', 'size'] {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.992935 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] glance_keyfile = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.993567 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] glance_num_retries = 3 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.993844 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] glance_request_timeout = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.994213 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] goodness_function = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.994603 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] graceful_shutdown_timeout = 5 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.994995 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] group_api_class = cinder.group.api.API {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.995899 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] host = ncdv-master-1 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.997817 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] iet_conf = /etc/iet/ietd.conf {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.998257 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] image_compress_on_upload = True {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:02.998652 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] image_conversion_address_space_limit = 1 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.000049 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] image_conversion_cpu_limit = 60 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.000621 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] image_conversion_dir = /opt/stack/data/cinder/conversion {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.000995 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] image_upload_use_cinder_backend = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.001369 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] image_upload_use_internal_tenant = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.001731 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] image_volume_cache_enabled = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.002092 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] image_volume_cache_max_count = 0 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.002452 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] image_volume_cache_max_size_gb = 0 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.002836 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] instance_format = [instance: %(uuid)s] {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.003198 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] instance_uuid_format = [instance: %(uuid)s] {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.003614 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] iscsi_iotype = fileio {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.004016 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] iscsi_secondary_ip_addresses = [] {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.004404 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] iscsi_target_flags = {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.004774 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] iscsi_write_cache = on {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.005147 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] iser_helper = tgtadm {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.005749 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] iser_ip_address = 10.222.0.30 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.006153 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] iser_port = 3260 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.006527 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.006910 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] log_config_append = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.007277 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.007620 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] log_dir = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.007991 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] log_file = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.008285 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] log_options = True {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.008681 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] log_rotate_interval = 1 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.009062 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] log_rotate_interval_type = days {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.009447 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] log_rotation_type = none {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.009745 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.010038 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.010403 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.010871 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.011662 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] logging_user_identity_format = %(user)s %(tenant)s %(domain)s %(user_domain)s %(project_domain)s {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.012059 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] max_age = 0 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.012415 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] max_logfile_count = 30 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.012774 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] max_logfile_size_mb = 200 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.013164 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] max_over_subscription_ratio = 20.0 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.013545 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] message_reap_interval = 86400 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.014142 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] message_ttl = 2592000 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.014503 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] monkey_patch = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.014862 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] monkey_patch_modules = [] {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.015188 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] my_ip = 10.222.0.30 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.015562 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] no_snapshot_gb_quota = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.015987 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] num_iser_scan_tries = 3 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.016391 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] num_shell_tries = 3 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.016741 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] num_volume_device_scan_tries = 3 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.017133 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] nvmet_ns_id = 10 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.017494 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] nvmet_port_id = 1 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.017894 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] osapi_max_limit = 1000 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.018289 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] osapi_volume_ext_list = [] {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.018839 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] osapi_volume_extension = ['cinder.api.contrib.standard_extensions'] {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.019091 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] osapi_volume_listen = 0.0.0.0 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.019462 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] osapi_volume_listen_port = 8776 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.019825 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] osapi_volume_use_ssl = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.020185 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] osapi_volume_workers = 2 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.020561 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] per_volume_size_limit = -1 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.020913 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] periodic_fuzzy_delay = 60 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.021283 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] periodic_interval = 60 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.021648 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] public_endpoint = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.022010 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] publish_errors = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.022430 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] quota_backup_gigabytes = 1000 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.022776 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] quota_backups = 10 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.023142 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] quota_consistencygroups = 10 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.023506 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] quota_driver = cinder.quota.DbQuotaDriver {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.023850 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] quota_gigabytes = 1000 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.024241 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] quota_groups = 10 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.024917 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] quota_snapshots = 10 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.025292 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] quota_volumes = 10 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.026165 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] rate_limit_burst = 0 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.026595 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] rate_limit_except_level = CRITICAL {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.026953 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] rate_limit_interval = 0 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.027330 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] replication_device = **** {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.028297 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] report_discard_supported = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.028297 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] report_interval = 10 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.028607 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] reservation_clean_interval = 86400 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.028813 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] reservation_expire = 86400 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.029163 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] reserved_percentage = 0 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.029556 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] resource_query_filters_file = /etc/cinder/resource_filters.json {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.029914 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] restore_discard_excess_bytes = True {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.030271 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] rootwrap_config = /etc/cinder/rootwrap.conf {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.030670 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] rpc_response_timeout = 120 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.031015 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] run_external_periodic_tasks = True {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.031574 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] scheduler_manager = cinder.scheduler.manager.SchedulerManager {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.031957 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] scst_target_driver = iscsi {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.032300 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] scst_target_iqn_name = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.032618 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] service_down_time = 60 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.033020 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] snapshot_name_template = snapshot-%s {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.033446 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] split_loggers = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.033657 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] state_path = /opt/stack/data/cinder {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.034024 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] storage_availability_zone = nova {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.034399 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] storage_protocol = iscsi {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.034774 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] syslog_log_facility = LOG_USER {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.035123 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] target_helper = lioadm {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.035566 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] target_ip_address = 10.222.0.30 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.035934 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] target_port = 3260 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.036286 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] target_prefix = iqn.2010-10.org.openstack: {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.036673 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] target_protocol = iscsi {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.037042 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] trace_flags = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.037399 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] transfer_api_class = cinder.transfer.api.API {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.037779 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] transport_url = **** {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.038129 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] until_refresh = 0 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.038472 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] use_chap_auth = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.038819 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] use_default_quota_class = True {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.039227 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] use_eventlog = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.039490 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] use_journal = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.039829 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] use_json = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.040169 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] use_multipath_for_image_xfer = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.047555 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] use_stderr = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.047555 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] use_syslog = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.047555 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] verify_glance_signatures = enabled {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.047555 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] volume_api_class = cinder.volume.api.API {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.047555 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] volume_backend_name = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.047555 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] volume_clear = zero {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.047555 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] volume_clear_ionice = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.048028 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] volume_clear_size = 0 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.048028 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.048028 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] volume_copy_bps_limit = 0 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.048028 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] volume_dd_blocksize = 1M {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.048028 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] volume_manager = cinder.volume.manager.VolumeManager {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.048028 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] volume_name_template = volume-%s {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.048441 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] volume_usage_audit_period = month {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.048441 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] volumes_dir = /opt/stack/data/cinder/volumes {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.048441 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] watch_log_file = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Feb 19 20:41:03.048441 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.048441 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.048441 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_policy.enforce_new_defaults = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.048813 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_policy.enforce_scope = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.048813 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_policy.policy_default_rule = default {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.048938 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_policy.policy_dirs = ['policy.d'] {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.052422 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_policy.policy_file = policy.yaml {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.052422 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.052422 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.052422 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.052422 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.052422 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.052842 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_concurrency.disable_process_locking = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.052842 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_concurrency.lock_path = /opt/stack/data/cinder {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.052842 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.admin_password = **** {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.053114 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.admin_tenant_name = admin {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.053576 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.admin_token = **** {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.053576 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.admin_user = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.055649 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.auth_admin_prefix = {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.055649 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.055649 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.auth_port = 35357 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.055649 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.auth_protocol = https {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.055649 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.auth_section = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.055974 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.auth_type = password {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.056823 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.auth_uri = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.056823 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.auth_version = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.056823 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.cache = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.057203 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.cafile = /opt/stack/data/ca-bundle.pem {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.057501 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.certfile = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.058597 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.delay_auth_decision = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.058986 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.enforce_token_bind = permissive {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.059334 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.http_connect_timeout = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.064420 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.http_request_max_retries = 3 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.064420 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.identity_uri = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.064420 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.include_service_catalog = True {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.064420 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.insecure = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.064420 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.interface = public {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.064420 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.keyfile = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.064921 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.064921 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.064921 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.064921 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.064921 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.064921 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.memcache_secret_key = **** {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.065278 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.memcache_security_strategy = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.065278 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.065278 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.memcached_servers = ['localhost:11211'] {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.065278 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.region_name = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.065278 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.service_token_roles = ['service'] {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.065581 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.service_token_roles_required = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.066389 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.service_type = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.066389 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.token_cache_time = 300 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.066675 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] keystone_authtoken.www_authenticate_uri = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.067045 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.backend = dogpile.cache.null {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.067429 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.backend_argument = **** {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.067785 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.config_prefix = cache.oslo {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.068198 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.dead_timeout = 60.0 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.068913 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.debug_cache_backend = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.069278 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.enable_retry_client = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.069627 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.enable_socket_keepalive = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.069974 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.enabled = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.070349 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.expiration_time = 600 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.071289 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.hashclient_retry_attempts = 2 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.071289 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.hashclient_retry_delay = 1.0 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.071445 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.memcache_dead_retry = 300 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.071757 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.memcache_pool_connection_get_timeout = 10 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.072104 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.memcache_pool_flush_on_reconnect = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.072491 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.memcache_pool_maxsize = 10 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.072834 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.memcache_pool_unused_timeout = 60 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.073278 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.memcache_servers = ['localhost:11211'] {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.076404 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.memcache_socket_timeout = 1.0 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.076404 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.proxies = [] {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.076404 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.retry_attempts = 2 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.076404 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.retry_delay = 0.0 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.076404 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.socket_keepalive_count = 1 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.076404 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.socket_keepalive_idle = 1 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.076404 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.socket_keepalive_interval = 1 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.077480 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.tls_allowed_ciphers = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.077480 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.tls_cafile = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.077792 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.tls_certfile = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.078105 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.tls_enabled = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.078456 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cache.tls_keyfile = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.078849 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] coordination.backend_url = etcd3+http://10.222.0.30:2379 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.079275 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] database.backend = sqlalchemy {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.079567 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] database.connection = **** {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.080175 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] database.connection_debug = 0 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.080661 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] database.connection_parameters = {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.080931 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] database.connection_recycle_time = 3600 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.081289 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] database.connection_trace = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.084470 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] database.db_inc_retry_interval = True {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.084470 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] database.db_max_retries = 20 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.084470 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] database.db_max_retry_interval = 10 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.084470 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] database.db_retry_interval = 1 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.084470 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] database.max_overflow = 50 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.084470 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] database.max_pool_size = 5 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.084470 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] database.max_retries = 10 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.086159 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] database.mysql_enable_ndb = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.086159 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] database.mysql_sql_mode = TRADITIONAL {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.086159 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] database.pool_timeout = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.086159 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] database.retry_interval = 10 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.086159 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] database.slave_connection = **** {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.086159 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] database.sqlite_synchronous = True {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.086508 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] database.use_db_reconnect = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.087050 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.087155 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.087484 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_metrics.metrics_process_name = {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.087845 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.088191 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.088762 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] profiler.connection_string = messaging:// {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.089136 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] profiler.enabled = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.089519 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] profiler.es_doc_type = notification {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.089893 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] profiler.es_scroll_size = 10000 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.090234 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] profiler.es_scroll_time = 2m {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.090589 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] profiler.filter_error_trace = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.090958 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] profiler.hmac_keys = SECRET_KEY {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.093269 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] profiler.sentinel_service_name = mymaster {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.093269 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] profiler.socket_timeout = 0.1 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.093269 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] profiler.trace_sqlalchemy = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.093269 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] privsep_osbrick.capabilities = [21] {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.093269 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] privsep_osbrick.group = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.093625 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] privsep_osbrick.helper_command = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.093984 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] privsep_osbrick.logger_name = oslo_privsep.daemon {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.094246 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] privsep_osbrick.thread_pool_size = 4 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.094797 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] privsep_osbrick.user = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.095811 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cinder_sys_admin.capabilities = [0, 1, 2, 3, 12, 21] {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.095811 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cinder_sys_admin.group = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.095811 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cinder_sys_admin.helper_command = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.096034 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cinder_sys_admin.logger_name = oslo_privsep.daemon {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.096403 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cinder_sys_admin.thread_pool_size = 4 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.096781 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] cinder_sys_admin.user = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.097148 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] service_user.auth_section = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.097493 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] service_user.auth_type = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.098029 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] service_user.cafile = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.098377 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] service_user.certfile = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.098747 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] service_user.collect_timing = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.099106 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] service_user.insecure = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.099437 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] service_user.keyfile = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.099781 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] service_user.send_service_user_token = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.100136 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] service_user.split_loggers = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.100522 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] service_user.timeout = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.101315 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.backend_availability_zone = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.101315 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.chap_password = **** {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.101661 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.chap_username = {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.102022 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.103379 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.deferred_deletion_delay = 0 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.103379 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.deferred_deletion_purge_interval = 60 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.103379 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.driver_client_cert = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.103379 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.driver_client_cert_key = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.103700 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.driver_data_namespace = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.104194 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.driver_ssl_cert_path = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.105145 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.driver_ssl_cert_verify = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.105750 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.driver_use_ssl = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.106342 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.enable_deferred_deletion = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.106914 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.enable_unsupported_driver = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.107472 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.enforce_multipath_for_image_xfer = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.108054 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.filter_function = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.108815 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.goodness_function = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.109412 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.iet_conf = /etc/iet/ietd.conf {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.110218 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.image_upload_use_cinder_backend = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.110790 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.image_upload_use_internal_tenant = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.111347 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.image_volume_cache_enabled = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.111940 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.image_volume_cache_max_count = 0 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.112548 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.image_volume_cache_max_size_gb = 0 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.113123 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.iscsi_iotype = fileio {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.113689 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.iscsi_secondary_ip_addresses = [] {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.114256 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.iscsi_target_flags = {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.114849 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.iscsi_write_cache = on {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.115435 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.iser_helper = tgtadm {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.116201 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.iser_ip_address = 10.222.0.30 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.117022 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.iser_port = 3260 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.117608 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.118192 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.max_over_subscription_ratio = 20.0 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.118757 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.num_iser_scan_tries = 3 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.119333 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.num_shell_tries = 3 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.119895 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.num_volume_device_scan_tries = 3 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.121044 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.nvmet_ns_id = 10 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.122307 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.nvmet_port_id = 1 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.122928 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.rados_connect_timeout = -1 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.123709 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.rados_connection_interval = 5 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.124270 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.rados_connection_retries = 3 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.124890 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.rbd_ceph_conf = {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.125491 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.rbd_cluster_name = ceph {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.126235 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.rbd_exclusive_cinder_pool = True {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.126821 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.rbd_flatten_volume_from_snapshot = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.127390 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.rbd_max_clone_depth = 5 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.127976 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.rbd_pool = rbd {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.128559 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.rbd_secret_uuid = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.129152 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.rbd_store_chunk_size = 4 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.129943 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.rbd_user = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.130861 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.replication_connect_timeout = 5 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.131427 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.replication_device = **** {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.131982 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.report_discard_supported = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.132908 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.report_dynamic_total_capacity = True {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.133502 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.reserved_percentage = 0 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.134079 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.scst_target_driver = iscsi {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.134630 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.scst_target_iqn_name = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.135218 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.storage_protocol = iscsi {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.136372 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.target_helper = tgtadm {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.137070 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.target_ip_address = 10.222.0.30 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.137671 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.target_port = 3260 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.138250 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.target_prefix = iqn.2010-10.org.openstack: {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.138835 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.target_protocol = iscsi {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.139391 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.trace_flags = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.139948 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.unique_fqdn_network = True {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.140791 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.use_chap_auth = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.141355 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.use_multipath_for_image_xfer = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.141922 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.volume_backend_name = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.142501 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.volume_clear = zero {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.143081 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.volume_clear_ionice = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.143657 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.volume_clear_size = 0 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.144236 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.144834 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.volume_copy_bps_limit = 0 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.145426 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.volume_dd_blocksize = 1M {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.146104 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] backend_defaults.volumes_dir = /opt/stack/data/cinder/volumes {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.146693 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_reports.file_event_handler = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.147310 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_reports.file_event_handler_interval = 1 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.147879 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_reports.log_dir = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.148568 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.149151 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.149753 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.150342 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.150928 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.151556 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.152133 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_rabbit.heartbeat_in_pthread = True {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.152949 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_rabbit.heartbeat_rate = 2 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.153521 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.154109 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.154728 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.155296 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.155886 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.156507 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.157071 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.157677 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.158255 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.158822 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.159392 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.159971 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.160574 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.161174 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.161758 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_rabbit.ssl = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.162458 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_rabbit.ssl_ca_file = {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.163286 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_rabbit.ssl_cert_file = {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.163864 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_rabbit.ssl_key_file = {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.164495 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_rabbit.ssl_version = {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.165122 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_notifications.driver = [] {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.165712 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_notifications.retry = -1 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.166309 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.166909 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] oslo_messaging_notifications.transport_url = **** {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.168113 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] key_manager.backend = cinder.keymgr.conf_key_mgr.ConfKeyManager {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.168786 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] barbican.auth_endpoint = http://localhost/identity/v3 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.169609 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] barbican.barbican_api_version = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.170171 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] barbican.barbican_endpoint = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.170768 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] barbican.barbican_endpoint_type = public {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.171317 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] barbican.barbican_region_name = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.171885 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] barbican.cafile = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.172490 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] barbican.certfile = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.173075 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] barbican.collect_timing = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.173630 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] barbican.insecure = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.174216 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] barbican.keyfile = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.174873 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] barbican.number_of_retries = 60 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.175429 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] barbican.retry_delay = 1 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.176015 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] barbican.send_service_user_token = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.176637 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] barbican.split_loggers = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.177188 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] barbican.timeout = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.177766 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] barbican.verify_ssl = True {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.178326 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] barbican.verify_ssl_path = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.178890 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] barbican_service_user.auth_section = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.179445 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] barbican_service_user.auth_type = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.180011 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] barbican_service_user.cafile = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.181347 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] barbican_service_user.certfile = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.181943 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] barbican_service_user.collect_timing = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.182510 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] barbican_service_user.insecure = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.183644 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] barbican_service_user.keyfile = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.184201 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] barbican_service_user.split_loggers = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.184779 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] barbican_service_user.timeout = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.185396 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] vault.approle_role_id = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.185949 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] vault.approle_secret_id = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.186511 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] vault.cafile = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.187080 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] vault.certfile = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.187657 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] vault.collect_timing = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.188228 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] vault.insecure = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.188849 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] vault.keyfile = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.189452 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] vault.kv_mountpoint = secret {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.190011 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] vault.kv_version = 2 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.190561 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] vault.namespace = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.191108 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] vault.root_token_id = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.191682 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] vault.split_loggers = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.192239 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] vault.ssl_ca_crt_file = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.192988 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] vault.timeout = None {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.193561 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] vault.use_ssl = False {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.194151 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] vault.vault_url = http://127.0.0.1:8200 {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Feb 19 20:41:03.194728 ncdv-master-1 cinder-backup[128784]: DEBUG oslo_service.service [None req-12635c3c-7b93-46d6-918f-aa29dfcf79eb None None] ******************************************************************************** {{(pid=128784) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Feb 19 20:41:03.218341 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_db.sqlalchemy.engines [None req-9f40f1c8-9ce6-4cbf-9fab-b5785891bfaf None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129230) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Feb 19 20:41:03.260499 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.lockutils [None req-9f40f1c8-9ce6-4cbf-9fab-b5785891bfaf None None] Lock "cleanup_incomplete_backups_128784" "released" by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: held 0.295s {{(pid=129230) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:400}} Feb 19 20:41:03.266633 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_db.sqlalchemy.engines [None req-9f40f1c8-9ce6-4cbf-9fab-b5785891bfaf None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129230) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Feb 19 20:41:03.273033 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-9f40f1c8-9ce6-4cbf-9fab-b5785891bfaf None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 20:41:03.290268 ncdv-master-1 cinder-backup[129230]: INFO cinder.keymgr.migration [None req-91e16c9b-70c5-44d5-a6c8-7cb65e27beaa None None] Not migrating encryption keys because the ConfKeyManager is still in use. Feb 19 20:41:03.317116 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.service [None req-9f40f1c8-9ce6-4cbf-9fab-b5785891bfaf None None] Creating RPC server for service cinder-backup {{(pid=129230) start /opt/stack/cinder/cinder/service.py:231}} Feb 19 20:41:03.330760 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.service [None req-9f40f1c8-9ce6-4cbf-9fab-b5785891bfaf None None] Pinning object versions for RPC server serializer to 1.39 {{(pid=129230) start /opt/stack/cinder/cinder/service.py:237}} Feb 19 20:42:03.376364 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 20:42:03.377060 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 20:44:03.379138 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 20:44:03.379749 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 20:45:03.384175 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 20:45:03.384776 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 20:46:03.389185 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 20:46:03.389831 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 20:47:03.395258 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 20:47:03.395826 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 20:48:03.400193 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 20:48:03.400796 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 20:49:03.405315 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 20:49:03.405899 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 20:50:03.410291 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 20:50:03.411015 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 20:51:03.407347 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 20:51:03.408911 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 20:52:03.404637 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 20:52:03.405934 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 20:53:03.404308 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 20:53:03.405005 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 20:54:03.404395 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 20:54:03.405118 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 20:55:03.405396 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 20:55:03.406563 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 20:56:03.403768 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 20:56:03.404418 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 20:57:03.401508 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 20:57:03.402819 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 20:58:03.402900 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 20:58:03.403519 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 20:59:03.407743 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 20:59:03.408452 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:00:03.403986 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:00:03.405185 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:01:03.400484 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:01:03.401107 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:02:03.398728 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:02:03.400197 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:03:03.398965 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:03:03.399623 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:04:00.165288 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] Create backup started, backup: e1c58687-1994-4d46-9f9b-29fc0aee6533 volume: 3c4659bd-63a9-4ccf-8885-d4725562e8a0 snapshot: 1a3ea589-1e1c-4cc0-ad9c-0f18a0026976. Feb 19 21:04:00.179859 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-02-19T21:03:59Z,data_timestamp=2022-02-19T21:03:58Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Backup-314568189',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=e1c58687-1994-4d46-9f9b-29fc0aee6533,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='ba9b396737a34280bdc448a2b0df5a7e',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id='1a3ea589-1e1c-4cc0-ad9c-0f18a0026976',status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='0edf621841d44514b6082172d15af2a7',volume_id=3c4659bd-63a9-4ccf-8885-d4725562e8a0) Feb 19 21:04:00.853462 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.utils [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:04:00.861503 ncdv-master-1 cinder-backup[129230]: INFO oslo.privsep.daemon [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] Running privsep helper: ['sudo', 'cinder-rootwrap', '/etc/cinder/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/cinder/cinder.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmp3gzg53ih/privsep.sock'] Feb 19 21:04:00.954654 ncdv-master-1 sudo[150036]: ubuntu : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf privsep-helper --config-file /etc/cinder/cinder.conf --privsep_context os_brick.privileged.default --privsep_sock_path /tmp/tmp3gzg53ih/privsep.sock Feb 19 21:04:00.961522 ncdv-master-1 sudo[150036]: pam_unix(sudo:session): session opened for user root by (uid=0) Feb 19 21:04:03.396185 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:04:03.397266 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:04:03.576375 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_db.sqlalchemy.engines [None req-0310f1a8-baa1-43c0-8dfd-d981a2de117f None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129230) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Feb 19 21:04:06.663716 ncdv-master-1 sudo[150036]: pam_unix(sudo:session): session closed for user root Feb 19 21:04:06.679457 ncdv-master-1 cinder-backup[129230]: INFO oslo.privsep.daemon [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] Spawned new privsep daemon via rootwrap Feb 19 21:04:06.682094 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] Accepted privsep connection to /tmp/tmp3gzg53ih/privsep.sock {{(pid=129230) __init__ /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:362}} Feb 19 21:04:06.683837 ncdv-master-1 cinder-backup[129230]: INFO oslo.privsep.daemon [-] privsep daemon starting Feb 19 21:04:06.684410 ncdv-master-1 cinder-backup[129230]: INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 Feb 19 21:04:06.685031 ncdv-master-1 cinder-backup[129230]: INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none Feb 19 21:04:06.685618 ncdv-master-1 cinder-backup[129230]: INFO oslo.privsep.daemon [-] privsep daemon running as pid 150093 Feb 19 21:04:06.702280 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[776d6474-75d0-4cb5-b14b-a2050725cddc]: (2,) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:04:07.181016 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:04:07.233641 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.053s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:04:07.233972 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[ee624361-ca83-4178-805e-43cd4e501d5e]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:6e78a5d1a02d\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:04:07.235411 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:04:07.267391 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Feb 19 21:04:07.290278 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[c813494f-fb69-4af2-887f-cf9c9e7c5174]: [Errno 2] No such file or directory: 'systool' {{(pid=150093) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Feb 19 21:04:07.290278 ncdv-master-1 cinder-backup[129230]: Traceback (most recent call last): Feb 19 21:04:07.290278 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Feb 19 21:04:07.290278 ncdv-master-1 cinder-backup[129230]: ret = func(*f_args, **f_kwargs) Feb 19 21:04:07.290278 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Feb 19 21:04:07.290278 ncdv-master-1 cinder-backup[129230]: return func(*args, **kwargs) Feb 19 21:04:07.290278 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Feb 19 21:04:07.290278 ncdv-master-1 cinder-backup[129230]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Feb 19 21:04:07.290278 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Feb 19 21:04:07.290278 ncdv-master-1 cinder-backup[129230]: return putils.execute(on_execute=on_execute, Feb 19 21:04:07.290278 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Feb 19 21:04:07.290278 ncdv-master-1 cinder-backup[129230]: obj = subprocess.Popen(cmd, Feb 19 21:04:07.290278 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Feb 19 21:04:07.290278 ncdv-master-1 cinder-backup[129230]: self._execute_child(args, executable, preexec_fn, close_fds, Feb 19 21:04:07.290278 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Feb 19 21:04:07.290278 ncdv-master-1 cinder-backup[129230]: raise child_exception_type(errno_num, err_msg, err_filename) Feb 19 21:04:07.290278 ncdv-master-1 cinder-backup[129230]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Feb 19 21:04:07.292384 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[c813494f-fb69-4af2-887f-cf9c9e7c5174]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:04:07.293396 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:04:07.375730 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Feb 19 21:04:07.376253 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[1d456d77-aa94-446f-918c-65125d540fe3]: [Errno 2] No such file or directory: 'systool' {{(pid=150093) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Feb 19 21:04:07.376253 ncdv-master-1 cinder-backup[129230]: Traceback (most recent call last): Feb 19 21:04:07.376253 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Feb 19 21:04:07.376253 ncdv-master-1 cinder-backup[129230]: ret = func(*f_args, **f_kwargs) Feb 19 21:04:07.376253 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Feb 19 21:04:07.376253 ncdv-master-1 cinder-backup[129230]: return func(*args, **kwargs) Feb 19 21:04:07.376253 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Feb 19 21:04:07.376253 ncdv-master-1 cinder-backup[129230]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Feb 19 21:04:07.376253 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Feb 19 21:04:07.376253 ncdv-master-1 cinder-backup[129230]: return putils.execute(on_execute=on_execute, Feb 19 21:04:07.376253 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Feb 19 21:04:07.376253 ncdv-master-1 cinder-backup[129230]: obj = subprocess.Popen(cmd, Feb 19 21:04:07.376253 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Feb 19 21:04:07.376253 ncdv-master-1 cinder-backup[129230]: self._execute_child(args, executable, preexec_fn, close_fds, Feb 19 21:04:07.376253 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Feb 19 21:04:07.376253 ncdv-master-1 cinder-backup[129230]: raise child_exception_type(errno_num, err_msg, err_filename) Feb 19 21:04:07.376253 ncdv-master-1 cinder-backup[129230]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Feb 19 21:04:07.377687 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[1d456d77-aa94-446f-918c-65125d540fe3]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:04:07.379603 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:04:07.453553 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.074s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:04:07.454005 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[f91b0136-6f37-4b64-85bf-019baef93b9a]: (4, ('/dev/sda1\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:04:07.456122 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:04:07.560494 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.104s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:04:07.564262 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[0f8365cd-d885-4b82-9779-8947ef907982]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:04:07.566620 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:04:07.613968 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.047s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:04:07.614841 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[757c1c86-c50e-4f71-8332-ee66640aeae7]: (4, ('856ceada-2af3-8e47-a879-77ebed99126e\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:04:07.616637 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] Running cmd (subprocess): nvme version {{(pid=129230) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:04:07.679592 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] CMD "nvme version" returned: 0 in 0.063s {{(pid=129230) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:04:07.695294 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129230) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Feb 19 21:04:07.695835 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129230) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Feb 19 21:04:07.696492 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:9cde57c4-ac04-4e2d-ab4a-203cb09943a6 dsc: {{(pid=129230) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Feb 19 21:04:07.697921 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.utils [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] <== get_connector_properties: return (6843ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:6e78a5d1a02d', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '856ceada-2af3-8e47-a879-77ebed99126e', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:9cde57c4-ac04-4e2d-ab4a-203cb09943a6', 'found_dsc': ''} {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:04:08.969514 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connector [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] Factory for rbd on None {{(pid=129230) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Feb 19 21:04:08.971977 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-66378b9d-c00b-4108-96c5-6e98f5196713', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '66378b9d-c00b-4108-96c5-6e98f5196713', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:04:08.973398 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.linuxrbd [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] opening connection to ceph cluster (timeout=-1). {{(pid=129230) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Feb 19 21:04:09.096973 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] <== connect_volume: return (122ms) {'path': } {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:04:09.122910 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] Starting backup of volume='3c4659bd-63a9-4ccf-8885-d4725562e8a0'. {{(pid=129230) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Feb 19 21:04:09.188165 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129230) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Feb 19 21:04:09.189568 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] Creating backup base image='volume-3c4659bd-63a9-4ccf-8885-d4725562e8a0.backup.base' for volume 3c4659bd-63a9-4ccf-8885-d4725562e8a0. {{(pid=129230) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:830}} Feb 19 21:04:10.462166 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] Copying data from volume 3c4659bd-63a9-4ccf-8885-d4725562e8a0. {{(pid=129230) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:842}} Feb 19 21:04:10.487909 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] Transferring data between 'volume-3c4659bd-63a9-4ccf-8885-d4725562e8a0' and 'volume-3c4659bd-63a9-4ccf-8885-d4725562e8a0.backup.base' {{(pid=129230) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:399}} Feb 19 21:04:10.489369 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] 8 chunks of 134217728 bytes to be transferred {{(pid=129230) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:403}} Feb 19 21:04:12.720121 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] Transferred chunk 1 of 8 (58787K/s) {{(pid=129230) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Feb 19 21:04:14.285724 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] Transferred chunk 2 of 8 (83936K/s) {{(pid=129230) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Feb 19 21:04:16.086381 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] Transferred chunk 3 of 8 (72818K/s) {{(pid=129230) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Feb 19 21:04:17.582853 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] Transferred chunk 4 of 8 (87633K/s) {{(pid=129230) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Feb 19 21:04:19.233078 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] Transferred chunk 5 of 8 (79548K/s) {{(pid=129230) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Feb 19 21:04:20.780118 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] Transferred chunk 6 of 8 (84802K/s) {{(pid=129230) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Feb 19 21:04:22.041712 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] Transferred chunk 7 of 8 (103952K/s) {{(pid=129230) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Feb 19 21:04:23.429397 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] Transferred chunk 8 of 8 (94549K/s) {{(pid=129230) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Feb 19 21:04:23.488672 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] Getting metadata type 'volume-base-metadata' {{(pid=129230) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Feb 19 21:04:23.512284 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129230) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Feb 19 21:04:23.512897 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] Getting metadata type 'volume-metadata' {{(pid=129230) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Feb 19 21:04:23.527793 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] No metadata type 'volume-metadata' available {{(pid=129230) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Feb 19 21:04:23.531994 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] Getting metadata type 'volume-glance-metadata' {{(pid=129230) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Feb 19 21:04:23.554440 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] No metadata type 'volume-glance-metadata' available {{(pid=129230) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Feb 19 21:04:23.555179 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] Backing up metadata for volume 3c4659bd-63a9-4ccf-8885-d4725562e8a0. {{(pid=129230) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Feb 19 21:04:23.656569 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] Backup 'e1c58687-1994-4d46-9f9b-29fc0aee6533' of volume 3c4659bd-63a9-4ccf-8885-d4725562e8a0 finished. {{(pid=129230) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Feb 19 21:04:23.657744 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-66378b9d-c00b-4108-96c5-6e98f5196713', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '66378b9d-c00b-4108-96c5-6e98f5196713', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:04:23.679007 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] <== disconnect_volume: return (20ms) None {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:04:23.856169 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-628316d9-3cff-4536-880c-663a36eb7d87 tempest-VolumesSnapshotTestJSON-2140298196 None] Create backup finished. backup: e1c58687-1994-4d46-9f9b-29fc0aee6533. Feb 19 21:04:24.439210 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-b238b9a8-d8c1-4572-975c-c93775fec306 tempest-VolumesSnapshotTestJSON-2140298196 None] Delete backup started, backup: e1c58687-1994-4d46-9f9b-29fc0aee6533. Feb 19 21:04:24.443725 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-b238b9a8-d8c1-4572-975c-c93775fec306 tempest-VolumesSnapshotTestJSON-2140298196 None] Delete started for backup=e1c58687-1994-4d46-9f9b-29fc0aee6533 {{(pid=129230) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Feb 19 21:04:24.445467 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-b238b9a8-d8c1-4572-975c-c93775fec306 tempest-VolumesSnapshotTestJSON-2140298196 None] Trying diff format basename='volume-3c4659bd-63a9-4ccf-8885-d4725562e8a0.backup.e1c58687-1994-4d46-9f9b-29fc0aee6533' for backup base image of volume 3c4659bd-63a9-4ccf-8885-d4725562e8a0. {{(pid=129230) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Feb 19 21:04:24.528939 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-b238b9a8-d8c1-4572-975c-c93775fec306 tempest-VolumesSnapshotTestJSON-2140298196 None] Image 'volume-3c4659bd-63a9-4ccf-8885-d4725562e8a0.backup.e1c58687-1994-4d46-9f9b-29fc0aee6533' not found - trying diff format name {{(pid=129230) _rbd_image_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:661}} Feb 19 21:04:24.555945 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-b238b9a8-d8c1-4572-975c-c93775fec306 tempest-VolumesSnapshotTestJSON-2140298196 None] Looking for snapshot of backup base 'volume-3c4659bd-63a9-4ccf-8885-d4725562e8a0.backup.base' {{(pid=129230) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Feb 19 21:04:24.557104 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-b238b9a8-d8c1-4572-975c-c93775fec306 tempest-VolumesSnapshotTestJSON-2140298196 None] Backup base 'volume-3c4659bd-63a9-4ccf-8885-d4725562e8a0.backup.base' has no snapshots {{(pid=129230) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:908}} Feb 19 21:04:24.558212 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-b238b9a8-d8c1-4572-975c-c93775fec306 tempest-VolumesSnapshotTestJSON-2140298196 None] No backup snapshot to delete {{(pid=129230) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:476}} Feb 19 21:04:24.568846 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.drivers.ceph [None req-b238b9a8-d8c1-4572-975c-c93775fec306 tempest-VolumesSnapshotTestJSON-2140298196 None] Deleting backup base image='volume-3c4659bd-63a9-4ccf-8885-d4725562e8a0.backup.base' of volume 3c4659bd-63a9-4ccf-8885-d4725562e8a0. Feb 19 21:04:25.745615 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-b238b9a8-d8c1-4572-975c-c93775fec306 tempest-VolumesSnapshotTestJSON-2140298196 None] Base backup image='volume-3c4659bd-63a9-4ccf-8885-d4725562e8a0.backup.base' of volume 3c4659bd-63a9-4ccf-8885-d4725562e8a0 deleted. {{(pid=129230) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Feb 19 21:04:25.830772 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-b238b9a8-d8c1-4572-975c-c93775fec306 tempest-VolumesSnapshotTestJSON-2140298196 None] Delete of backup 'e1c58687-1994-4d46-9f9b-29fc0aee6533' for volume '3c4659bd-63a9-4ccf-8885-d4725562e8a0' finished. {{(pid=129230) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Feb 19 21:04:25.839736 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_db.sqlalchemy.engines [None req-b238b9a8-d8c1-4572-975c-c93775fec306 tempest-VolumesSnapshotTestJSON-2140298196 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129230) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Feb 19 21:04:25.889913 ncdv-master-1 cinder-backup[129230]: WARNING cinder.quota [None req-b238b9a8-d8c1-4572-975c-c93775fec306 tempest-VolumesSnapshotTestJSON-2140298196 None] Deprecated: Default quota for resource: backups is set by the default quota flag: quota_backups, it is now deprecated. Please use the default quota class for default quota. Feb 19 21:04:25.896113 ncdv-master-1 cinder-backup[129230]: WARNING cinder.quota [None req-b238b9a8-d8c1-4572-975c-c93775fec306 tempest-VolumesSnapshotTestJSON-2140298196 None] Deprecated: Default quota for resource: backup_gigabytes is set by the default quota flag: quota_backup_gigabytes, it is now deprecated. Please use the default quota class for default quota. Feb 19 21:04:25.957448 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.quota [None req-b238b9a8-d8c1-4572-975c-c93775fec306 tempest-VolumesSnapshotTestJSON-2140298196 None] Created reservations ['4edf2005-a233-4324-bea8-bdfe4612aca2', '2f978145-9bd7-4767-a416-8e7340abc93a'] {{(pid=129230) reserve /opt/stack/cinder/cinder/quota.py:755}} Feb 19 21:04:26.050999 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-b238b9a8-d8c1-4572-975c-c93775fec306 tempest-VolumesSnapshotTestJSON-2140298196 None] Delete backup finished, backup e1c58687-1994-4d46-9f9b-29fc0aee6533 deleted. Feb 19 21:05:03.400995 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:05:03.402236 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:06:03.403495 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:06:03.404251 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:07:03.401222 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:07:03.402009 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:08:03.406605 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:08:03.407223 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:09:03.404619 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:09:03.405426 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:10:03.402585 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:10:03.404094 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:11:00.618209 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] Create backup started, backup: d8578643-04d6-4c96-9ae5-e6d15d45c94b volume: 46718fc1-de22-4f32-9233-e05f81bf31e3. Feb 19 21:11:00.638107 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-02-19T21:11:00Z,data_timestamp=2022-02-19T21:11:00Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsTest-Backup-1501626097',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=d8578643-04d6-4c96-9ae5-e6d15d45c94b,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='463a97fa588d4fc2bb01401ca0fcb8aa',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='52c7a837477849768d5a90636fddf7b4',volume_id=46718fc1-de22-4f32-9233-e05f81bf31e3) Feb 19 21:11:00.702447 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.utils [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:11:00.704484 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:11:00.738600 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.034s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:11:00.739247 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[1ee70046-925f-4221-ba56-4e21748b63c2]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:6e78a5d1a02d\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:11:00.741577 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:11:00.767776 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Feb 19 21:11:00.768904 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[e48bc859-d831-4da4-96f5-49f7a1fef17b]: [Errno 2] No such file or directory: 'systool' {{(pid=150093) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Feb 19 21:11:00.768904 ncdv-master-1 cinder-backup[129230]: Traceback (most recent call last): Feb 19 21:11:00.768904 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Feb 19 21:11:00.768904 ncdv-master-1 cinder-backup[129230]: ret = func(*f_args, **f_kwargs) Feb 19 21:11:00.768904 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Feb 19 21:11:00.768904 ncdv-master-1 cinder-backup[129230]: return func(*args, **kwargs) Feb 19 21:11:00.768904 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Feb 19 21:11:00.768904 ncdv-master-1 cinder-backup[129230]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Feb 19 21:11:00.768904 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Feb 19 21:11:00.768904 ncdv-master-1 cinder-backup[129230]: return putils.execute(on_execute=on_execute, Feb 19 21:11:00.768904 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Feb 19 21:11:00.768904 ncdv-master-1 cinder-backup[129230]: obj = subprocess.Popen(cmd, Feb 19 21:11:00.768904 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Feb 19 21:11:00.768904 ncdv-master-1 cinder-backup[129230]: self._execute_child(args, executable, preexec_fn, close_fds, Feb 19 21:11:00.768904 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Feb 19 21:11:00.768904 ncdv-master-1 cinder-backup[129230]: raise child_exception_type(errno_num, err_msg, err_filename) Feb 19 21:11:00.768904 ncdv-master-1 cinder-backup[129230]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Feb 19 21:11:00.770712 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[e48bc859-d831-4da4-96f5-49f7a1fef17b]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:11:00.774635 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:11:00.782785 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Feb 19 21:11:00.783331 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[583b35f5-3a13-44f7-b583-33cd174e4c41]: [Errno 2] No such file or directory: 'systool' {{(pid=150093) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Feb 19 21:11:00.783331 ncdv-master-1 cinder-backup[129230]: Traceback (most recent call last): Feb 19 21:11:00.783331 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Feb 19 21:11:00.783331 ncdv-master-1 cinder-backup[129230]: ret = func(*f_args, **f_kwargs) Feb 19 21:11:00.783331 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Feb 19 21:11:00.783331 ncdv-master-1 cinder-backup[129230]: return func(*args, **kwargs) Feb 19 21:11:00.783331 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Feb 19 21:11:00.783331 ncdv-master-1 cinder-backup[129230]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Feb 19 21:11:00.783331 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Feb 19 21:11:00.783331 ncdv-master-1 cinder-backup[129230]: return putils.execute(on_execute=on_execute, Feb 19 21:11:00.783331 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Feb 19 21:11:00.783331 ncdv-master-1 cinder-backup[129230]: obj = subprocess.Popen(cmd, Feb 19 21:11:00.783331 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Feb 19 21:11:00.783331 ncdv-master-1 cinder-backup[129230]: self._execute_child(args, executable, preexec_fn, close_fds, Feb 19 21:11:00.783331 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Feb 19 21:11:00.783331 ncdv-master-1 cinder-backup[129230]: raise child_exception_type(errno_num, err_msg, err_filename) Feb 19 21:11:00.783331 ncdv-master-1 cinder-backup[129230]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Feb 19 21:11:00.785122 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[583b35f5-3a13-44f7-b583-33cd174e4c41]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:11:00.789142 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:11:00.821504 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.034s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:11:00.824826 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[917eb307-f396-465c-b346-5fc617e33df1]: (4, ('/dev/sda1\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:11:00.829616 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:11:00.890364 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.064s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:11:00.891021 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[4157ca22-b7ff-4416-aa07-4c409c9a69d6]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:11:00.894289 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:11:00.906364 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.013s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:11:00.907414 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[03c55c99-ecbb-4752-b956-9ef052138565]: (4, ('856ceada-2af3-8e47-a879-77ebed99126e\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:11:00.908853 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] Running cmd (subprocess): nvme version {{(pid=129230) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:11:00.947626 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] CMD "nvme version" returned: 0 in 0.036s {{(pid=129230) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:11:00.947626 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129230) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Feb 19 21:11:00.947626 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129230) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Feb 19 21:11:00.947626 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:9cde57c4-ac04-4e2d-ab4a-203cb09943a6 dsc: {{(pid=129230) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Feb 19 21:11:00.947965 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.utils [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] <== get_connector_properties: return (244ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:6e78a5d1a02d', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '856ceada-2af3-8e47-a879-77ebed99126e', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:9cde57c4-ac04-4e2d-ab4a-203cb09943a6', 'found_dsc': ''} {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:11:01.960426 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connector [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] Factory for rbd on None {{(pid=129230) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Feb 19 21:11:01.962608 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-46718fc1-de22-4f32-9233-e05f81bf31e3', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '46718fc1-de22-4f32-9233-e05f81bf31e3', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:11:01.964036 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.linuxrbd [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] opening connection to ceph cluster (timeout=-1). {{(pid=129230) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Feb 19 21:11:02.040252 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] <== connect_volume: return (77ms) {'path': } {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:11:02.051714 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_db.sqlalchemy.engines [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129230) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Feb 19 21:11:02.083280 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] Starting backup of volume='46718fc1-de22-4f32-9233-e05f81bf31e3'. {{(pid=129230) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Feb 19 21:11:02.084542 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] Volume file is RBD: attempting optimized backup {{(pid=129230) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Feb 19 21:11:02.135952 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] Creating base image 'volume-46718fc1-de22-4f32-9233-e05f81bf31e3.backup.d8578643-04d6-4c96-9ae5-e6d15d45c94b' {{(pid=129230) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Feb 19 21:11:02.136493 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129230) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Feb 19 21:11:02.192204 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] Using --from-snap 'None' for incremental backup of volume 46718fc1-de22-4f32-9233-e05f81bf31e3. {{(pid=129230) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Feb 19 21:11:02.192758 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] Creating backup snapshot='backup.d8578643-04d6-4c96-9ae5-e6d15d45c94b.snap.1645305062.1922343' {{(pid=129230) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Feb 19 21:11:02.840878 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] Performing differential transfer from 'volume-46718fc1-de22-4f32-9233-e05f81bf31e3' to 'volume-46718fc1-de22-4f32-9233-e05f81bf31e3.backup.d8578643-04d6-4c96-9ae5-e6d15d45c94b' {{(pid=129230) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Feb 19 21:11:02.841501 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_udoqtb_w --pool volumes volumes/volume-46718fc1-de22-4f32-9233-e05f81bf31e3@backup.d8578643-04d6-4c96-9ae5-e6d15d45c94b.snap.1645305062.1922343 -' into... {{(pid=129230) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Feb 19 21:11:02.841895 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-46718fc1-de22-4f32-9233-e05f81bf31e3.backup.d8578643-04d6-4c96-9ae5-e6d15d45c94b' {{(pid=129230) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Feb 19 21:11:03.401035 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:11:03.401613 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:11:03.966242 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] Differential backup transfer completed in 1.1248s {{(pid=129230) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Feb 19 21:11:03.967342 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] Getting metadata type 'volume-base-metadata' {{(pid=129230) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Feb 19 21:11:03.995846 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129230) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Feb 19 21:11:03.996606 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] Getting metadata type 'volume-metadata' {{(pid=129230) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Feb 19 21:11:04.007343 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] No metadata type 'volume-metadata' available {{(pid=129230) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Feb 19 21:11:04.007999 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] Getting metadata type 'volume-glance-metadata' {{(pid=129230) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Feb 19 21:11:04.018297 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] No metadata type 'volume-glance-metadata' available {{(pid=129230) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Feb 19 21:11:04.020640 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] Backing up metadata for volume 46718fc1-de22-4f32-9233-e05f81bf31e3. {{(pid=129230) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Feb 19 21:11:04.114239 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] Backup 'd8578643-04d6-4c96-9ae5-e6d15d45c94b' of volume 46718fc1-de22-4f32-9233-e05f81bf31e3 finished. {{(pid=129230) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Feb 19 21:11:04.115808 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-46718fc1-de22-4f32-9233-e05f81bf31e3', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '46718fc1-de22-4f32-9233-e05f81bf31e3', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:11:04.134855 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] <== disconnect_volume: return (18ms) None {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:11:04.274114 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-b4b9b5ce-1066-45af-9c5c-1a23c9534869 tempest-VolumesBackupsTest-1229161186 None] Create backup finished. backup: d8578643-04d6-4c96-9ae5-e6d15d45c94b. Feb 19 21:11:04.915498 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-54824182-9016-420c-a94d-a90e58dca23e tempest-VolumesBackupsTest-1229161186 None] Delete backup started, backup: d8578643-04d6-4c96-9ae5-e6d15d45c94b. Feb 19 21:11:04.918464 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-54824182-9016-420c-a94d-a90e58dca23e tempest-VolumesBackupsTest-1229161186 None] Delete started for backup=d8578643-04d6-4c96-9ae5-e6d15d45c94b {{(pid=129230) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Feb 19 21:11:04.918892 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-54824182-9016-420c-a94d-a90e58dca23e tempest-VolumesBackupsTest-1229161186 None] Trying diff format basename='volume-46718fc1-de22-4f32-9233-e05f81bf31e3.backup.d8578643-04d6-4c96-9ae5-e6d15d45c94b' for backup base image of volume 46718fc1-de22-4f32-9233-e05f81bf31e3. {{(pid=129230) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Feb 19 21:11:05.028675 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-54824182-9016-420c-a94d-a90e58dca23e tempest-VolumesBackupsTest-1229161186 None] Looking for snapshot of backup base 'volume-46718fc1-de22-4f32-9233-e05f81bf31e3.backup.d8578643-04d6-4c96-9ae5-e6d15d45c94b' {{(pid=129230) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Feb 19 21:11:05.029906 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-54824182-9016-420c-a94d-a90e58dca23e tempest-VolumesBackupsTest-1229161186 None] Found snapshot 'backup.d8578643-04d6-4c96-9ae5-e6d15d45c94b.snap.1645305062.1922343' {{(pid=129230) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Feb 19 21:11:05.030508 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-54824182-9016-420c-a94d-a90e58dca23e tempest-VolumesBackupsTest-1229161186 None] Deleting backup snapshot='backup.d8578643-04d6-4c96-9ae5-e6d15d45c94b.snap.1645305062.1922343' {{(pid=129230) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Feb 19 21:11:06.025704 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.drivers.ceph [None req-54824182-9016-420c-a94d-a90e58dca23e tempest-VolumesBackupsTest-1229161186 None] Deleting backup base image='volume-46718fc1-de22-4f32-9233-e05f81bf31e3.backup.d8578643-04d6-4c96-9ae5-e6d15d45c94b' of volume 46718fc1-de22-4f32-9233-e05f81bf31e3. Feb 19 21:11:06.478092 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-54824182-9016-420c-a94d-a90e58dca23e tempest-VolumesBackupsTest-1229161186 None] Base backup image='volume-46718fc1-de22-4f32-9233-e05f81bf31e3.backup.d8578643-04d6-4c96-9ae5-e6d15d45c94b' of volume 46718fc1-de22-4f32-9233-e05f81bf31e3 deleted. {{(pid=129230) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Feb 19 21:11:06.592391 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-54824182-9016-420c-a94d-a90e58dca23e tempest-VolumesBackupsTest-1229161186 None] Delete of backup 'd8578643-04d6-4c96-9ae5-e6d15d45c94b' for volume '46718fc1-de22-4f32-9233-e05f81bf31e3' finished. {{(pid=129230) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Feb 19 21:11:06.600133 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_db.sqlalchemy.engines [None req-54824182-9016-420c-a94d-a90e58dca23e tempest-VolumesBackupsTest-1229161186 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129230) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Feb 19 21:11:06.655453 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.quota [None req-54824182-9016-420c-a94d-a90e58dca23e tempest-VolumesBackupsTest-1229161186 None] Created reservations ['8976a58d-7312-40c7-a963-9159e84a5adf', 'f95c1f2f-b18a-4175-81ed-cb4adb9b3feb'] {{(pid=129230) reserve /opt/stack/cinder/cinder/quota.py:755}} Feb 19 21:11:06.718988 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-54824182-9016-420c-a94d-a90e58dca23e tempest-VolumesBackupsTest-1229161186 None] Delete backup finished, backup d8578643-04d6-4c96-9ae5-e6d15d45c94b deleted. Feb 19 21:11:19.131439 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] Create backup started, backup: 67ef1e43-c3f0-4ae3-9684-383d535bc6fa volume: da39e370-fdb6-4ef1-a986-ad5655c11ff7. Feb 19 21:11:19.139555 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-02-19T21:11:19Z,data_timestamp=2022-02-19T21:11:19Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsTest-Backup-2104697020',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=67ef1e43-c3f0-4ae3-9684-383d535bc6fa,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='463a97fa588d4fc2bb01401ca0fcb8aa',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='52c7a837477849768d5a90636fddf7b4',volume_id=da39e370-fdb6-4ef1-a986-ad5655c11ff7) Feb 19 21:11:19.189763 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.utils [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:11:19.192878 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:11:19.215035 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.022s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:11:19.215597 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[e04e9975-2138-4658-afd1-8e3147c5192a]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:6e78a5d1a02d\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:11:19.220260 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:11:19.245623 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Feb 19 21:11:19.246176 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[80c45517-8335-42e5-9a40-c7cd45f70dae]: [Errno 2] No such file or directory: 'systool' {{(pid=150093) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Feb 19 21:11:19.246176 ncdv-master-1 cinder-backup[129230]: Traceback (most recent call last): Feb 19 21:11:19.246176 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Feb 19 21:11:19.246176 ncdv-master-1 cinder-backup[129230]: ret = func(*f_args, **f_kwargs) Feb 19 21:11:19.246176 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Feb 19 21:11:19.246176 ncdv-master-1 cinder-backup[129230]: return func(*args, **kwargs) Feb 19 21:11:19.246176 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Feb 19 21:11:19.246176 ncdv-master-1 cinder-backup[129230]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Feb 19 21:11:19.246176 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Feb 19 21:11:19.246176 ncdv-master-1 cinder-backup[129230]: return putils.execute(on_execute=on_execute, Feb 19 21:11:19.246176 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Feb 19 21:11:19.246176 ncdv-master-1 cinder-backup[129230]: obj = subprocess.Popen(cmd, Feb 19 21:11:19.246176 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Feb 19 21:11:19.246176 ncdv-master-1 cinder-backup[129230]: self._execute_child(args, executable, preexec_fn, close_fds, Feb 19 21:11:19.246176 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Feb 19 21:11:19.246176 ncdv-master-1 cinder-backup[129230]: raise child_exception_type(errno_num, err_msg, err_filename) Feb 19 21:11:19.246176 ncdv-master-1 cinder-backup[129230]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Feb 19 21:11:19.247957 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[80c45517-8335-42e5-9a40-c7cd45f70dae]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:11:19.249399 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:11:19.259980 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Feb 19 21:11:19.260633 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[03fa2128-1e6e-4bfc-adb9-f2c35251a686]: [Errno 2] No such file or directory: 'systool' {{(pid=150093) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Feb 19 21:11:19.260633 ncdv-master-1 cinder-backup[129230]: Traceback (most recent call last): Feb 19 21:11:19.260633 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Feb 19 21:11:19.260633 ncdv-master-1 cinder-backup[129230]: ret = func(*f_args, **f_kwargs) Feb 19 21:11:19.260633 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Feb 19 21:11:19.260633 ncdv-master-1 cinder-backup[129230]: return func(*args, **kwargs) Feb 19 21:11:19.260633 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Feb 19 21:11:19.260633 ncdv-master-1 cinder-backup[129230]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Feb 19 21:11:19.260633 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Feb 19 21:11:19.260633 ncdv-master-1 cinder-backup[129230]: return putils.execute(on_execute=on_execute, Feb 19 21:11:19.260633 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Feb 19 21:11:19.260633 ncdv-master-1 cinder-backup[129230]: obj = subprocess.Popen(cmd, Feb 19 21:11:19.260633 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Feb 19 21:11:19.260633 ncdv-master-1 cinder-backup[129230]: self._execute_child(args, executable, preexec_fn, close_fds, Feb 19 21:11:19.260633 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Feb 19 21:11:19.260633 ncdv-master-1 cinder-backup[129230]: raise child_exception_type(errno_num, err_msg, err_filename) Feb 19 21:11:19.260633 ncdv-master-1 cinder-backup[129230]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Feb 19 21:11:19.261916 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[03fa2128-1e6e-4bfc-adb9-f2c35251a686]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:11:19.265301 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:11:19.280748 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.016s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:11:19.281500 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[c64d88d5-f0a4-4201-b0da-caa7be05a484]: (4, ('/dev/sda1\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:11:19.282887 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:11:19.298473 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.014s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:11:19.298748 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[2683acbf-b6ee-4157-9147-9e2abd31b4d3]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:11:19.301311 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:11:19.319027 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.016s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:11:19.319351 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[d705dd7c-d58b-489e-9531-40cc79ee081f]: (4, ('856ceada-2af3-8e47-a879-77ebed99126e\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:11:19.320356 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] Running cmd (subprocess): nvme version {{(pid=129230) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:11:19.349694 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] CMD "nvme version" returned: 0 in 0.028s {{(pid=129230) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:11:19.352494 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129230) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Feb 19 21:11:19.353108 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129230) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Feb 19 21:11:19.354553 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:9cde57c4-ac04-4e2d-ab4a-203cb09943a6 dsc: {{(pid=129230) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Feb 19 21:11:19.355869 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.utils [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] <== get_connector_properties: return (165ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:6e78a5d1a02d', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '856ceada-2af3-8e47-a879-77ebed99126e', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:9cde57c4-ac04-4e2d-ab4a-203cb09943a6', 'found_dsc': ''} {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:11:20.398115 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connector [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] Factory for rbd on None {{(pid=129230) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Feb 19 21:11:20.402532 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-da39e370-fdb6-4ef1-a986-ad5655c11ff7', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'da39e370-fdb6-4ef1-a986-ad5655c11ff7', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:11:20.404468 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.linuxrbd [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] opening connection to ceph cluster (timeout=-1). {{(pid=129230) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Feb 19 21:11:20.573425 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] <== connect_volume: return (169ms) {'path': } {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:11:20.583588 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_db.sqlalchemy.engines [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129230) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Feb 19 21:11:20.610996 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] Starting backup of volume='da39e370-fdb6-4ef1-a986-ad5655c11ff7'. {{(pid=129230) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Feb 19 21:11:20.611992 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] Volume file is RBD: attempting optimized backup {{(pid=129230) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Feb 19 21:11:20.705261 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] Creating base image 'volume-da39e370-fdb6-4ef1-a986-ad5655c11ff7.backup.67ef1e43-c3f0-4ae3-9684-383d535bc6fa' {{(pid=129230) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Feb 19 21:11:20.705936 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129230) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Feb 19 21:11:20.841159 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] Using --from-snap 'None' for incremental backup of volume da39e370-fdb6-4ef1-a986-ad5655c11ff7. {{(pid=129230) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Feb 19 21:11:20.842096 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] Creating backup snapshot='backup.67ef1e43-c3f0-4ae3-9684-383d535bc6fa.snap.1645305080.8415394' {{(pid=129230) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Feb 19 21:11:22.003665 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] Performing differential transfer from 'volume-da39e370-fdb6-4ef1-a986-ad5655c11ff7' to 'volume-da39e370-fdb6-4ef1-a986-ad5655c11ff7.backup.67ef1e43-c3f0-4ae3-9684-383d535bc6fa' {{(pid=129230) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Feb 19 21:11:22.008598 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_yispivdt --pool volumes volumes/volume-da39e370-fdb6-4ef1-a986-ad5655c11ff7@backup.67ef1e43-c3f0-4ae3-9684-383d535bc6fa.snap.1645305080.8415394 -' into... {{(pid=129230) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Feb 19 21:11:22.008971 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-da39e370-fdb6-4ef1-a986-ad5655c11ff7.backup.67ef1e43-c3f0-4ae3-9684-383d535bc6fa' {{(pid=129230) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Feb 19 21:11:31.457047 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] Differential backup transfer completed in 9.4523s {{(pid=129230) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Feb 19 21:11:31.457783 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] Getting metadata type 'volume-base-metadata' {{(pid=129230) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Feb 19 21:11:31.471275 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129230) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Feb 19 21:11:31.471855 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] Getting metadata type 'volume-metadata' {{(pid=129230) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Feb 19 21:11:31.479923 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] No metadata type 'volume-metadata' available {{(pid=129230) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Feb 19 21:11:31.480572 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] Getting metadata type 'volume-glance-metadata' {{(pid=129230) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Feb 19 21:11:31.490900 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=129230) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}} Feb 19 21:11:31.491799 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] Backing up metadata for volume da39e370-fdb6-4ef1-a986-ad5655c11ff7. {{(pid=129230) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Feb 19 21:11:31.545047 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] Backup '67ef1e43-c3f0-4ae3-9684-383d535bc6fa' of volume da39e370-fdb6-4ef1-a986-ad5655c11ff7 finished. {{(pid=129230) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Feb 19 21:11:31.547035 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-da39e370-fdb6-4ef1-a986-ad5655c11ff7', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'da39e370-fdb6-4ef1-a986-ad5655c11ff7', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:11:31.595770 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] <== disconnect_volume: return (46ms) None {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:11:31.720679 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-945b2e74-fec9-4be3-8415-529ca9147cc3 tempest-VolumesBackupsTest-1229161186 None] Create backup finished. backup: 67ef1e43-c3f0-4ae3-9684-383d535bc6fa. Feb 19 21:11:34.281328 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] Restore backup started, backup: 67ef1e43-c3f0-4ae3-9684-383d535bc6fa volume: f4f29f8b-297f-4715-abaa-8ec0acbf7a27. Feb 19 21:11:34.305516 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.utils [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:11:34.307438 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:11:34.346320 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.039s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:11:34.347597 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[14eb4f29-950a-4eeb-8dc5-5c23c98b804d]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:6e78a5d1a02d\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:11:34.351787 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:11:34.365450 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Feb 19 21:11:34.366029 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[32b8cd39-df5b-48a8-ba42-d16ebd15298e]: [Errno 2] No such file or directory: 'systool' {{(pid=150093) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Feb 19 21:11:34.366029 ncdv-master-1 cinder-backup[129230]: Traceback (most recent call last): Feb 19 21:11:34.366029 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Feb 19 21:11:34.366029 ncdv-master-1 cinder-backup[129230]: ret = func(*f_args, **f_kwargs) Feb 19 21:11:34.366029 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Feb 19 21:11:34.366029 ncdv-master-1 cinder-backup[129230]: return func(*args, **kwargs) Feb 19 21:11:34.366029 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Feb 19 21:11:34.366029 ncdv-master-1 cinder-backup[129230]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Feb 19 21:11:34.366029 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Feb 19 21:11:34.366029 ncdv-master-1 cinder-backup[129230]: return putils.execute(on_execute=on_execute, Feb 19 21:11:34.366029 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Feb 19 21:11:34.366029 ncdv-master-1 cinder-backup[129230]: obj = subprocess.Popen(cmd, Feb 19 21:11:34.366029 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Feb 19 21:11:34.366029 ncdv-master-1 cinder-backup[129230]: self._execute_child(args, executable, preexec_fn, close_fds, Feb 19 21:11:34.366029 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Feb 19 21:11:34.366029 ncdv-master-1 cinder-backup[129230]: raise child_exception_type(errno_num, err_msg, err_filename) Feb 19 21:11:34.366029 ncdv-master-1 cinder-backup[129230]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Feb 19 21:11:34.367975 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[32b8cd39-df5b-48a8-ba42-d16ebd15298e]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:11:34.368403 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:11:34.380242 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Feb 19 21:11:34.380834 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[50d9240a-89bb-4ddf-8163-cdf5d5588314]: [Errno 2] No such file or directory: 'systool' {{(pid=150093) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Feb 19 21:11:34.380834 ncdv-master-1 cinder-backup[129230]: Traceback (most recent call last): Feb 19 21:11:34.380834 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Feb 19 21:11:34.380834 ncdv-master-1 cinder-backup[129230]: ret = func(*f_args, **f_kwargs) Feb 19 21:11:34.380834 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Feb 19 21:11:34.380834 ncdv-master-1 cinder-backup[129230]: return func(*args, **kwargs) Feb 19 21:11:34.380834 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Feb 19 21:11:34.380834 ncdv-master-1 cinder-backup[129230]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Feb 19 21:11:34.380834 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Feb 19 21:11:34.380834 ncdv-master-1 cinder-backup[129230]: return putils.execute(on_execute=on_execute, Feb 19 21:11:34.380834 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Feb 19 21:11:34.380834 ncdv-master-1 cinder-backup[129230]: obj = subprocess.Popen(cmd, Feb 19 21:11:34.380834 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Feb 19 21:11:34.380834 ncdv-master-1 cinder-backup[129230]: self._execute_child(args, executable, preexec_fn, close_fds, Feb 19 21:11:34.380834 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Feb 19 21:11:34.380834 ncdv-master-1 cinder-backup[129230]: raise child_exception_type(errno_num, err_msg, err_filename) Feb 19 21:11:34.380834 ncdv-master-1 cinder-backup[129230]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Feb 19 21:11:34.382575 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[50d9240a-89bb-4ddf-8163-cdf5d5588314]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:11:34.387341 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:11:34.403015 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.017s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:11:34.403015 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[67dee800-62d7-42f8-9e43-c74d67eff294]: (4, ('/dev/sda1\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:11:34.412188 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:11:34.461782 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.050s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:11:34.462454 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[512e8114-f004-45ba-9b84-6e3d715e20c5]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:11:34.464169 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:11:34.499608 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.035s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:11:34.500590 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[83d88b3b-384b-4f8b-afd1-09941c03a95c]: (4, ('856ceada-2af3-8e47-a879-77ebed99126e\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:11:34.502051 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] Running cmd (subprocess): nvme version {{(pid=129230) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:11:34.541888 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] CMD "nvme version" returned: 0 in 0.040s {{(pid=129230) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:11:34.544987 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129230) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Feb 19 21:11:34.545833 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129230) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Feb 19 21:11:34.546586 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:9cde57c4-ac04-4e2d-ab4a-203cb09943a6 dsc: {{(pid=129230) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Feb 19 21:11:34.547364 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.utils [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] <== get_connector_properties: return (241ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:6e78a5d1a02d', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '856ceada-2af3-8e47-a879-77ebed99126e', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:9cde57c4-ac04-4e2d-ab4a-203cb09943a6', 'found_dsc': ''} {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:11:35.616195 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connector [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] Factory for rbd on None {{(pid=129230) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Feb 19 21:11:35.617674 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-f4f29f8b-297f-4715-abaa-8ec0acbf7a27', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'f4f29f8b-297f-4715-abaa-8ec0acbf7a27', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:11:35.618736 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.linuxrbd [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] opening connection to ceph cluster (timeout=-1). {{(pid=129230) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Feb 19 21:11:35.691740 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] <== connect_volume: return (73ms) {'path': } {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:11:35.699834 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_db.sqlalchemy.engines [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129230) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Feb 19 21:11:35.712804 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] Starting restore from Ceph backup=67ef1e43-c3f0-4ae3-9684-383d535bc6fa to volume=volume-f4f29f8b-297f-4715-abaa-8ec0acbf7a27 {{(pid=129230) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1274}} Feb 19 21:11:35.831849 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] Looking for snapshot of backup base 'volume-da39e370-fdb6-4ef1-a986-ad5655c11ff7.backup.67ef1e43-c3f0-4ae3-9684-383d535bc6fa' {{(pid=129230) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Feb 19 21:11:35.832235 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] Found snapshot 'backup.67ef1e43-c3f0-4ae3-9684-383d535bc6fa.snap.1645305080.8415394' {{(pid=129230) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Feb 19 21:11:35.844640 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] Volume file is RBD. {{(pid=129230) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1184}} Feb 19 21:11:35.924802 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] Attempting differential restore. {{(pid=129230) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1234}} Feb 19 21:11:35.927129 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] Attempting incremental restore from base='volume-da39e370-fdb6-4ef1-a986-ad5655c11ff7.backup.67ef1e43-c3f0-4ae3-9684-383d535bc6fa' snap='backup.67ef1e43-c3f0-4ae3-9684-383d535bc6fa.snap.1645305080.8415394' {{(pid=129230) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1089}} Feb 19 21:11:35.927129 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] Performing differential transfer from 'volume-da39e370-fdb6-4ef1-a986-ad5655c11ff7.backup.67ef1e43-c3f0-4ae3-9684-383d535bc6fa' to 'volume-f4f29f8b-297f-4715-abaa-8ec0acbf7a27' {{(pid=129230) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Feb 19 21:11:35.927129 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-da39e370-fdb6-4ef1-a986-ad5655c11ff7.backup.67ef1e43-c3f0-4ae3-9684-383d535bc6fa@backup.67ef1e43-c3f0-4ae3-9684-383d535bc6fa.snap.1645305080.8415394 -' into... {{(pid=129230) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Feb 19 21:11:35.927129 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_hk0etasa --pool volumes - volumes/volume-f4f29f8b-297f-4715-abaa-8ec0acbf7a27' {{(pid=129230) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Feb 19 21:11:42.044462 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] Restore transfer completed in 6.1172s {{(pid=129230) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1111}} Feb 19 21:11:42.047762 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=129230) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1285}} Feb 19 21:11:42.102637 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] Restoring volume base metadata {{(pid=129230) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} Feb 19 21:11:42.163392 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] No metadata of type 'volume-metadata' to restore {{(pid=129230) put /opt/stack/cinder/cinder/backup/driver.py:345}} Feb 19 21:11:42.164097 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] Restoring volume glance metadata {{(pid=129230) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}} Feb 19 21:11:42.411743 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] Restore to volume f4f29f8b-297f-4715-abaa-8ec0acbf7a27 finished successfully. {{(pid=129230) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1292}} Feb 19 21:11:42.414393 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-f4f29f8b-297f-4715-abaa-8ec0acbf7a27', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'f4f29f8b-297f-4715-abaa-8ec0acbf7a27', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:11:42.435767 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] <== disconnect_volume: return (20ms) None {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:11:42.497695 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.manager [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] Encryption key ID for volume f4f29f8b-297f-4715-abaa-8ec0acbf7a27 already matches encryption key ID in backup 67ef1e43-c3f0-4ae3-9684-383d535bc6fa. {{(pid=129230) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} Feb 19 21:11:42.550546 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-65900320-2e89-4bbd-9551-e1f64383f2ad tempest-VolumesBackupsTest-1229161186 None] Finished restoring backup 67ef1e43-c3f0-4ae3-9684-383d535bc6fa to volume f4f29f8b-297f-4715-abaa-8ec0acbf7a27. Feb 19 21:11:45.403551 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-69b90858-cec0-4c96-9f6f-b10c29fd0e8d tempest-VolumesBackupsTest-1229161186 None] Delete backup started, backup: 67ef1e43-c3f0-4ae3-9684-383d535bc6fa. Feb 19 21:11:45.406162 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-69b90858-cec0-4c96-9f6f-b10c29fd0e8d tempest-VolumesBackupsTest-1229161186 None] Delete started for backup=67ef1e43-c3f0-4ae3-9684-383d535bc6fa {{(pid=129230) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Feb 19 21:11:45.406657 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-69b90858-cec0-4c96-9f6f-b10c29fd0e8d tempest-VolumesBackupsTest-1229161186 None] Trying diff format basename='volume-da39e370-fdb6-4ef1-a986-ad5655c11ff7.backup.67ef1e43-c3f0-4ae3-9684-383d535bc6fa' for backup base image of volume da39e370-fdb6-4ef1-a986-ad5655c11ff7. {{(pid=129230) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Feb 19 21:11:45.494472 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-69b90858-cec0-4c96-9f6f-b10c29fd0e8d tempest-VolumesBackupsTest-1229161186 None] Looking for snapshot of backup base 'volume-da39e370-fdb6-4ef1-a986-ad5655c11ff7.backup.67ef1e43-c3f0-4ae3-9684-383d535bc6fa' {{(pid=129230) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Feb 19 21:11:45.495241 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-69b90858-cec0-4c96-9f6f-b10c29fd0e8d tempest-VolumesBackupsTest-1229161186 None] Found snapshot 'backup.67ef1e43-c3f0-4ae3-9684-383d535bc6fa.snap.1645305080.8415394' {{(pid=129230) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Feb 19 21:11:45.495930 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-69b90858-cec0-4c96-9f6f-b10c29fd0e8d tempest-VolumesBackupsTest-1229161186 None] Deleting backup snapshot='backup.67ef1e43-c3f0-4ae3-9684-383d535bc6fa.snap.1645305080.8415394' {{(pid=129230) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Feb 19 21:11:46.083868 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.drivers.ceph [None req-69b90858-cec0-4c96-9f6f-b10c29fd0e8d tempest-VolumesBackupsTest-1229161186 None] Deleting backup base image='volume-da39e370-fdb6-4ef1-a986-ad5655c11ff7.backup.67ef1e43-c3f0-4ae3-9684-383d535bc6fa' of volume da39e370-fdb6-4ef1-a986-ad5655c11ff7. Feb 19 21:11:46.947265 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-69b90858-cec0-4c96-9f6f-b10c29fd0e8d tempest-VolumesBackupsTest-1229161186 None] Base backup image='volume-da39e370-fdb6-4ef1-a986-ad5655c11ff7.backup.67ef1e43-c3f0-4ae3-9684-383d535bc6fa' of volume da39e370-fdb6-4ef1-a986-ad5655c11ff7 deleted. {{(pid=129230) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Feb 19 21:11:47.008640 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-69b90858-cec0-4c96-9f6f-b10c29fd0e8d tempest-VolumesBackupsTest-1229161186 None] Delete of backup '67ef1e43-c3f0-4ae3-9684-383d535bc6fa' for volume 'da39e370-fdb6-4ef1-a986-ad5655c11ff7' finished. {{(pid=129230) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Feb 19 21:11:47.014969 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_db.sqlalchemy.engines [None req-69b90858-cec0-4c96-9f6f-b10c29fd0e8d tempest-VolumesBackupsTest-1229161186 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129230) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Feb 19 21:11:47.046627 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.quota [None req-69b90858-cec0-4c96-9f6f-b10c29fd0e8d tempest-VolumesBackupsTest-1229161186 None] Created reservations ['4f6e1abf-788f-477e-b113-e9995d17b4f5', '880788b0-65d4-4f31-a293-f2b769106430'] {{(pid=129230) reserve /opt/stack/cinder/cinder/quota.py:755}} Feb 19 21:11:47.078127 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-69b90858-cec0-4c96-9f6f-b10c29fd0e8d tempest-VolumesBackupsTest-1229161186 None] Delete backup finished, backup 67ef1e43-c3f0-4ae3-9684-383d535bc6fa deleted. Feb 19 21:12:03.398693 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:12:03.399388 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:13:03.395142 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:13:03.395764 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:14:03.392761 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:14:03.394161 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:15:03.397782 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:15:03.398506 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:16:03.395613 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:16:03.396749 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:16:40.944729 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] Create backup started, backup: 3f130c1c-0507-4751-9f60-5b0e437ac9ba volume: 7b79b82a-e264-4ba9-85ff-025343a3ea6d. Feb 19 21:16:40.962677 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-02-19T21:16:40Z,data_timestamp=2022-02-19T21:16:40Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsAdminTest-Backup-1271261638',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=3f130c1c-0507-4751-9f60-5b0e437ac9ba,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='485f24939ca34525a63e45e52f4d8cfa',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='73abd5e75b5e4f549e59660ee17226c4',volume_id=7b79b82a-e264-4ba9-85ff-025343a3ea6d) Feb 19 21:16:41.020006 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.utils [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:16:41.022002 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:16:41.046361 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.024s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:16:41.047050 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[383bf384-9309-4778-8bb8-6fef8dd5b9d6]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:6e78a5d1a02d\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:16:41.051089 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:16:41.058013 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Feb 19 21:16:41.058543 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[a43dbf53-941d-4775-b665-a36a5a203b0c]: [Errno 2] No such file or directory: 'systool' {{(pid=150093) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Feb 19 21:16:41.058543 ncdv-master-1 cinder-backup[129230]: Traceback (most recent call last): Feb 19 21:16:41.058543 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Feb 19 21:16:41.058543 ncdv-master-1 cinder-backup[129230]: ret = func(*f_args, **f_kwargs) Feb 19 21:16:41.058543 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Feb 19 21:16:41.058543 ncdv-master-1 cinder-backup[129230]: return func(*args, **kwargs) Feb 19 21:16:41.058543 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Feb 19 21:16:41.058543 ncdv-master-1 cinder-backup[129230]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Feb 19 21:16:41.058543 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Feb 19 21:16:41.058543 ncdv-master-1 cinder-backup[129230]: return putils.execute(on_execute=on_execute, Feb 19 21:16:41.058543 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Feb 19 21:16:41.058543 ncdv-master-1 cinder-backup[129230]: obj = subprocess.Popen(cmd, Feb 19 21:16:41.058543 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Feb 19 21:16:41.058543 ncdv-master-1 cinder-backup[129230]: self._execute_child(args, executable, preexec_fn, close_fds, Feb 19 21:16:41.058543 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Feb 19 21:16:41.058543 ncdv-master-1 cinder-backup[129230]: raise child_exception_type(errno_num, err_msg, err_filename) Feb 19 21:16:41.058543 ncdv-master-1 cinder-backup[129230]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Feb 19 21:16:41.059874 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[a43dbf53-941d-4775-b665-a36a5a203b0c]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:16:41.061381 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:16:41.075498 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Feb 19 21:16:41.076639 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[020a7d02-2de0-4049-8f33-e4171dde797a]: [Errno 2] No such file or directory: 'systool' {{(pid=150093) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Feb 19 21:16:41.076639 ncdv-master-1 cinder-backup[129230]: Traceback (most recent call last): Feb 19 21:16:41.076639 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Feb 19 21:16:41.076639 ncdv-master-1 cinder-backup[129230]: ret = func(*f_args, **f_kwargs) Feb 19 21:16:41.076639 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Feb 19 21:16:41.076639 ncdv-master-1 cinder-backup[129230]: return func(*args, **kwargs) Feb 19 21:16:41.076639 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Feb 19 21:16:41.076639 ncdv-master-1 cinder-backup[129230]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Feb 19 21:16:41.076639 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Feb 19 21:16:41.076639 ncdv-master-1 cinder-backup[129230]: return putils.execute(on_execute=on_execute, Feb 19 21:16:41.076639 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Feb 19 21:16:41.076639 ncdv-master-1 cinder-backup[129230]: obj = subprocess.Popen(cmd, Feb 19 21:16:41.076639 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Feb 19 21:16:41.076639 ncdv-master-1 cinder-backup[129230]: self._execute_child(args, executable, preexec_fn, close_fds, Feb 19 21:16:41.076639 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Feb 19 21:16:41.076639 ncdv-master-1 cinder-backup[129230]: raise child_exception_type(errno_num, err_msg, err_filename) Feb 19 21:16:41.076639 ncdv-master-1 cinder-backup[129230]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Feb 19 21:16:41.078424 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[020a7d02-2de0-4049-8f33-e4171dde797a]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:16:41.080032 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:16:41.093435 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.013s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:16:41.094230 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[4b3f52ba-486f-4270-80eb-be1791372cfe]: (4, ('/dev/sda1\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:16:41.095466 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:16:41.115642 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.019s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:16:41.115975 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[4f813b26-7d78-4dfa-be88-0365c066c68d]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:16:41.117785 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:16:41.155875 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.036s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:16:41.157534 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[aa064fd5-02a6-4ce5-9fa9-44a3eaf67bf9]: (4, ('856ceada-2af3-8e47-a879-77ebed99126e\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:16:41.159035 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] Running cmd (subprocess): nvme version {{(pid=129230) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:16:41.205156 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] CMD "nvme version" returned: 0 in 0.046s {{(pid=129230) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:16:41.207963 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129230) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Feb 19 21:16:41.208882 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129230) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Feb 19 21:16:41.209443 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:9cde57c4-ac04-4e2d-ab4a-203cb09943a6 dsc: {{(pid=129230) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Feb 19 21:16:41.211895 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.utils [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] <== get_connector_properties: return (191ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:6e78a5d1a02d', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '856ceada-2af3-8e47-a879-77ebed99126e', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:9cde57c4-ac04-4e2d-ab4a-203cb09943a6', 'found_dsc': ''} {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:16:42.524604 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connector [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] Factory for rbd on None {{(pid=129230) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Feb 19 21:16:42.526366 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-7b79b82a-e264-4ba9-85ff-025343a3ea6d', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '7b79b82a-e264-4ba9-85ff-025343a3ea6d', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:16:42.527415 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.linuxrbd [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] opening connection to ceph cluster (timeout=-1). {{(pid=129230) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Feb 19 21:16:42.643180 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] <== connect_volume: return (116ms) {'path': } {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:16:42.658313 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_db.sqlalchemy.engines [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129230) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Feb 19 21:16:42.718425 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] Starting backup of volume='7b79b82a-e264-4ba9-85ff-025343a3ea6d'. {{(pid=129230) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Feb 19 21:16:42.720190 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] Volume file is RBD: attempting optimized backup {{(pid=129230) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Feb 19 21:16:42.777089 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] Creating base image 'volume-7b79b82a-e264-4ba9-85ff-025343a3ea6d.backup.3f130c1c-0507-4751-9f60-5b0e437ac9ba' {{(pid=129230) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Feb 19 21:16:42.777596 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129230) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Feb 19 21:16:42.863964 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] Using --from-snap 'None' for incremental backup of volume 7b79b82a-e264-4ba9-85ff-025343a3ea6d. {{(pid=129230) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Feb 19 21:16:42.864610 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] Creating backup snapshot='backup.3f130c1c-0507-4751-9f60-5b0e437ac9ba.snap.1645305402.8641887' {{(pid=129230) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Feb 19 21:16:43.701752 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] Performing differential transfer from 'volume-7b79b82a-e264-4ba9-85ff-025343a3ea6d' to 'volume-7b79b82a-e264-4ba9-85ff-025343a3ea6d.backup.3f130c1c-0507-4751-9f60-5b0e437ac9ba' {{(pid=129230) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Feb 19 21:16:43.703763 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_xzvd92v0 --pool volumes volumes/volume-7b79b82a-e264-4ba9-85ff-025343a3ea6d@backup.3f130c1c-0507-4751-9f60-5b0e437ac9ba.snap.1645305402.8641887 -' into... {{(pid=129230) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Feb 19 21:16:43.704431 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-7b79b82a-e264-4ba9-85ff-025343a3ea6d.backup.3f130c1c-0507-4751-9f60-5b0e437ac9ba' {{(pid=129230) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Feb 19 21:16:44.758426 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] Differential backup transfer completed in 1.0561s {{(pid=129230) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Feb 19 21:16:44.759295 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] Getting metadata type 'volume-base-metadata' {{(pid=129230) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Feb 19 21:16:44.783730 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129230) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Feb 19 21:16:44.785002 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] Getting metadata type 'volume-metadata' {{(pid=129230) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Feb 19 21:16:44.804773 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] No metadata type 'volume-metadata' available {{(pid=129230) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Feb 19 21:16:44.805402 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] Getting metadata type 'volume-glance-metadata' {{(pid=129230) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Feb 19 21:16:44.814472 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] No metadata type 'volume-glance-metadata' available {{(pid=129230) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Feb 19 21:16:44.815293 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] Backing up metadata for volume 7b79b82a-e264-4ba9-85ff-025343a3ea6d. {{(pid=129230) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Feb 19 21:16:44.899060 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] Backup '3f130c1c-0507-4751-9f60-5b0e437ac9ba' of volume 7b79b82a-e264-4ba9-85ff-025343a3ea6d finished. {{(pid=129230) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Feb 19 21:16:44.909659 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-7b79b82a-e264-4ba9-85ff-025343a3ea6d', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '7b79b82a-e264-4ba9-85ff-025343a3ea6d', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:16:44.990282 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] <== disconnect_volume: return (78ms) None {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:16:45.108886 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-176f1edc-3ecb-4ec6-bc60-fe848c4dd00b tempest-VolumesBackupsAdminTest-588090949 None] Create backup finished. backup: 3f130c1c-0507-4751-9f60-5b0e437ac9ba. Feb 19 21:16:45.941067 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-a3c6ff10-c526-491d-b598-36ea1cfd058c tempest-VolumesBackupsAdminTest-1987963770 None] Export record started, backup: 3f130c1c-0507-4751-9f60-5b0e437ac9ba. Feb 19 21:16:45.943246 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-a3c6ff10-c526-491d-b598-36ea1cfd058c tempest-VolumesBackupsAdminTest-1987963770 None] Export record finished, backup 3f130c1c-0507-4751-9f60-5b0e437ac9ba exported. Feb 19 21:16:46.197323 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-2a57cfcd-4dba-4191-8f3e-fc68041ac0ef tempest-VolumesBackupsAdminTest-1987963770 None] Import record started, backup_url: eyJkcml2ZXJfaW5mbyI6IHt9LCAiaWQiOiAiM2YzOGJkYjgtMmVkNi00M2I1LWI4ZjUtMTBhODZkYThhNDkzIiwgInVzZXJfaWQiOiAiNzNhYmQ1ZTc1YjVlNGY1NDllNTk2NjBlZTE3MjI2YzQiLCAicHJvamVjdF9pZCI6ICI0ODVmMjQ5MzljYTM0NTI1YTYzZTQ1ZTUyZjRkOGNmYSIsICJ2b2x1bWVfaWQiOiAiN2I3OWI4MmEtZTI2NC00YmE5LTg1ZmYtMDI1MzQzYTNlYTZkIiwgImhvc3QiOiAibmNkdi1tYXN0ZXItMSIsICJhdmFpbGFiaWxpdHlfem9uZSI6IG51bGwsICJjb250YWluZXIiOiAiYmFja3VwcyIsICJwYXJlbnRfaWQiOiBudWxsLCAic3RhdHVzIjogImF2YWlsYWJsZSIsICJmYWlsX3JlYXNvbiI6IG51bGwsICJzaXplIjogMSwgImRpc3BsYXlfbmFtZSI6ICJ0ZW1wZXN0LVZvbHVtZXNCYWNrdXBzQWRtaW5UZXN0LUJhY2t1cC0xMjcxMjYxNjM4IiwgImRpc3BsYXlfZGVzY3JpcHRpb24iOiBudWxsLCAic2VydmljZV9tZXRhZGF0YSI6ICJ7XCJiYXNlXCI6IFwidm9sdW1lLTdiNzliODJhLWUyNjQtNGJhOS04NWZmLTAyNTM0M2EzZWE2ZC5iYWNrdXAuM2YxMzBjMWMtMDUwNy00NzUxLTlmNjAtNWIwZTQzN2FjOWJhXCJ9IiwgInNlcnZpY2UiOiAiY2luZGVyLmJhY2t1cC5kcml2ZXJzLmNlcGguQ2VwaEJhY2t1cERyaXZlciIsICJvYmplY3RfY291bnQiOiAwLCAidGVtcF92b2x1bWVfaWQiOiBudWxsLCAidGVtcF9zbmFwc2hvdF9pZCI6IG51bGwsICJudW1fZGVwZW5kZW50X2JhY2t1cHMiOiAwLCAic25hcHNob3RfaWQiOiBudWxsLCAiZGF0YV90aW1lc3RhbXAiOiAiMjAyMi0wMi0xOVQyMToxNjo0MVoiLCAicmVzdG9yZV92b2x1bWVfaWQiOiBudWxsLCAibWV0YWRhdGEiOiB7fSwgImVuY3J5cHRpb25fa2V5X2lkIjogbnVsbCwgImNyZWF0ZWRfYXQiOiAiMjAyMi0wMi0xOVQyMToxNjo0MVoiLCAidXBkYXRlZF9hdCI6ICIyMDIyLTAyLTE5VDIxOjE2OjQ1WiIsICJkZWxldGVkX2F0IjogbnVsbCwgImRlbGV0ZWQiOiBmYWxzZX0=. Feb 19 21:16:46.206348 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_db.sqlalchemy.engines [None req-2a57cfcd-4dba-4191-8f3e-fc68041ac0ef tempest-VolumesBackupsAdminTest-1987963770 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129230) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Feb 19 21:16:46.378388 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-2a57cfcd-4dba-4191-8f3e-fc68041ac0ef tempest-VolumesBackupsAdminTest-1987963770 None] Import record id 3f38bdb8-2ed6-43b5-b8f5-10a86da8a493 metadata from driver finished. Feb 19 21:16:49.236597 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] Restore backup started, backup: 3f130c1c-0507-4751-9f60-5b0e437ac9ba volume: 4d42190d-741e-4833-8730-d3d6028bd7d3. Feb 19 21:16:49.258247 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.utils [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:16:49.259858 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:16:49.320750 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.055s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:16:49.320750 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[342903fe-9395-429a-97ae-eb622567d42e]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:6e78a5d1a02d\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:16:49.320750 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:16:49.332286 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Feb 19 21:16:49.333285 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[d170f820-c324-4ca7-96d1-4da0413603c4]: [Errno 2] No such file or directory: 'systool' {{(pid=150093) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Feb 19 21:16:49.333285 ncdv-master-1 cinder-backup[129230]: Traceback (most recent call last): Feb 19 21:16:49.333285 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Feb 19 21:16:49.333285 ncdv-master-1 cinder-backup[129230]: ret = func(*f_args, **f_kwargs) Feb 19 21:16:49.333285 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Feb 19 21:16:49.333285 ncdv-master-1 cinder-backup[129230]: return func(*args, **kwargs) Feb 19 21:16:49.333285 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Feb 19 21:16:49.333285 ncdv-master-1 cinder-backup[129230]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Feb 19 21:16:49.333285 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Feb 19 21:16:49.333285 ncdv-master-1 cinder-backup[129230]: return putils.execute(on_execute=on_execute, Feb 19 21:16:49.333285 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Feb 19 21:16:49.333285 ncdv-master-1 cinder-backup[129230]: obj = subprocess.Popen(cmd, Feb 19 21:16:49.333285 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Feb 19 21:16:49.333285 ncdv-master-1 cinder-backup[129230]: self._execute_child(args, executable, preexec_fn, close_fds, Feb 19 21:16:49.333285 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Feb 19 21:16:49.333285 ncdv-master-1 cinder-backup[129230]: raise child_exception_type(errno_num, err_msg, err_filename) Feb 19 21:16:49.333285 ncdv-master-1 cinder-backup[129230]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Feb 19 21:16:49.335235 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[d170f820-c324-4ca7-96d1-4da0413603c4]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:16:49.340151 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:16:49.354291 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Feb 19 21:16:49.355281 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[9470d9da-346a-4b33-be41-c503b9e8ab25]: [Errno 2] No such file or directory: 'systool' {{(pid=150093) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Feb 19 21:16:49.355281 ncdv-master-1 cinder-backup[129230]: Traceback (most recent call last): Feb 19 21:16:49.355281 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Feb 19 21:16:49.355281 ncdv-master-1 cinder-backup[129230]: ret = func(*f_args, **f_kwargs) Feb 19 21:16:49.355281 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Feb 19 21:16:49.355281 ncdv-master-1 cinder-backup[129230]: return func(*args, **kwargs) Feb 19 21:16:49.355281 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Feb 19 21:16:49.355281 ncdv-master-1 cinder-backup[129230]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Feb 19 21:16:49.355281 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Feb 19 21:16:49.355281 ncdv-master-1 cinder-backup[129230]: return putils.execute(on_execute=on_execute, Feb 19 21:16:49.355281 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Feb 19 21:16:49.355281 ncdv-master-1 cinder-backup[129230]: obj = subprocess.Popen(cmd, Feb 19 21:16:49.355281 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Feb 19 21:16:49.355281 ncdv-master-1 cinder-backup[129230]: self._execute_child(args, executable, preexec_fn, close_fds, Feb 19 21:16:49.355281 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Feb 19 21:16:49.355281 ncdv-master-1 cinder-backup[129230]: raise child_exception_type(errno_num, err_msg, err_filename) Feb 19 21:16:49.355281 ncdv-master-1 cinder-backup[129230]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Feb 19 21:16:49.356610 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[9470d9da-346a-4b33-be41-c503b9e8ab25]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:16:49.359921 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:16:49.387922 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.029s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:16:49.388495 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[7ea6cddf-2544-4701-8922-95f730aa0ac9]: (4, ('/dev/sda1\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:16:49.391060 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:16:49.425243 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.034s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:16:49.425718 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[441d63e0-bbf2-45eb-b584-97283dd45959]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:16:49.429204 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:16:49.458308 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.028s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:16:49.458726 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[bedabc3d-4da9-4bdd-9933-e0144d661d30]: (4, ('856ceada-2af3-8e47-a879-77ebed99126e\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:16:49.459900 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] Running cmd (subprocess): nvme version {{(pid=129230) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:16:49.504907 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] CMD "nvme version" returned: 0 in 0.044s {{(pid=129230) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:16:49.509342 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129230) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Feb 19 21:16:49.510105 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129230) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Feb 19 21:16:49.511000 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:9cde57c4-ac04-4e2d-ab4a-203cb09943a6 dsc: {{(pid=129230) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Feb 19 21:16:49.512009 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.utils [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] <== get_connector_properties: return (252ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:6e78a5d1a02d', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '856ceada-2af3-8e47-a879-77ebed99126e', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:9cde57c4-ac04-4e2d-ab4a-203cb09943a6', 'found_dsc': ''} {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:16:50.651668 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connector [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] Factory for rbd on None {{(pid=129230) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Feb 19 21:16:50.656900 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-4d42190d-741e-4833-8730-d3d6028bd7d3', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '4d42190d-741e-4833-8730-d3d6028bd7d3', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:16:50.659575 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.linuxrbd [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] opening connection to ceph cluster (timeout=-1). {{(pid=129230) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Feb 19 21:16:50.761449 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] <== connect_volume: return (103ms) {'path': } {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:16:50.771110 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_db.sqlalchemy.engines [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129230) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Feb 19 21:16:50.786495 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] Starting restore from Ceph backup=3f130c1c-0507-4751-9f60-5b0e437ac9ba to volume=volume-4d42190d-741e-4833-8730-d3d6028bd7d3 {{(pid=129230) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1274}} Feb 19 21:16:50.927606 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] Looking for snapshot of backup base 'volume-7b79b82a-e264-4ba9-85ff-025343a3ea6d.backup.3f130c1c-0507-4751-9f60-5b0e437ac9ba' {{(pid=129230) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Feb 19 21:16:50.928080 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] Found snapshot 'backup.3f130c1c-0507-4751-9f60-5b0e437ac9ba.snap.1645305402.8641887' {{(pid=129230) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Feb 19 21:16:50.933906 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] Volume file is RBD. {{(pid=129230) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1184}} Feb 19 21:16:50.994680 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] Attempting differential restore. {{(pid=129230) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1234}} Feb 19 21:16:50.995316 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] Attempting incremental restore from base='volume-7b79b82a-e264-4ba9-85ff-025343a3ea6d.backup.3f130c1c-0507-4751-9f60-5b0e437ac9ba' snap='backup.3f130c1c-0507-4751-9f60-5b0e437ac9ba.snap.1645305402.8641887' {{(pid=129230) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1089}} Feb 19 21:16:50.996021 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] Performing differential transfer from 'volume-7b79b82a-e264-4ba9-85ff-025343a3ea6d.backup.3f130c1c-0507-4751-9f60-5b0e437ac9ba' to 'volume-4d42190d-741e-4833-8730-d3d6028bd7d3' {{(pid=129230) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Feb 19 21:16:50.996674 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-7b79b82a-e264-4ba9-85ff-025343a3ea6d.backup.3f130c1c-0507-4751-9f60-5b0e437ac9ba@backup.3f130c1c-0507-4751-9f60-5b0e437ac9ba.snap.1645305402.8641887 -' into... {{(pid=129230) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Feb 19 21:16:50.998386 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_eskihua0 --pool volumes - volumes/volume-4d42190d-741e-4833-8730-d3d6028bd7d3' {{(pid=129230) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Feb 19 21:16:51.838938 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] Restore transfer completed in 0.8421s {{(pid=129230) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1111}} Feb 19 21:16:51.841276 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=129230) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1285}} Feb 19 21:16:51.924760 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] Restoring volume base metadata {{(pid=129230) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} Feb 19 21:16:51.957782 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] No metadata of type 'volume-metadata' to restore {{(pid=129230) put /opt/stack/cinder/cinder/backup/driver.py:345}} Feb 19 21:16:51.958443 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] No metadata of type 'volume-glance-metadata' to restore {{(pid=129230) put /opt/stack/cinder/cinder/backup/driver.py:345}} Feb 19 21:16:51.965055 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] Restore to volume 4d42190d-741e-4833-8730-d3d6028bd7d3 finished successfully. {{(pid=129230) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1292}} Feb 19 21:16:51.970073 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-4d42190d-741e-4833-8730-d3d6028bd7d3', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '4d42190d-741e-4833-8730-d3d6028bd7d3', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:16:52.004737 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] <== disconnect_volume: return (33ms) None {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:16:52.158353 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.manager [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] Encryption key ID for volume 4d42190d-741e-4833-8730-d3d6028bd7d3 already matches encryption key ID in backup 3f130c1c-0507-4751-9f60-5b0e437ac9ba. {{(pid=129230) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} Feb 19 21:16:52.234648 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-ed168719-b98a-4db1-8c3d-ea0b13f7ae98 tempest-VolumesBackupsAdminTest-588090949 None] Finished restoring backup 3f130c1c-0507-4751-9f60-5b0e437ac9ba to volume 4d42190d-741e-4833-8730-d3d6028bd7d3. Feb 19 21:16:52.958987 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-6819c82a-35b6-42be-8540-c09a65d6fc57 tempest-VolumesBackupsAdminTest-1987963770 None] Delete backup started, backup: 3f38bdb8-2ed6-43b5-b8f5-10a86da8a493. Feb 19 21:16:52.963773 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-6819c82a-35b6-42be-8540-c09a65d6fc57 tempest-VolumesBackupsAdminTest-1987963770 None] Delete started for backup=3f38bdb8-2ed6-43b5-b8f5-10a86da8a493 {{(pid=129230) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Feb 19 21:16:52.964908 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-6819c82a-35b6-42be-8540-c09a65d6fc57 tempest-VolumesBackupsAdminTest-1987963770 None] Trying diff format basename='volume-7b79b82a-e264-4ba9-85ff-025343a3ea6d.backup.3f130c1c-0507-4751-9f60-5b0e437ac9ba' for backup base image of volume 7b79b82a-e264-4ba9-85ff-025343a3ea6d. {{(pid=129230) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Feb 19 21:16:53.074699 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-6819c82a-35b6-42be-8540-c09a65d6fc57 tempest-VolumesBackupsAdminTest-1987963770 None] Looking for snapshot of backup base 'volume-7b79b82a-e264-4ba9-85ff-025343a3ea6d.backup.3f130c1c-0507-4751-9f60-5b0e437ac9ba' {{(pid=129230) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Feb 19 21:16:53.075337 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-6819c82a-35b6-42be-8540-c09a65d6fc57 tempest-VolumesBackupsAdminTest-1987963770 None] Backup '3f38bdb8-2ed6-43b5-b8f5-10a86da8a493' has no snapshot {{(pid=129230) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:915}} Feb 19 21:16:53.076836 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-6819c82a-35b6-42be-8540-c09a65d6fc57 tempest-VolumesBackupsAdminTest-1987963770 None] No backup snapshot to delete {{(pid=129230) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:476}} Feb 19 21:16:53.092403 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.drivers.ceph [None req-6819c82a-35b6-42be-8540-c09a65d6fc57 tempest-VolumesBackupsAdminTest-1987963770 None] Backup base image of volume 7b79b82a-e264-4ba9-85ff-025343a3ea6d still has 1 snapshots so skipping base image delete. Feb 19 21:16:53.182117 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-6819c82a-35b6-42be-8540-c09a65d6fc57 tempest-VolumesBackupsAdminTest-1987963770 None] Metadata backup object 'backup.3f38bdb8-2ed6-43b5-b8f5-10a86da8a493.meta' not found - ignoring {{(pid=129230) remove_if_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:160}} Feb 19 21:16:53.189393 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-6819c82a-35b6-42be-8540-c09a65d6fc57 tempest-VolumesBackupsAdminTest-1987963770 None] Delete of backup '3f38bdb8-2ed6-43b5-b8f5-10a86da8a493' for volume '7b79b82a-e264-4ba9-85ff-025343a3ea6d' finished. {{(pid=129230) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Feb 19 21:16:53.196657 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_db.sqlalchemy.engines [None req-6819c82a-35b6-42be-8540-c09a65d6fc57 tempest-VolumesBackupsAdminTest-1987963770 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129230) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Feb 19 21:16:53.245231 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.quota [None req-6819c82a-35b6-42be-8540-c09a65d6fc57 tempest-VolumesBackupsAdminTest-1987963770 None] Created reservations ['f24aaa36-2404-4ffd-99e7-9ab3a2b3bd3d', '53f6a732-9b8b-4aec-ad54-804b6ff8fc2a'] {{(pid=129230) reserve /opt/stack/cinder/cinder/quota.py:755}} Feb 19 21:16:53.282371 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-6819c82a-35b6-42be-8540-c09a65d6fc57 tempest-VolumesBackupsAdminTest-1987963770 None] Delete backup finished, backup 3f38bdb8-2ed6-43b5-b8f5-10a86da8a493 deleted. Feb 19 21:16:54.121524 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-d28fc5b0-4eec-4cf4-b093-9d05aacc95c4 tempest-VolumesBackupsAdminTest-588090949 None] Delete backup started, backup: 3f130c1c-0507-4751-9f60-5b0e437ac9ba. Feb 19 21:16:54.124508 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d28fc5b0-4eec-4cf4-b093-9d05aacc95c4 tempest-VolumesBackupsAdminTest-588090949 None] Delete started for backup=3f130c1c-0507-4751-9f60-5b0e437ac9ba {{(pid=129230) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Feb 19 21:16:54.124946 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d28fc5b0-4eec-4cf4-b093-9d05aacc95c4 tempest-VolumesBackupsAdminTest-588090949 None] Trying diff format basename='volume-7b79b82a-e264-4ba9-85ff-025343a3ea6d.backup.3f130c1c-0507-4751-9f60-5b0e437ac9ba' for backup base image of volume 7b79b82a-e264-4ba9-85ff-025343a3ea6d. {{(pid=129230) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Feb 19 21:16:54.303551 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d28fc5b0-4eec-4cf4-b093-9d05aacc95c4 tempest-VolumesBackupsAdminTest-588090949 None] Looking for snapshot of backup base 'volume-7b79b82a-e264-4ba9-85ff-025343a3ea6d.backup.3f130c1c-0507-4751-9f60-5b0e437ac9ba' {{(pid=129230) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Feb 19 21:16:54.304562 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d28fc5b0-4eec-4cf4-b093-9d05aacc95c4 tempest-VolumesBackupsAdminTest-588090949 None] Found snapshot 'backup.3f130c1c-0507-4751-9f60-5b0e437ac9ba.snap.1645305402.8641887' {{(pid=129230) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Feb 19 21:16:54.305645 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d28fc5b0-4eec-4cf4-b093-9d05aacc95c4 tempest-VolumesBackupsAdminTest-588090949 None] Deleting backup snapshot='backup.3f130c1c-0507-4751-9f60-5b0e437ac9ba.snap.1645305402.8641887' {{(pid=129230) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Feb 19 21:16:54.779042 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.drivers.ceph [None req-d28fc5b0-4eec-4cf4-b093-9d05aacc95c4 tempest-VolumesBackupsAdminTest-588090949 None] Deleting backup base image='volume-7b79b82a-e264-4ba9-85ff-025343a3ea6d.backup.3f130c1c-0507-4751-9f60-5b0e437ac9ba' of volume 7b79b82a-e264-4ba9-85ff-025343a3ea6d. Feb 19 21:16:55.116797 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d28fc5b0-4eec-4cf4-b093-9d05aacc95c4 tempest-VolumesBackupsAdminTest-588090949 None] Base backup image='volume-7b79b82a-e264-4ba9-85ff-025343a3ea6d.backup.3f130c1c-0507-4751-9f60-5b0e437ac9ba' of volume 7b79b82a-e264-4ba9-85ff-025343a3ea6d deleted. {{(pid=129230) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Feb 19 21:16:55.192720 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d28fc5b0-4eec-4cf4-b093-9d05aacc95c4 tempest-VolumesBackupsAdminTest-588090949 None] Delete of backup '3f130c1c-0507-4751-9f60-5b0e437ac9ba' for volume '7b79b82a-e264-4ba9-85ff-025343a3ea6d' finished. {{(pid=129230) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Feb 19 21:16:55.201803 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_db.sqlalchemy.engines [None req-d28fc5b0-4eec-4cf4-b093-9d05aacc95c4 tempest-VolumesBackupsAdminTest-588090949 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129230) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Feb 19 21:16:55.255989 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.quota [None req-d28fc5b0-4eec-4cf4-b093-9d05aacc95c4 tempest-VolumesBackupsAdminTest-588090949 None] Created reservations ['5773d36d-3fad-47d3-b766-1d57948e2da0', 'fedee761-af7c-4525-959b-0eec25be8d2e'] {{(pid=129230) reserve /opt/stack/cinder/cinder/quota.py:755}} Feb 19 21:16:55.317130 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-d28fc5b0-4eec-4cf4-b093-9d05aacc95c4 tempest-VolumesBackupsAdminTest-588090949 None] Delete backup finished, backup 3f130c1c-0507-4751-9f60-5b0e437ac9ba deleted. Feb 19 21:16:58.285680 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] Create backup started, backup: ce63ae77-dd24-45b6-a29a-042128e43237 volume: ff8d82f1-155e-46d0-80f9-d03d972d6568. Feb 19 21:16:58.303414 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-02-19T21:16:58Z,data_timestamp=2022-02-19T21:16:58Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsAdminTest-Backup-977686304',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=ce63ae77-dd24-45b6-a29a-042128e43237,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='485f24939ca34525a63e45e52f4d8cfa',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='73abd5e75b5e4f549e59660ee17226c4',volume_id=ff8d82f1-155e-46d0-80f9-d03d972d6568) Feb 19 21:16:58.363897 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.utils [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:16:58.370825 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:16:58.405087 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.035s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:16:58.405512 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[94202333-7151-45e1-b3a5-94898b01a7bf]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:6e78a5d1a02d\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:16:58.407195 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:16:58.441363 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Feb 19 21:16:58.442447 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[7184a3d4-806b-4f78-94a8-08d1d0dd334e]: [Errno 2] No such file or directory: 'systool' {{(pid=150093) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Feb 19 21:16:58.442447 ncdv-master-1 cinder-backup[129230]: Traceback (most recent call last): Feb 19 21:16:58.442447 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Feb 19 21:16:58.442447 ncdv-master-1 cinder-backup[129230]: ret = func(*f_args, **f_kwargs) Feb 19 21:16:58.442447 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Feb 19 21:16:58.442447 ncdv-master-1 cinder-backup[129230]: return func(*args, **kwargs) Feb 19 21:16:58.442447 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Feb 19 21:16:58.442447 ncdv-master-1 cinder-backup[129230]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Feb 19 21:16:58.442447 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Feb 19 21:16:58.442447 ncdv-master-1 cinder-backup[129230]: return putils.execute(on_execute=on_execute, Feb 19 21:16:58.442447 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Feb 19 21:16:58.442447 ncdv-master-1 cinder-backup[129230]: obj = subprocess.Popen(cmd, Feb 19 21:16:58.442447 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Feb 19 21:16:58.442447 ncdv-master-1 cinder-backup[129230]: self._execute_child(args, executable, preexec_fn, close_fds, Feb 19 21:16:58.442447 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Feb 19 21:16:58.442447 ncdv-master-1 cinder-backup[129230]: raise child_exception_type(errno_num, err_msg, err_filename) Feb 19 21:16:58.442447 ncdv-master-1 cinder-backup[129230]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Feb 19 21:16:58.443759 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[7184a3d4-806b-4f78-94a8-08d1d0dd334e]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:16:58.445439 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:16:58.467936 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Feb 19 21:16:58.468653 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[10ee08f4-f584-47fa-bd0a-18920d58bcc3]: [Errno 2] No such file or directory: 'systool' {{(pid=150093) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Feb 19 21:16:58.468653 ncdv-master-1 cinder-backup[129230]: Traceback (most recent call last): Feb 19 21:16:58.468653 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Feb 19 21:16:58.468653 ncdv-master-1 cinder-backup[129230]: ret = func(*f_args, **f_kwargs) Feb 19 21:16:58.468653 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Feb 19 21:16:58.468653 ncdv-master-1 cinder-backup[129230]: return func(*args, **kwargs) Feb 19 21:16:58.468653 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Feb 19 21:16:58.468653 ncdv-master-1 cinder-backup[129230]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Feb 19 21:16:58.468653 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Feb 19 21:16:58.468653 ncdv-master-1 cinder-backup[129230]: return putils.execute(on_execute=on_execute, Feb 19 21:16:58.468653 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Feb 19 21:16:58.468653 ncdv-master-1 cinder-backup[129230]: obj = subprocess.Popen(cmd, Feb 19 21:16:58.468653 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Feb 19 21:16:58.468653 ncdv-master-1 cinder-backup[129230]: self._execute_child(args, executable, preexec_fn, close_fds, Feb 19 21:16:58.468653 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Feb 19 21:16:58.468653 ncdv-master-1 cinder-backup[129230]: raise child_exception_type(errno_num, err_msg, err_filename) Feb 19 21:16:58.468653 ncdv-master-1 cinder-backup[129230]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Feb 19 21:16:58.470322 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[10ee08f4-f584-47fa-bd0a-18920d58bcc3]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:16:58.471995 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:16:58.507798 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.035s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:16:58.509067 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[2dbeb88c-744b-42d2-afa6-1901c99d22a4]: (4, ('/dev/sda1\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:16:58.510840 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:16:58.534736 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.023s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:16:58.535318 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[c9ee3197-0930-4676-b81f-8d34aeb57c6f]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:16:58.539878 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:16:58.582307 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.044s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:16:58.583958 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[e53ef21e-628a-43b2-95cd-adb73772da7c]: (4, ('856ceada-2af3-8e47-a879-77ebed99126e\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:16:58.585346 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] Running cmd (subprocess): nvme version {{(pid=129230) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:16:58.642214 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] CMD "nvme version" returned: 0 in 0.056s {{(pid=129230) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:16:58.645245 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129230) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Feb 19 21:16:58.645760 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129230) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Feb 19 21:16:58.646144 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:9cde57c4-ac04-4e2d-ab4a-203cb09943a6 dsc: {{(pid=129230) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Feb 19 21:16:58.647217 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.utils [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] <== get_connector_properties: return (278ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:6e78a5d1a02d', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '856ceada-2af3-8e47-a879-77ebed99126e', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:9cde57c4-ac04-4e2d-ab4a-203cb09943a6', 'found_dsc': ''} {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:16:59.963638 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connector [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] Factory for rbd on None {{(pid=129230) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Feb 19 21:16:59.965260 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-ff8d82f1-155e-46d0-80f9-d03d972d6568', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'ff8d82f1-155e-46d0-80f9-d03d972d6568', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:16:59.966911 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.linuxrbd [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] opening connection to ceph cluster (timeout=-1). {{(pid=129230) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Feb 19 21:17:00.089033 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] <== connect_volume: return (122ms) {'path': } {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:17:00.098730 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_db.sqlalchemy.engines [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129230) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Feb 19 21:17:00.149737 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] Starting backup of volume='ff8d82f1-155e-46d0-80f9-d03d972d6568'. {{(pid=129230) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Feb 19 21:17:00.150998 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] Volume file is RBD: attempting optimized backup {{(pid=129230) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Feb 19 21:17:00.218948 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] Creating base image 'volume-ff8d82f1-155e-46d0-80f9-d03d972d6568.backup.ce63ae77-dd24-45b6-a29a-042128e43237' {{(pid=129230) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Feb 19 21:17:00.219638 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129230) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Feb 19 21:17:00.292869 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] Using --from-snap 'None' for incremental backup of volume ff8d82f1-155e-46d0-80f9-d03d972d6568. {{(pid=129230) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Feb 19 21:17:00.293318 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] Creating backup snapshot='backup.ce63ae77-dd24-45b6-a29a-042128e43237.snap.1645305420.2928944' {{(pid=129230) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Feb 19 21:17:00.607147 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] Performing differential transfer from 'volume-ff8d82f1-155e-46d0-80f9-d03d972d6568' to 'volume-ff8d82f1-155e-46d0-80f9-d03d972d6568.backup.ce63ae77-dd24-45b6-a29a-042128e43237' {{(pid=129230) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Feb 19 21:17:00.607730 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_xga52790 --pool volumes volumes/volume-ff8d82f1-155e-46d0-80f9-d03d972d6568@backup.ce63ae77-dd24-45b6-a29a-042128e43237.snap.1645305420.2928944 -' into... {{(pid=129230) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Feb 19 21:17:00.608221 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-ff8d82f1-155e-46d0-80f9-d03d972d6568.backup.ce63ae77-dd24-45b6-a29a-042128e43237' {{(pid=129230) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Feb 19 21:17:01.613429 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] Differential backup transfer completed in 1.0059s {{(pid=129230) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Feb 19 21:17:01.614117 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] Getting metadata type 'volume-base-metadata' {{(pid=129230) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Feb 19 21:17:01.633219 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129230) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Feb 19 21:17:01.634006 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] Getting metadata type 'volume-metadata' {{(pid=129230) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Feb 19 21:17:01.643111 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] No metadata type 'volume-metadata' available {{(pid=129230) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Feb 19 21:17:01.643713 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] Getting metadata type 'volume-glance-metadata' {{(pid=129230) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Feb 19 21:17:01.651803 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] No metadata type 'volume-glance-metadata' available {{(pid=129230) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Feb 19 21:17:01.652519 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] Backing up metadata for volume ff8d82f1-155e-46d0-80f9-d03d972d6568. {{(pid=129230) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Feb 19 21:17:01.736441 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] Backup 'ce63ae77-dd24-45b6-a29a-042128e43237' of volume ff8d82f1-155e-46d0-80f9-d03d972d6568 finished. {{(pid=129230) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Feb 19 21:17:01.739109 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-ff8d82f1-155e-46d0-80f9-d03d972d6568', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'ff8d82f1-155e-46d0-80f9-d03d972d6568', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:17:01.780943 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] <== disconnect_volume: return (40ms) None {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:17:01.905809 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-d1cda4db-4c5f-4c16-aef0-f7affabcefe2 tempest-VolumesBackupsAdminTest-588090949 None] Create backup finished. backup: ce63ae77-dd24-45b6-a29a-042128e43237. Feb 19 21:17:02.635298 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-f5e4bc99-320a-44f7-88b6-d7392ac4beff tempest-VolumesBackupsAdminTest-1987963770 None] Reset backup status started, backup_id: ce63ae77-dd24-45b6-a29a-042128e43237, status: error. Feb 19 21:17:02.636303 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-f5e4bc99-320a-44f7-88b6-d7392ac4beff tempest-VolumesBackupsAdminTest-1987963770 None] Backup service: cinder.backup.drivers.ceph.CephBackupDriver. Feb 19 21:17:02.774896 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-991c934c-5deb-4b9c-b32a-000483820823 tempest-VolumesBackupsAdminTest-588090949 None] Delete backup started, backup: ce63ae77-dd24-45b6-a29a-042128e43237. Feb 19 21:17:02.774896 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-991c934c-5deb-4b9c-b32a-000483820823 tempest-VolumesBackupsAdminTest-588090949 None] Delete started for backup=ce63ae77-dd24-45b6-a29a-042128e43237 {{(pid=129230) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Feb 19 21:17:02.775473 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-991c934c-5deb-4b9c-b32a-000483820823 tempest-VolumesBackupsAdminTest-588090949 None] Trying diff format basename='volume-ff8d82f1-155e-46d0-80f9-d03d972d6568.backup.ce63ae77-dd24-45b6-a29a-042128e43237' for backup base image of volume ff8d82f1-155e-46d0-80f9-d03d972d6568. {{(pid=129230) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Feb 19 21:17:02.880815 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-991c934c-5deb-4b9c-b32a-000483820823 tempest-VolumesBackupsAdminTest-588090949 None] Looking for snapshot of backup base 'volume-ff8d82f1-155e-46d0-80f9-d03d972d6568.backup.ce63ae77-dd24-45b6-a29a-042128e43237' {{(pid=129230) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Feb 19 21:17:02.882207 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-991c934c-5deb-4b9c-b32a-000483820823 tempest-VolumesBackupsAdminTest-588090949 None] Found snapshot 'backup.ce63ae77-dd24-45b6-a29a-042128e43237.snap.1645305420.2928944' {{(pid=129230) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Feb 19 21:17:02.883486 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-991c934c-5deb-4b9c-b32a-000483820823 tempest-VolumesBackupsAdminTest-588090949 None] Deleting backup snapshot='backup.ce63ae77-dd24-45b6-a29a-042128e43237.snap.1645305420.2928944' {{(pid=129230) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Feb 19 21:17:03.395763 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:17:03.396111 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:17:03.677469 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_db.sqlalchemy.engines [None req-0310f1a8-baa1-43c0-8dfd-d981a2de117f None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129230) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Feb 19 21:17:03.764583 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.drivers.ceph [None req-991c934c-5deb-4b9c-b32a-000483820823 tempest-VolumesBackupsAdminTest-588090949 None] Deleting backup base image='volume-ff8d82f1-155e-46d0-80f9-d03d972d6568.backup.ce63ae77-dd24-45b6-a29a-042128e43237' of volume ff8d82f1-155e-46d0-80f9-d03d972d6568. Feb 19 21:17:04.286322 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-991c934c-5deb-4b9c-b32a-000483820823 tempest-VolumesBackupsAdminTest-588090949 None] Base backup image='volume-ff8d82f1-155e-46d0-80f9-d03d972d6568.backup.ce63ae77-dd24-45b6-a29a-042128e43237' of volume ff8d82f1-155e-46d0-80f9-d03d972d6568 deleted. {{(pid=129230) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Feb 19 21:17:04.389393 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-991c934c-5deb-4b9c-b32a-000483820823 tempest-VolumesBackupsAdminTest-588090949 None] Delete of backup 'ce63ae77-dd24-45b6-a29a-042128e43237' for volume 'ff8d82f1-155e-46d0-80f9-d03d972d6568' finished. {{(pid=129230) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Feb 19 21:17:04.466007 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.quota [None req-991c934c-5deb-4b9c-b32a-000483820823 tempest-VolumesBackupsAdminTest-588090949 None] Created reservations ['75af7efc-2dde-4976-9381-918089aea7f0', '25c48aa1-1983-4188-9438-f863f1374ae9'] {{(pid=129230) reserve /opt/stack/cinder/cinder/quota.py:755}} Feb 19 21:17:04.550321 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-991c934c-5deb-4b9c-b32a-000483820823 tempest-VolumesBackupsAdminTest-588090949 None] Delete backup finished, backup ce63ae77-dd24-45b6-a29a-042128e43237 deleted. Feb 19 21:18:03.400464 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:18:03.400464 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:19:03.390598 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:19:03.391352 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:19:51.282354 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] Create backup started, backup: 7548e5b3-fb1e-4200-8c85-0ab573d5c1c3 volume: 0311b7ea-7813-49d0-9380-40c61569e790. Feb 19 21:19:51.293319 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-02-19T21:19:51Z,data_timestamp=2022-02-19T21:19:51Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsV39Test-Backup-1254871827',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=7548e5b3-fb1e-4200-8c85-0ab573d5c1c3,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='4735cc20b18f46a1b0468386b0b7a632',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='27dbcebd833d49dbbeca6c661518336f',volume_id=0311b7ea-7813-49d0-9380-40c61569e790) Feb 19 21:19:51.350085 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.utils [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:19:51.352155 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:19:51.381800 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.029s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:19:51.382665 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[8df9d315-6b6a-4b2b-aecd-9ab897dc2014]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:6e78a5d1a02d\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:19:51.386812 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:19:51.409855 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Feb 19 21:19:51.410202 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[e71de202-cac6-46b0-8680-f32e2e80f3b2]: [Errno 2] No such file or directory: 'systool' {{(pid=150093) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Feb 19 21:19:51.410202 ncdv-master-1 cinder-backup[129230]: Traceback (most recent call last): Feb 19 21:19:51.410202 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Feb 19 21:19:51.410202 ncdv-master-1 cinder-backup[129230]: ret = func(*f_args, **f_kwargs) Feb 19 21:19:51.410202 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Feb 19 21:19:51.410202 ncdv-master-1 cinder-backup[129230]: return func(*args, **kwargs) Feb 19 21:19:51.410202 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Feb 19 21:19:51.410202 ncdv-master-1 cinder-backup[129230]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Feb 19 21:19:51.410202 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Feb 19 21:19:51.410202 ncdv-master-1 cinder-backup[129230]: return putils.execute(on_execute=on_execute, Feb 19 21:19:51.410202 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Feb 19 21:19:51.410202 ncdv-master-1 cinder-backup[129230]: obj = subprocess.Popen(cmd, Feb 19 21:19:51.410202 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Feb 19 21:19:51.410202 ncdv-master-1 cinder-backup[129230]: self._execute_child(args, executable, preexec_fn, close_fds, Feb 19 21:19:51.410202 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Feb 19 21:19:51.410202 ncdv-master-1 cinder-backup[129230]: raise child_exception_type(errno_num, err_msg, err_filename) Feb 19 21:19:51.410202 ncdv-master-1 cinder-backup[129230]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Feb 19 21:19:51.412509 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[e71de202-cac6-46b0-8680-f32e2e80f3b2]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:19:51.422589 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:19:51.440557 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Feb 19 21:19:51.440870 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[b748f894-91ea-41e8-9263-4cc6b6a7573e]: [Errno 2] No such file or directory: 'systool' {{(pid=150093) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Feb 19 21:19:51.440870 ncdv-master-1 cinder-backup[129230]: Traceback (most recent call last): Feb 19 21:19:51.440870 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Feb 19 21:19:51.440870 ncdv-master-1 cinder-backup[129230]: ret = func(*f_args, **f_kwargs) Feb 19 21:19:51.440870 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Feb 19 21:19:51.440870 ncdv-master-1 cinder-backup[129230]: return func(*args, **kwargs) Feb 19 21:19:51.440870 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Feb 19 21:19:51.440870 ncdv-master-1 cinder-backup[129230]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Feb 19 21:19:51.440870 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Feb 19 21:19:51.440870 ncdv-master-1 cinder-backup[129230]: return putils.execute(on_execute=on_execute, Feb 19 21:19:51.440870 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Feb 19 21:19:51.440870 ncdv-master-1 cinder-backup[129230]: obj = subprocess.Popen(cmd, Feb 19 21:19:51.440870 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Feb 19 21:19:51.440870 ncdv-master-1 cinder-backup[129230]: self._execute_child(args, executable, preexec_fn, close_fds, Feb 19 21:19:51.440870 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Feb 19 21:19:51.440870 ncdv-master-1 cinder-backup[129230]: raise child_exception_type(errno_num, err_msg, err_filename) Feb 19 21:19:51.440870 ncdv-master-1 cinder-backup[129230]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Feb 19 21:19:51.442127 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[b748f894-91ea-41e8-9263-4cc6b6a7573e]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:19:51.446135 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:19:51.486776 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.043s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:19:51.487362 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[c2c24a70-03cb-4a2c-8f38-4fb02a9488ac]: (4, ('/dev/sda1\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:19:51.492949 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:19:51.532780 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.042s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:19:51.534370 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[7cabd579-760d-493b-8625-5f4c0d7df4ac]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:19:51.537872 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:19:51.576769 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.037s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:19:51.578037 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[c8581eef-40fb-4826-8de6-72c1c5ccb389]: (4, ('856ceada-2af3-8e47-a879-77ebed99126e\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:19:51.580642 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] Running cmd (subprocess): nvme version {{(pid=129230) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:19:51.629733 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] CMD "nvme version" returned: 0 in 0.048s {{(pid=129230) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:19:51.631911 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129230) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Feb 19 21:19:51.633073 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129230) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Feb 19 21:19:51.643437 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:9cde57c4-ac04-4e2d-ab4a-203cb09943a6 dsc: {{(pid=129230) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Feb 19 21:19:51.645391 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.utils [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] <== get_connector_properties: return (293ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:6e78a5d1a02d', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '856ceada-2af3-8e47-a879-77ebed99126e', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:9cde57c4-ac04-4e2d-ab4a-203cb09943a6', 'found_dsc': ''} {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:19:52.919106 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connector [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] Factory for rbd on None {{(pid=129230) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Feb 19 21:19:52.921352 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-0311b7ea-7813-49d0-9380-40c61569e790', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '0311b7ea-7813-49d0-9380-40c61569e790', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:19:52.922574 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.linuxrbd [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] opening connection to ceph cluster (timeout=-1). {{(pid=129230) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Feb 19 21:19:53.006374 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] <== connect_volume: return (84ms) {'path': } {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:19:53.014608 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_db.sqlalchemy.engines [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129230) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Feb 19 21:19:53.038417 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] Starting backup of volume='0311b7ea-7813-49d0-9380-40c61569e790'. {{(pid=129230) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Feb 19 21:19:53.039695 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] Volume file is RBD: attempting optimized backup {{(pid=129230) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Feb 19 21:19:53.076883 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] Creating base image 'volume-0311b7ea-7813-49d0-9380-40c61569e790.backup.7548e5b3-fb1e-4200-8c85-0ab573d5c1c3' {{(pid=129230) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Feb 19 21:19:53.077749 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129230) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Feb 19 21:19:53.176828 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] Using --from-snap 'None' for incremental backup of volume 0311b7ea-7813-49d0-9380-40c61569e790. {{(pid=129230) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Feb 19 21:19:53.177422 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] Creating backup snapshot='backup.7548e5b3-fb1e-4200-8c85-0ab573d5c1c3.snap.1645305593.177021' {{(pid=129230) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Feb 19 21:19:53.989321 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] Performing differential transfer from 'volume-0311b7ea-7813-49d0-9380-40c61569e790' to 'volume-0311b7ea-7813-49d0-9380-40c61569e790.backup.7548e5b3-fb1e-4200-8c85-0ab573d5c1c3' {{(pid=129230) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Feb 19 21:19:53.989961 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_ktt3ms_8 --pool volumes volumes/volume-0311b7ea-7813-49d0-9380-40c61569e790@backup.7548e5b3-fb1e-4200-8c85-0ab573d5c1c3.snap.1645305593.177021 -' into... {{(pid=129230) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Feb 19 21:19:53.990114 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-0311b7ea-7813-49d0-9380-40c61569e790.backup.7548e5b3-fb1e-4200-8c85-0ab573d5c1c3' {{(pid=129230) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Feb 19 21:19:55.098819 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] Differential backup transfer completed in 1.1070s {{(pid=129230) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Feb 19 21:19:55.099739 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] Getting metadata type 'volume-base-metadata' {{(pid=129230) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Feb 19 21:19:55.118713 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129230) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Feb 19 21:19:55.119424 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] Getting metadata type 'volume-metadata' {{(pid=129230) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Feb 19 21:19:55.128850 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] No metadata type 'volume-metadata' available {{(pid=129230) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Feb 19 21:19:55.129579 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] Getting metadata type 'volume-glance-metadata' {{(pid=129230) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Feb 19 21:19:55.138532 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] No metadata type 'volume-glance-metadata' available {{(pid=129230) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Feb 19 21:19:55.152161 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] Backing up metadata for volume 0311b7ea-7813-49d0-9380-40c61569e790. {{(pid=129230) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Feb 19 21:19:55.226017 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] Backup '7548e5b3-fb1e-4200-8c85-0ab573d5c1c3' of volume 0311b7ea-7813-49d0-9380-40c61569e790 finished. {{(pid=129230) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Feb 19 21:19:55.228296 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-0311b7ea-7813-49d0-9380-40c61569e790', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '0311b7ea-7813-49d0-9380-40c61569e790', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:19:55.273044 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] <== disconnect_volume: return (43ms) None {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:19:56.537957 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-61c65bee-ea8b-4391-a4ef-b1caf66a9ef4 tempest-VolumesBackupsV39Test-918988790 None] Create backup finished. backup: 7548e5b3-fb1e-4200-8c85-0ab573d5c1c3. Feb 19 21:19:58.394873 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-4cb7bc0b-5374-4003-9a27-7ecb84c78c43 tempest-VolumesBackupsV39Test-918988790 None] Delete backup started, backup: 7548e5b3-fb1e-4200-8c85-0ab573d5c1c3. Feb 19 21:19:58.397495 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-4cb7bc0b-5374-4003-9a27-7ecb84c78c43 tempest-VolumesBackupsV39Test-918988790 None] Delete started for backup=7548e5b3-fb1e-4200-8c85-0ab573d5c1c3 {{(pid=129230) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Feb 19 21:19:58.397969 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-4cb7bc0b-5374-4003-9a27-7ecb84c78c43 tempest-VolumesBackupsV39Test-918988790 None] Trying diff format basename='volume-0311b7ea-7813-49d0-9380-40c61569e790.backup.7548e5b3-fb1e-4200-8c85-0ab573d5c1c3' for backup base image of volume 0311b7ea-7813-49d0-9380-40c61569e790. {{(pid=129230) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Feb 19 21:19:58.479900 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-4cb7bc0b-5374-4003-9a27-7ecb84c78c43 tempest-VolumesBackupsV39Test-918988790 None] Looking for snapshot of backup base 'volume-0311b7ea-7813-49d0-9380-40c61569e790.backup.7548e5b3-fb1e-4200-8c85-0ab573d5c1c3' {{(pid=129230) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Feb 19 21:19:58.480294 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-4cb7bc0b-5374-4003-9a27-7ecb84c78c43 tempest-VolumesBackupsV39Test-918988790 None] Found snapshot 'backup.7548e5b3-fb1e-4200-8c85-0ab573d5c1c3.snap.1645305593.177021' {{(pid=129230) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Feb 19 21:19:58.480697 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-4cb7bc0b-5374-4003-9a27-7ecb84c78c43 tempest-VolumesBackupsV39Test-918988790 None] Deleting backup snapshot='backup.7548e5b3-fb1e-4200-8c85-0ab573d5c1c3.snap.1645305593.177021' {{(pid=129230) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Feb 19 21:19:59.143221 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.drivers.ceph [None req-4cb7bc0b-5374-4003-9a27-7ecb84c78c43 tempest-VolumesBackupsV39Test-918988790 None] Deleting backup base image='volume-0311b7ea-7813-49d0-9380-40c61569e790.backup.7548e5b3-fb1e-4200-8c85-0ab573d5c1c3' of volume 0311b7ea-7813-49d0-9380-40c61569e790. Feb 19 21:19:59.462884 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-4cb7bc0b-5374-4003-9a27-7ecb84c78c43 tempest-VolumesBackupsV39Test-918988790 None] Base backup image='volume-0311b7ea-7813-49d0-9380-40c61569e790.backup.7548e5b3-fb1e-4200-8c85-0ab573d5c1c3' of volume 0311b7ea-7813-49d0-9380-40c61569e790 deleted. {{(pid=129230) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Feb 19 21:19:59.536773 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-4cb7bc0b-5374-4003-9a27-7ecb84c78c43 tempest-VolumesBackupsV39Test-918988790 None] Delete of backup '7548e5b3-fb1e-4200-8c85-0ab573d5c1c3' for volume '0311b7ea-7813-49d0-9380-40c61569e790' finished. {{(pid=129230) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Feb 19 21:19:59.546558 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_db.sqlalchemy.engines [None req-4cb7bc0b-5374-4003-9a27-7ecb84c78c43 tempest-VolumesBackupsV39Test-918988790 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129230) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Feb 19 21:19:59.585536 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.quota [None req-4cb7bc0b-5374-4003-9a27-7ecb84c78c43 tempest-VolumesBackupsV39Test-918988790 None] Created reservations ['89717acf-9864-47ec-97de-9d7e1f2d19ba', 'f11c6071-d23f-477f-934f-1d57a636b471'] {{(pid=129230) reserve /opt/stack/cinder/cinder/quota.py:755}} Feb 19 21:19:59.622913 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-4cb7bc0b-5374-4003-9a27-7ecb84c78c43 tempest-VolumesBackupsV39Test-918988790 None] Delete backup finished, backup 7548e5b3-fb1e-4200-8c85-0ab573d5c1c3 deleted. Feb 19 21:20:03.396100 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:20:03.397173 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:21:03.397086 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:21:03.397656 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:22:03.395257 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:22:03.395991 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:23:03.392421 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:23:03.393186 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:24:03.389057 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:24:03.389729 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:25:03.394278 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:25:03.394861 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:26:03.399757 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:26:03.400527 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:27:03.405686 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:27:03.406390 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:28:03.411205 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:28:03.411864 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:29:03.408035 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:29:03.408756 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:30:03.412720 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:30:03.413396 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:31:03.417959 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:31:03.418657 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:32:03.423567 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:32:03.424293 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:32:10.226354 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] Create backup started, backup: f0df62d4-2f93-4fe0-acec-5a1cbed6c181 volume: 8ef288d9-4908-4e07-8642-09ca661890e3. Feb 19 21:32:10.234739 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-02-19T21:32:10Z,data_timestamp=2022-02-19T21:32:10Z,deleted=False,deleted_at=None,display_description=None,display_name=None,encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=f0df62d4-2f93-4fe0-acec-5a1cbed6c181,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='72f5105ec09c4f3c82c087633532c76a',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='f489d295f12b43719f3ba84f7d2dc683',volume_id=8ef288d9-4908-4e07-8642-09ca661890e3) Feb 19 21:32:10.274044 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.utils [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:32:10.276693 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:32:10.284654 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.009s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:32:10.284996 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[9c1ec587-0666-4bd6-b57e-b5c6c60dce54]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:6e78a5d1a02d\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:32:10.286717 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:32:10.293283 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Feb 19 21:32:10.293909 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[e843227a-d666-49d9-a88d-a36c90ea46c1]: [Errno 2] No such file or directory: 'systool' {{(pid=150093) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Feb 19 21:32:10.293909 ncdv-master-1 cinder-backup[129230]: Traceback (most recent call last): Feb 19 21:32:10.293909 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Feb 19 21:32:10.293909 ncdv-master-1 cinder-backup[129230]: ret = func(*f_args, **f_kwargs) Feb 19 21:32:10.293909 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Feb 19 21:32:10.293909 ncdv-master-1 cinder-backup[129230]: return func(*args, **kwargs) Feb 19 21:32:10.293909 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Feb 19 21:32:10.293909 ncdv-master-1 cinder-backup[129230]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Feb 19 21:32:10.293909 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Feb 19 21:32:10.293909 ncdv-master-1 cinder-backup[129230]: return putils.execute(on_execute=on_execute, Feb 19 21:32:10.293909 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Feb 19 21:32:10.293909 ncdv-master-1 cinder-backup[129230]: obj = subprocess.Popen(cmd, Feb 19 21:32:10.293909 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Feb 19 21:32:10.293909 ncdv-master-1 cinder-backup[129230]: self._execute_child(args, executable, preexec_fn, close_fds, Feb 19 21:32:10.293909 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Feb 19 21:32:10.293909 ncdv-master-1 cinder-backup[129230]: raise child_exception_type(errno_num, err_msg, err_filename) Feb 19 21:32:10.293909 ncdv-master-1 cinder-backup[129230]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Feb 19 21:32:10.295149 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[e843227a-d666-49d9-a88d-a36c90ea46c1]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:32:10.296264 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:32:10.302885 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Feb 19 21:32:10.303865 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[ba421fcb-861c-4cf8-85c0-93fabc73a06e]: [Errno 2] No such file or directory: 'systool' {{(pid=150093) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Feb 19 21:32:10.303865 ncdv-master-1 cinder-backup[129230]: Traceback (most recent call last): Feb 19 21:32:10.303865 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Feb 19 21:32:10.303865 ncdv-master-1 cinder-backup[129230]: ret = func(*f_args, **f_kwargs) Feb 19 21:32:10.303865 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Feb 19 21:32:10.303865 ncdv-master-1 cinder-backup[129230]: return func(*args, **kwargs) Feb 19 21:32:10.303865 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Feb 19 21:32:10.303865 ncdv-master-1 cinder-backup[129230]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Feb 19 21:32:10.303865 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Feb 19 21:32:10.303865 ncdv-master-1 cinder-backup[129230]: return putils.execute(on_execute=on_execute, Feb 19 21:32:10.303865 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Feb 19 21:32:10.303865 ncdv-master-1 cinder-backup[129230]: obj = subprocess.Popen(cmd, Feb 19 21:32:10.303865 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Feb 19 21:32:10.303865 ncdv-master-1 cinder-backup[129230]: self._execute_child(args, executable, preexec_fn, close_fds, Feb 19 21:32:10.303865 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Feb 19 21:32:10.303865 ncdv-master-1 cinder-backup[129230]: raise child_exception_type(errno_num, err_msg, err_filename) Feb 19 21:32:10.303865 ncdv-master-1 cinder-backup[129230]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Feb 19 21:32:10.305134 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[ba421fcb-861c-4cf8-85c0-93fabc73a06e]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:32:10.305947 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:32:10.314989 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.009s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:32:10.315311 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[b672a16c-e663-4234-a571-81a5c9d5cffb]: (4, ('/dev/sda1\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:32:10.316813 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:32:10.327451 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.010s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:32:10.327998 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[f76eb71e-547f-440a-863e-92fcbec53aff]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:32:10.330477 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:32:10.339377 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.008s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:32:10.339977 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[82d7e478-fd18-468d-bd2f-7857d1c02fc2]: (4, ('856ceada-2af3-8e47-a879-77ebed99126e\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:32:10.340939 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] Running cmd (subprocess): nvme version {{(pid=129230) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:32:10.361020 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] CMD "nvme version" returned: 0 in 0.020s {{(pid=129230) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:32:10.363077 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129230) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Feb 19 21:32:10.363474 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129230) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Feb 19 21:32:10.363843 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:9cde57c4-ac04-4e2d-ab4a-203cb09943a6 dsc: {{(pid=129230) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Feb 19 21:32:10.364522 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.utils [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] <== get_connector_properties: return (89ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:6e78a5d1a02d', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '856ceada-2af3-8e47-a879-77ebed99126e', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:9cde57c4-ac04-4e2d-ab4a-203cb09943a6', 'found_dsc': ''} {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:32:11.051784 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connector [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] Factory for rbd on None {{(pid=129230) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Feb 19 21:32:11.054019 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-8ef288d9-4908-4e07-8642-09ca661890e3', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '8ef288d9-4908-4e07-8642-09ca661890e3', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:32:11.055064 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.linuxrbd [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] opening connection to ceph cluster (timeout=-1). {{(pid=129230) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Feb 19 21:32:11.117762 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] <== connect_volume: return (62ms) {'path': } {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:32:11.125395 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_db.sqlalchemy.engines [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129230) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Feb 19 21:32:11.142086 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] Starting backup of volume='8ef288d9-4908-4e07-8642-09ca661890e3'. {{(pid=129230) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Feb 19 21:32:11.143057 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] Volume file is RBD: attempting optimized backup {{(pid=129230) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Feb 19 21:32:11.164558 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] Creating base image 'volume-8ef288d9-4908-4e07-8642-09ca661890e3.backup.f0df62d4-2f93-4fe0-acec-5a1cbed6c181' {{(pid=129230) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Feb 19 21:32:11.164965 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129230) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Feb 19 21:32:11.195910 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] Using --from-snap 'None' for incremental backup of volume 8ef288d9-4908-4e07-8642-09ca661890e3. {{(pid=129230) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Feb 19 21:32:11.196441 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] Creating backup snapshot='backup.f0df62d4-2f93-4fe0-acec-5a1cbed6c181.snap.1645306331.1959336' {{(pid=129230) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Feb 19 21:32:12.092868 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] Performing differential transfer from 'volume-8ef288d9-4908-4e07-8642-09ca661890e3' to 'volume-8ef288d9-4908-4e07-8642-09ca661890e3.backup.f0df62d4-2f93-4fe0-acec-5a1cbed6c181' {{(pid=129230) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Feb 19 21:32:12.094144 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_94cn_dx5 --pool volumes volumes/volume-8ef288d9-4908-4e07-8642-09ca661890e3@backup.f0df62d4-2f93-4fe0-acec-5a1cbed6c181.snap.1645306331.1959336 -' into... {{(pid=129230) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Feb 19 21:32:12.094391 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-8ef288d9-4908-4e07-8642-09ca661890e3.backup.f0df62d4-2f93-4fe0-acec-5a1cbed6c181' {{(pid=129230) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Feb 19 21:32:16.715656 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] Differential backup transfer completed in 4.6225s {{(pid=129230) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Feb 19 21:32:16.716319 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] Getting metadata type 'volume-base-metadata' {{(pid=129230) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Feb 19 21:32:16.727682 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129230) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Feb 19 21:32:16.728091 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] Getting metadata type 'volume-metadata' {{(pid=129230) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Feb 19 21:32:16.734275 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] No metadata type 'volume-metadata' available {{(pid=129230) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Feb 19 21:32:16.734658 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] Getting metadata type 'volume-glance-metadata' {{(pid=129230) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Feb 19 21:32:16.742078 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=129230) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}} Feb 19 21:32:16.742655 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] Backing up metadata for volume 8ef288d9-4908-4e07-8642-09ca661890e3. {{(pid=129230) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Feb 19 21:32:16.773199 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] Backup 'f0df62d4-2f93-4fe0-acec-5a1cbed6c181' of volume 8ef288d9-4908-4e07-8642-09ca661890e3 finished. {{(pid=129230) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Feb 19 21:32:16.774850 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-8ef288d9-4908-4e07-8642-09ca661890e3', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '8ef288d9-4908-4e07-8642-09ca661890e3', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:32:16.792397 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] <== disconnect_volume: return (17ms) None {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:32:16.851321 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-8cbc2389-d8a3-4b05-ad9b-2f6cfabef208 tempest-TestVolumeBackupRestore-808817250 None] Create backup finished. backup: f0df62d4-2f93-4fe0-acec-5a1cbed6c181. Feb 19 21:32:18.725116 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] Restore backup started, backup: f0df62d4-2f93-4fe0-acec-5a1cbed6c181 volume: 78bcad5a-4a24-4ba3-af4e-0ce1c804279c. Feb 19 21:32:18.747464 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.utils [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.30', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:32:18.749127 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:32:18.764591 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.016s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:32:18.764931 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[7d4778dc-9d9a-4401-bd68-78f3658aac78]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:6e78a5d1a02d\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:32:18.766187 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:32:18.773286 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Feb 19 21:32:18.774538 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[ab6ace38-8d9b-4b09-a396-76c4fe384d97]: [Errno 2] No such file or directory: 'systool' {{(pid=150093) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Feb 19 21:32:18.774538 ncdv-master-1 cinder-backup[129230]: Traceback (most recent call last): Feb 19 21:32:18.774538 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Feb 19 21:32:18.774538 ncdv-master-1 cinder-backup[129230]: ret = func(*f_args, **f_kwargs) Feb 19 21:32:18.774538 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Feb 19 21:32:18.774538 ncdv-master-1 cinder-backup[129230]: return func(*args, **kwargs) Feb 19 21:32:18.774538 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Feb 19 21:32:18.774538 ncdv-master-1 cinder-backup[129230]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Feb 19 21:32:18.774538 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Feb 19 21:32:18.774538 ncdv-master-1 cinder-backup[129230]: return putils.execute(on_execute=on_execute, Feb 19 21:32:18.774538 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Feb 19 21:32:18.774538 ncdv-master-1 cinder-backup[129230]: obj = subprocess.Popen(cmd, Feb 19 21:32:18.774538 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Feb 19 21:32:18.774538 ncdv-master-1 cinder-backup[129230]: self._execute_child(args, executable, preexec_fn, close_fds, Feb 19 21:32:18.774538 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Feb 19 21:32:18.774538 ncdv-master-1 cinder-backup[129230]: raise child_exception_type(errno_num, err_msg, err_filename) Feb 19 21:32:18.774538 ncdv-master-1 cinder-backup[129230]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Feb 19 21:32:18.775828 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[ab6ace38-8d9b-4b09-a396-76c4fe384d97]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:32:18.776498 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:32:18.782509 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Feb 19 21:32:18.783103 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[2ad1037d-9796-4d9a-b538-aaca37292826]: [Errno 2] No such file or directory: 'systool' {{(pid=150093) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} Feb 19 21:32:18.783103 ncdv-master-1 cinder-backup[129230]: Traceback (most recent call last): Feb 19 21:32:18.783103 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd Feb 19 21:32:18.783103 ncdv-master-1 cinder-backup[129230]: ret = func(*f_args, **f_kwargs) Feb 19 21:32:18.783103 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Feb 19 21:32:18.783103 ncdv-master-1 cinder-backup[129230]: return func(*args, **kwargs) Feb 19 21:32:18.783103 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Feb 19 21:32:18.783103 ncdv-master-1 cinder-backup[129230]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Feb 19 21:32:18.783103 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Feb 19 21:32:18.783103 ncdv-master-1 cinder-backup[129230]: return putils.execute(on_execute=on_execute, Feb 19 21:32:18.783103 ncdv-master-1 cinder-backup[129230]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Feb 19 21:32:18.783103 ncdv-master-1 cinder-backup[129230]: obj = subprocess.Popen(cmd, Feb 19 21:32:18.783103 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Feb 19 21:32:18.783103 ncdv-master-1 cinder-backup[129230]: self._execute_child(args, executable, preexec_fn, close_fds, Feb 19 21:32:18.783103 ncdv-master-1 cinder-backup[129230]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Feb 19 21:32:18.783103 ncdv-master-1 cinder-backup[129230]: raise child_exception_type(errno_num, err_msg, err_filename) Feb 19 21:32:18.783103 ncdv-master-1 cinder-backup[129230]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Feb 19 21:32:18.784300 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[2ad1037d-9796-4d9a-b538-aaca37292826]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:32:18.784791 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:32:18.792680 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.008s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:32:18.793196 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[a9389bf6-4259-44a6-b289-94540dd77a0a]: (4, ('/dev/sda1\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:32:18.794310 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:32:18.803061 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.009s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:32:18.803399 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[1b7562a7-2663-47f7-801f-9db566fc7ddf]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:32:18.804548 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:32:18.810811 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.006s {{(pid=150093) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:32:18.811237 ncdv-master-1 cinder-backup[129230]: DEBUG oslo.privsep.daemon [-] privsep: reply[19eb77f6-2635-456c-81f5-9e8f4980d347]: (4, ('856ceada-2af3-8e47-a879-77ebed99126e\n', '')) {{(pid=150093) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} Feb 19 21:32:18.812020 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] Running cmd (subprocess): nvme version {{(pid=129230) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Feb 19 21:32:18.831128 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_concurrency.processutils [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] CMD "nvme version" returned: 0 in 0.019s {{(pid=129230) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Feb 19 21:32:18.832893 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=129230) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} Feb 19 21:32:18.833268 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=129230) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} Feb 19 21:32:18.833629 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.lightos [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:9cde57c4-ac04-4e2d-ab4a-203cb09943a6 dsc: {{(pid=129230) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} Feb 19 21:32:18.834281 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.utils [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] <== get_connector_properties: return (86ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.30', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:6e78a5d1a02d', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '856ceada-2af3-8e47-a879-77ebed99126e', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:9cde57c4-ac04-4e2d-ab4a-203cb09943a6', 'found_dsc': ''} {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:32:19.506861 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connector [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] Factory for rbd on None {{(pid=129230) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} Feb 19 21:32:19.508292 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-78bcad5a-4a24-4ba3-af4e-0ce1c804279c', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '78bcad5a-4a24-4ba3-af4e-0ce1c804279c', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:32:19.509370 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.linuxrbd [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] opening connection to ceph cluster (timeout=-1). {{(pid=129230) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Feb 19 21:32:19.549007 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] <== connect_volume: return (39ms) {'path': } {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:32:19.556138 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_db.sqlalchemy.engines [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129230) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Feb 19 21:32:19.567353 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] Starting restore from Ceph backup=f0df62d4-2f93-4fe0-acec-5a1cbed6c181 to volume=volume-78bcad5a-4a24-4ba3-af4e-0ce1c804279c {{(pid=129230) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1274}} Feb 19 21:32:19.627307 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] Looking for snapshot of backup base 'volume-8ef288d9-4908-4e07-8642-09ca661890e3.backup.f0df62d4-2f93-4fe0-acec-5a1cbed6c181' {{(pid=129230) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Feb 19 21:32:19.627718 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] Found snapshot 'backup.f0df62d4-2f93-4fe0-acec-5a1cbed6c181.snap.1645306331.1959336' {{(pid=129230) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Feb 19 21:32:19.631917 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] Volume file is RBD. {{(pid=129230) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1184}} Feb 19 21:32:19.661888 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] Attempting differential restore. {{(pid=129230) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1234}} Feb 19 21:32:19.662378 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] Attempting incremental restore from base='volume-8ef288d9-4908-4e07-8642-09ca661890e3.backup.f0df62d4-2f93-4fe0-acec-5a1cbed6c181' snap='backup.f0df62d4-2f93-4fe0-acec-5a1cbed6c181.snap.1645306331.1959336' {{(pid=129230) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1089}} Feb 19 21:32:19.662773 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] Performing differential transfer from 'volume-8ef288d9-4908-4e07-8642-09ca661890e3.backup.f0df62d4-2f93-4fe0-acec-5a1cbed6c181' to 'volume-78bcad5a-4a24-4ba3-af4e-0ce1c804279c' {{(pid=129230) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Feb 19 21:32:19.663167 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-8ef288d9-4908-4e07-8642-09ca661890e3.backup.f0df62d4-2f93-4fe0-acec-5a1cbed6c181@backup.f0df62d4-2f93-4fe0-acec-5a1cbed6c181.snap.1645306331.1959336 -' into... {{(pid=129230) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Feb 19 21:32:19.663534 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_bynalp14 --pool volumes - volumes/volume-78bcad5a-4a24-4ba3-af4e-0ce1c804279c' {{(pid=129230) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Feb 19 21:32:22.817946 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] Restore transfer completed in 3.1546s {{(pid=129230) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1111}} Feb 19 21:32:22.819042 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=129230) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1285}} Feb 19 21:32:22.841445 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] Restoring volume base metadata {{(pid=129230) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} Feb 19 21:32:22.862270 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] No metadata of type 'volume-metadata' to restore {{(pid=129230) put /opt/stack/cinder/cinder/backup/driver.py:345}} Feb 19 21:32:22.862657 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.driver [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] Restoring volume glance metadata {{(pid=129230) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}} Feb 19 21:32:22.989512 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] Restore to volume 78bcad5a-4a24-4ba3-af4e-0ce1c804279c finished successfully. {{(pid=129230) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1292}} Feb 19 21:32:22.990995 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-78bcad5a-4a24-4ba3-af4e-0ce1c804279c', 'hosts': ['10.222.0.30'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '78bcad5a-4a24-4ba3-af4e-0ce1c804279c', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} Feb 19 21:32:23.000822 ncdv-master-1 cinder-backup[129230]: DEBUG os_brick.initiator.connectors.rbd [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] <== disconnect_volume: return (8ms) None {{(pid=129230) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} Feb 19 21:32:23.045961 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.manager [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] Encryption key ID for volume 78bcad5a-4a24-4ba3-af4e-0ce1c804279c already matches encryption key ID in backup f0df62d4-2f93-4fe0-acec-5a1cbed6c181. {{(pid=129230) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} Feb 19 21:32:23.074551 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-d969a2d2-cbf6-45bd-bcbc-6882d801c959 tempest-TestVolumeBackupRestore-808817250 None] Finished restoring backup f0df62d4-2f93-4fe0-acec-5a1cbed6c181 to volume 78bcad5a-4a24-4ba3-af4e-0ce1c804279c. Feb 19 21:33:03.420837 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:33:03.421545 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:33:22.896115 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-15809b9d-e314-42ee-933b-7a7676a6b173 tempest-TestVolumeBackupRestore-808817250 None] Delete backup started, backup: f0df62d4-2f93-4fe0-acec-5a1cbed6c181. Feb 19 21:33:22.898611 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-15809b9d-e314-42ee-933b-7a7676a6b173 tempest-TestVolumeBackupRestore-808817250 None] Delete started for backup=f0df62d4-2f93-4fe0-acec-5a1cbed6c181 {{(pid=129230) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Feb 19 21:33:22.899099 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-15809b9d-e314-42ee-933b-7a7676a6b173 tempest-TestVolumeBackupRestore-808817250 None] Trying diff format basename='volume-8ef288d9-4908-4e07-8642-09ca661890e3.backup.f0df62d4-2f93-4fe0-acec-5a1cbed6c181' for backup base image of volume 8ef288d9-4908-4e07-8642-09ca661890e3. {{(pid=129230) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Feb 19 21:33:22.960503 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-15809b9d-e314-42ee-933b-7a7676a6b173 tempest-TestVolumeBackupRestore-808817250 None] Looking for snapshot of backup base 'volume-8ef288d9-4908-4e07-8642-09ca661890e3.backup.f0df62d4-2f93-4fe0-acec-5a1cbed6c181' {{(pid=129230) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Feb 19 21:33:22.960896 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-15809b9d-e314-42ee-933b-7a7676a6b173 tempest-TestVolumeBackupRestore-808817250 None] Found snapshot 'backup.f0df62d4-2f93-4fe0-acec-5a1cbed6c181.snap.1645306331.1959336' {{(pid=129230) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Feb 19 21:33:22.961272 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-15809b9d-e314-42ee-933b-7a7676a6b173 tempest-TestVolumeBackupRestore-808817250 None] Deleting backup snapshot='backup.f0df62d4-2f93-4fe0-acec-5a1cbed6c181.snap.1645306331.1959336' {{(pid=129230) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Feb 19 21:33:23.828621 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_db.sqlalchemy.engines [None req-0310f1a8-baa1-43c0-8dfd-d981a2de117f None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129230) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Feb 19 21:33:23.877074 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.drivers.ceph [None req-15809b9d-e314-42ee-933b-7a7676a6b173 tempest-TestVolumeBackupRestore-808817250 None] Deleting backup base image='volume-8ef288d9-4908-4e07-8642-09ca661890e3.backup.f0df62d4-2f93-4fe0-acec-5a1cbed6c181' of volume 8ef288d9-4908-4e07-8642-09ca661890e3. Feb 19 21:33:24.324216 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-15809b9d-e314-42ee-933b-7a7676a6b173 tempest-TestVolumeBackupRestore-808817250 None] Base backup image='volume-8ef288d9-4908-4e07-8642-09ca661890e3.backup.f0df62d4-2f93-4fe0-acec-5a1cbed6c181' of volume 8ef288d9-4908-4e07-8642-09ca661890e3 deleted. {{(pid=129230) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Feb 19 21:33:24.358858 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.backup.drivers.ceph [None req-15809b9d-e314-42ee-933b-7a7676a6b173 tempest-TestVolumeBackupRestore-808817250 None] Delete of backup 'f0df62d4-2f93-4fe0-acec-5a1cbed6c181' for volume '8ef288d9-4908-4e07-8642-09ca661890e3' finished. {{(pid=129230) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Feb 19 21:33:24.380978 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.quota [None req-15809b9d-e314-42ee-933b-7a7676a6b173 tempest-TestVolumeBackupRestore-808817250 None] Created reservations ['1a580a13-f3c0-46ff-b8f1-e73b3a00b868', '72f63249-566b-45ce-8c10-5b49e8ca9aeb'] {{(pid=129230) reserve /opt/stack/cinder/cinder/quota.py:755}} Feb 19 21:33:24.404102 ncdv-master-1 cinder-backup[129230]: INFO cinder.backup.manager [None req-15809b9d-e314-42ee-933b-7a7676a6b173 tempest-TestVolumeBackupRestore-808817250 None] Delete backup finished, backup f0df62d4-2f93-4fe0-acec-5a1cbed6c181 deleted. Feb 19 21:34:03.426253 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:34:03.426882 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:35:03.423242 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:35:03.423987 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:36:03.419832 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:36:03.420649 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:37:03.416389 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:37:03.417123 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:38:03.421567 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:38:03.422289 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:39:03.427315 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:39:03.427952 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:40:03.427857 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:40:03.428497 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:41:03.424427 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:41:03.425149 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:42:03.425747 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:42:03.426509 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:43:03.430409 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:43:03.431138 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:44:03.435316 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:44:03.435932 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:45:03.439045 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:45:03.439721 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:46:03.445218 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:46:03.445937 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:47:03.450560 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:47:03.451288 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:48:03.455204 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:48:03.455865 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:49:03.460490 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:49:03.461191 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:50:03.456843 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:50:03.457558 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:51:03.462181 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:51:03.462884 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:52:03.458794 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:52:03.459538 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:54:03.466024 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:54:03.466723 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:55:03.471710 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:55:03.472478 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:56:03.469028 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:56:03.469761 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:57:03.465076 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:57:03.465832 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:58:03.461661 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:58:03.462380 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 21:59:03.467428 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 21:59:03.468282 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 22:00:03.464577 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 22:00:03.465275 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 22:01:03.461101 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 22:01:03.461849 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 22:02:03.466878 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 22:02:03.467586 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 22:03:03.464217 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 22:03:03.465028 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 22:04:03.460818 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 22:04:03.461750 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 22:05:03.457457 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 22:05:03.458236 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 22:06:03.465214 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 22:06:03.466462 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 22:07:03.463611 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 22:07:03.464402 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 22:08:03.460563 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 22:08:03.461307 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 22:09:03.457140 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 22:09:03.457909 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 22:10:03.462571 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 22:10:03.463466 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 22:11:03.460407 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 22:11:03.461247 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 22:12:03.460795 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 22:12:03.461541 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 22:13:03.458159 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 22:13:03.458878 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 22:14:03.463278 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 22:14:03.464021 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 22:15:03.460366 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 22:15:03.461083 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 22:16:03.457172 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 22:16:03.457172 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Feb 19 22:17:03.453334 ncdv-master-1 cinder-backup[129230]: DEBUG oslo_service.periodic_task [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129230) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Feb 19 22:17:03.454009 ncdv-master-1 cinder-backup[129230]: DEBUG cinder.manager [None req-1d02f955-af03-466b-9212-ae95b01aed54 None None] Notifying Schedulers of capabilities ... {{(pid=129230) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}