-- Logs begin at Sat 2021-12-25 20:01:33 UTC, end at Sat 2021-12-25 22:38:35 UTC. -- Dec 25 20:57:52.689025 ncdv-master-1 systemd[1]: Started Devstack devstack@c-bak.service. Dec 25 20:57:55.969748 ncdv-master-1 cinder-backup[129359]: INFO cinder.cmd.backup [-] Backup running with 1 processes. Dec 25 20:57:55.970513 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_concurrency.lockutils [-] Acquired lock "singleton_lock" {{(pid=129359) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:294}} Dec 25 20:57:55.970905 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_concurrency.lockutils [-] Releasing lock "singleton_lock" {{(pid=129359) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:312}} Dec 25 20:57:56.129986 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_db.sqlalchemy.engines [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129359) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 25 20:57:56.407798 ncdv-master-1 cinder-backup[129359]: INFO cinder.rpc [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] Automatically selected cinder-backup objects version 1.39 as minimum service version. Dec 25 20:57:56.412063 ncdv-master-1 cinder-backup[129359]: INFO cinder.rpc [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] Automatically selected cinder-backup RPC version 2.2 as minimum service version. Dec 25 20:57:56.416264 ncdv-master-1 cinder-backup[129359]: INFO cinder.rpc [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] Automatically selected cinder-volume objects version 1.39 as minimum service version. Dec 25 20:57:56.419936 ncdv-master-1 cinder-backup[129359]: INFO cinder.rpc [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] Automatically selected cinder-volume RPC version 3.16 as minimum service version. Dec 25 20:57:56.424838 ncdv-master-1 cinder-backup[129359]: INFO cinder.rpc [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] Automatically selected cinder-scheduler objects version 1.39 as minimum service version. Dec 25 20:57:56.428853 ncdv-master-1 cinder-backup[129359]: INFO cinder.rpc [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] Automatically selected cinder-scheduler RPC version 3.12 as minimum service version. Dec 25 20:57:56.479878 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_db.sqlalchemy.engines [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129359) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 25 20:57:56.609660 ncdv-master-1 cinder-backup[129359]: INFO oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] Starting 1 workers Dec 25 20:57:56.614561 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] Started child 129839 {{(pid=129359) _start_child /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:575}} Dec 25 20:57:56.615636 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] Full set of CONF: {{(pid=129359) wait /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:649}} Dec 25 20:57:56.616094 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] ******************************************************************************** {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Dec 25 20:57:56.616512 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] Configuration options gathered from: {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Dec 25 20:57:56.616943 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] command line args: ['--config-file', '/etc/cinder/cinder.conf'] {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Dec 25 20:57:56.617617 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] config files: ['/etc/cinder/cinder.conf'] {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Dec 25 20:57:56.617966 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] ================================================================================ {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Dec 25 20:57:56.618774 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] allow_availability_zone_fallback = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.619209 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] allow_compression_on_image_upload = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.619779 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] allowed_direct_url_schemes = [] {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.620217 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] api_rate_limit = True {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.620733 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] auth_strategy = keystone {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.621235 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_availability_zone = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.621674 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backup_api_class = cinder.backup.api.API {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.627180 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backup_ceph_chunk_size = 134217728 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.627670 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backup_ceph_conf = /etc/ceph/ceph.conf {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.628109 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backup_ceph_image_journals = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.628540 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backup_ceph_pool = backups {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.629049 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backup_ceph_stripe_count = 0 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.629487 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backup_ceph_stripe_unit = 0 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.629933 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backup_ceph_user = cinder-bak {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.630390 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backup_driver = cinder.backup.drivers.ceph.CephBackupDriver {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.630843 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backup_driver_init_check_interval = 60 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.631270 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backup_driver_stats_polling_interval = 60 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.631701 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backup_manager = cinder.backup.manager.BackupManager {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.632159 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backup_max_operations = 15 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.634115 ncdv-master-1 cinder-backup[129839]: INFO cinder.service [-] Starting cinder-backup node (version 19.1.0) Dec 25 20:57:56.635193 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backup_metadata_version = 2 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.635749 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backup_name_template = backup-%s {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.636190 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backup_native_threads_pool_size = 60 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.636685 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backup_object_number_per_notification = 10 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.637128 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backup_service_inithost_offload = True {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.643050 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backup_timer_interval = 120 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.643050 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backup_use_same_host = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.643050 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backup_use_temp_snapshot = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.643050 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backup_workers = 1 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.643050 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] chap_password = **** {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.643050 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] chap_username = {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.643050 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.643601 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] cinder_internal_tenant_project_id = f8de2c022c7e473abd53e78670253e88 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.643601 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] cinder_internal_tenant_user_id = 0f668b27e2af43fd812f309f6908199c {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.643601 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] compression_format = gzip {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.643601 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] config_dir = [] {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.643601 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] config_file = ['/etc/cinder/cinder.conf'] {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.643601 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] config_source = [] {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.643977 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] consistencygroup_api_class = cinder.consistencygroup.api.API {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.643977 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] control_exchange = openstack {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.643977 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] debug = True {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.644177 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] default_availability_zone = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.654813 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] default_group_type = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.655592 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'swiftclient=WARN', 'botocore=WARN'] {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.658254 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] default_volume_type = ceph {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.658892 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] driver_client_cert = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.659498 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] driver_client_cert_key = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.660197 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] driver_data_namespace = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.661467 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] driver_ssl_cert_path = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.662646 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] driver_ssl_cert_verify = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.663779 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] driver_use_ssl = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.664101 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] enable_new_services = True {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.665954 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] enable_unsupported_driver = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.667789 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] enabled_backends = ['ceph'] {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.668217 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] enforce_multipath_for_image_xfer = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.668582 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] filter_function = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.668975 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] glance_api_insecure = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.671062 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] glance_api_servers = ['http://10.222.0.32/image'] {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.671062 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] glance_api_ssl_compression = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.671062 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] glance_ca_certificates_file = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.671062 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] glance_catalog_info = image:glance:publicURL {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.671062 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] glance_certfile = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.671439 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] glance_core_properties = ['checksum', 'container_format', 'disk_format', 'image_name', 'image_id', 'min_disk', 'min_ram', 'name', 'size'] {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.671522 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] glance_keyfile = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.672021 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] glance_num_retries = 3 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.672895 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] glance_request_timeout = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.673272 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] goodness_function = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.673647 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] graceful_shutdown_timeout = 5 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.674001 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] group_api_class = cinder.group.api.API {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.674351 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] host = ncdv-master-1 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.674700 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] iet_conf = /etc/iet/ietd.conf {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.675073 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] image_compress_on_upload = True {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.675433 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] image_conversion_address_space_limit = 1 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.675794 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] image_conversion_cpu_limit = 60 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.676295 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] image_conversion_dir = /opt/stack/data/cinder/conversion {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.676655 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] image_upload_use_cinder_backend = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.689839 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] image_upload_use_internal_tenant = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.690290 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] image_volume_cache_enabled = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.690659 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] image_volume_cache_max_count = 0 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.691019 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] image_volume_cache_max_size_gb = 0 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.691394 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] instance_format = [instance: %(uuid)s] {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.691785 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] instance_uuid_format = [instance: %(uuid)s] {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.692163 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] iscsi_iotype = fileio {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.692532 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] iscsi_secondary_ip_addresses = [] {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.692905 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] iscsi_target_flags = {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.693263 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] iscsi_write_cache = on {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.693617 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] iser_helper = tgtadm {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.694208 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] iser_ip_address = 10.222.0.32 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.694615 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] iser_port = 3260 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.694967 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.695332 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] log_config_append = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.695681 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.696025 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] log_dir = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.696364 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] log_file = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.696649 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] log_options = True {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.697036 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] log_rotate_interval = 1 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.697426 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] log_rotate_interval_type = days {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.697784 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] log_rotation_type = none {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.698081 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.698361 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.698716 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.699083 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.699371 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] logging_user_identity_format = %(user)s %(tenant)s %(domain)s %(user_domain)s %(project_domain)s {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.699753 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] max_age = 0 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.700099 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] max_logfile_count = 30 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.700461 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] max_logfile_size_mb = 200 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.700831 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] max_over_subscription_ratio = 20.0 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.702232 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.lockutils [None req-dc3d93dc-ed05-4ec8-9777-a4bb3fdd77f8 None None] Lock "cleanup_incomplete_backups_129359" acquired by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: waited 0.013s {{(pid=129839) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:386}} Dec 25 20:57:56.702560 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-dc3d93dc-ed05-4ec8-9777-a4bb3fdd77f8 None None] Cleaning up incomplete backup operations. Dec 25 20:57:56.713421 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] message_reap_interval = 86400 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.714073 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] message_ttl = 2592000 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.714435 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] monkey_patch = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.715112 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] monkey_patch_modules = [] {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.715414 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] my_ip = 10.222.0.32 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.715855 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] no_snapshot_gb_quota = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.716235 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] num_iser_scan_tries = 3 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.716608 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] num_shell_tries = 3 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.717110 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] num_volume_device_scan_tries = 3 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.727754 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] nvmet_ns_id = 10 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.728163 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] nvmet_port_id = 1 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.728571 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] osapi_max_limit = 1000 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.728966 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] osapi_volume_ext_list = [] {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.729413 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] osapi_volume_extension = ['cinder.api.contrib.standard_extensions'] {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.729800 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] osapi_volume_listen = 0.0.0.0 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.730224 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] osapi_volume_listen_port = 8776 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.730524 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] osapi_volume_use_ssl = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.730883 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] osapi_volume_workers = 2 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.731232 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] per_volume_size_limit = -1 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.731581 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] periodic_fuzzy_delay = 60 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.731942 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] periodic_interval = 60 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.732298 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] public_endpoint = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.732641 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] publish_errors = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.733018 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] quota_backup_gigabytes = 1000 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.733377 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] quota_backups = 10 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.733738 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] quota_consistencygroups = 10 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.734093 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] quota_driver = cinder.quota.DbQuotaDriver {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.734552 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] quota_gigabytes = 1000 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.734913 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] quota_groups = 10 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.735267 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] quota_snapshots = 10 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.735615 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] quota_volumes = 10 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.735954 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] rate_limit_burst = 0 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.736798 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] rate_limit_except_level = CRITICAL {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.737310 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] rate_limit_interval = 0 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.737675 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] replication_device = **** {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.738034 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] report_discard_supported = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.738374 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] report_interval = 10 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.738818 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] reservation_clean_interval = 86400 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.739101 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] reservation_expire = 86400 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.739449 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] reserved_percentage = 0 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.739814 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] resource_query_filters_file = /etc/cinder/resource_filters.json {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.740161 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] restore_discard_excess_bytes = True {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.740509 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] rootwrap_config = /etc/cinder/rootwrap.conf {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.740910 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] rpc_response_timeout = 120 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.741251 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] run_external_periodic_tasks = True {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.741622 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] scheduler_manager = cinder.scheduler.manager.SchedulerManager {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.741985 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] scst_target_driver = iscsi {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.742325 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] scst_target_iqn_name = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.742611 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] service_down_time = 60 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.742991 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] snapshot_name_template = snapshot-%s {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.743327 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] split_loggers = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.743610 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] state_path = /opt/stack/data/cinder {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.743959 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] storage_availability_zone = nova {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.744306 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] storage_protocol = iscsi {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.744680 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] syslog_log_facility = LOG_USER {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.745071 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] target_helper = lioadm {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.745533 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] target_ip_address = 10.222.0.32 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.745873 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] target_port = 3260 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.746225 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] target_prefix = iqn.2010-10.org.openstack: {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.746574 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] target_protocol = iscsi {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.746917 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] trace_flags = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.747266 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] transfer_api_class = cinder.transfer.api.API {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.747608 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] transport_url = **** {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.747951 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] until_refresh = 0 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.748294 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] use_chap_auth = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.748628 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] use_default_quota_class = True {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.749182 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] use_eventlog = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.749529 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] use_journal = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.749869 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] use_json = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.750262 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] use_multipath_for_image_xfer = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.750610 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] use_stderr = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.750943 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] use_syslog = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.751325 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] verify_glance_signatures = enabled {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.751680 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] volume_api_class = cinder.volume.api.API {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.752014 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] volume_backend_name = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.752380 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] volume_clear = zero {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.752711 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] volume_clear_ionice = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.753070 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] volume_clear_size = 0 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.753420 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.753753 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] volume_copy_bps_limit = 0 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.754094 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] volume_dd_blocksize = 1M {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.754447 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] volume_manager = cinder.volume.manager.VolumeManager {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.754813 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] volume_name_template = volume-%s {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.755175 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] volume_usage_audit_period = month {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.755602 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] volumes_dir = /opt/stack/data/cinder/volumes {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.755936 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] watch_log_file = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Dec 25 20:57:56.756422 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.756820 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.757239 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_policy.enforce_new_defaults = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.757600 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_policy.enforce_scope = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.757996 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_policy.policy_default_rule = default {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.758395 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_policy.policy_dirs = ['policy.d'] {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.758767 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_policy.policy_file = policy.yaml {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.759146 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.759487 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.759826 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.760160 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.760502 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.760889 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_concurrency.disable_process_locking = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.761267 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_concurrency.lock_path = /opt/stack/data/cinder {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.761703 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.admin_password = **** {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.762267 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.admin_tenant_name = admin {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.762617 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.admin_token = **** {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.762965 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.admin_user = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.763349 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.auth_admin_prefix = {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.763701 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.764064 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.auth_port = 35357 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.764443 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.auth_protocol = https {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.764830 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.auth_section = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.765228 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.auth_type = password {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.765663 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.auth_uri = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.766008 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.auth_version = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.766349 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.cache = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.766721 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.cafile = /opt/stack/data/ca-bundle.pem {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.767055 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.certfile = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.767401 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.delay_auth_decision = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.767770 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.enforce_token_bind = permissive {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.768130 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.http_connect_timeout = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.768470 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.http_request_max_retries = 3 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.768826 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.identity_uri = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.769474 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.include_service_catalog = True {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.769828 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.insecure = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.770178 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.interface = public {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.770513 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.keyfile = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.770862 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.771224 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.771574 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.771914 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.772254 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.772615 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.memcache_secret_key = **** {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.773024 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.memcache_security_strategy = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.773373 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.773787 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.memcached_servers = ['localhost:11211'] {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.774129 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.region_name = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.774513 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.service_token_roles = ['service'] {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.774859 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.service_token_roles_required = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.775195 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.service_type = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.775534 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.token_cache_time = 300 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.775896 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] keystone_authtoken.www_authenticate_uri = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.776340 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] cache.backend = dogpile.cache.null {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.776721 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] cache.backend_argument = **** {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.777092 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] cache.config_prefix = cache.oslo {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.777477 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] cache.debug_cache_backend = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.778194 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] cache.enabled = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.780920 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] cache.expiration_time = 600 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.780920 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] cache.memcache_dead_retry = 300 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.780920 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] cache.memcache_pool_connection_get_timeout = 10 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.780920 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] cache.memcache_pool_flush_on_reconnect = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.780920 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] cache.memcache_pool_maxsize = 10 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.780920 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] cache.memcache_pool_unused_timeout = 60 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.780920 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] cache.memcache_servers = ['localhost:11211'] {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.782273 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] cache.memcache_socket_timeout = 1.0 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.782786 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] cache.proxies = [] {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.783271 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] cache.tls_allowed_ciphers = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.783756 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] cache.tls_cafile = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.784248 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] cache.tls_certfile = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.784768 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] cache.tls_enabled = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.785277 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] cache.tls_keyfile = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.785819 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] coordination.backend_url = etcd3+http://10.222.0.32:2379 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.786367 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] database.backend = sqlalchemy {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.786887 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] database.connection = **** {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.787592 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] database.connection_debug = 0 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.788128 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] database.connection_parameters = {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.788626 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] database.connection_recycle_time = 3600 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.789171 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] database.connection_trace = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.789659 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] database.db_inc_retry_interval = True {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.790146 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] database.db_max_retries = 20 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.790662 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] database.db_max_retry_interval = 10 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.791172 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] database.db_retry_interval = 1 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.791675 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] database.max_overflow = 50 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.792176 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] database.max_pool_size = 5 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.792676 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] database.max_retries = 10 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.793375 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] database.mysql_enable_ndb = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.793894 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] database.mysql_sql_mode = TRADITIONAL {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.794413 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] database.pool_timeout = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.794914 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] database.retry_interval = 10 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.795423 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] database.slave_connection = **** {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.795918 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] database.sqlite_synchronous = True {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.796429 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] database.use_db_reconnect = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.797569 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.798082 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.798576 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_metrics.metrics_process_name = {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.799088 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.799608 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.800160 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] profiler.connection_string = messaging:// {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.800590 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] profiler.enabled = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.801224 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] profiler.es_doc_type = notification {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.801891 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] profiler.es_scroll_size = 10000 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.802412 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] profiler.es_scroll_time = 2m {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.802921 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] profiler.filter_error_trace = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.803423 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] profiler.hmac_keys = SECRET_KEY {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.803929 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] profiler.sentinel_service_name = mymaster {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.804422 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] profiler.socket_timeout = 0.1 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.804937 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] profiler.trace_sqlalchemy = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.805502 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] privsep_osbrick.capabilities = [21] {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.806028 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] privsep_osbrick.group = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.806525 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] privsep_osbrick.helper_command = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.807025 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] privsep_osbrick.logger_name = oslo_privsep.daemon {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.807512 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] privsep_osbrick.thread_pool_size = 4 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.808240 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] privsep_osbrick.user = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.808777 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] cinder_sys_admin.capabilities = [0, 1, 2, 3, 12, 21] {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.809275 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] cinder_sys_admin.group = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.809753 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] cinder_sys_admin.helper_command = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.810247 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] cinder_sys_admin.logger_name = oslo_privsep.daemon {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.810746 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] cinder_sys_admin.thread_pool_size = 4 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.811237 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] cinder_sys_admin.user = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.811756 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] service_user.auth_section = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.812258 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] service_user.auth_type = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.812792 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] service_user.cafile = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.813312 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] service_user.certfile = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.814006 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] service_user.collect_timing = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.814488 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] service_user.insecure = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.814978 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] service_user.keyfile = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.815478 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] service_user.send_service_user_token = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.816051 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] service_user.split_loggers = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.816541 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] service_user.timeout = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.817120 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.backend_availability_zone = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.817628 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.chap_password = **** {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.818134 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.chap_username = {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.818626 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.819144 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.deferred_deletion_delay = 0 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.819639 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.deferred_deletion_purge_interval = 60 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.820145 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.driver_client_cert = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.820648 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.driver_client_cert_key = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.821380 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.driver_data_namespace = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.821879 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.driver_ssl_cert_path = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.822370 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.driver_ssl_cert_verify = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.823167 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.driver_use_ssl = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.823673 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.enable_deferred_deletion = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.824220 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.enable_unsupported_driver = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.824738 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.enforce_multipath_for_image_xfer = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.825274 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.filter_function = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.825751 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.goodness_function = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.826282 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.iet_conf = /etc/iet/ietd.conf {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.826766 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.image_upload_use_cinder_backend = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.827254 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.image_upload_use_internal_tenant = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.827751 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.image_volume_cache_enabled = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.828342 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.image_volume_cache_max_count = 0 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.829147 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.image_volume_cache_max_size_gb = 0 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.829647 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.iscsi_iotype = fileio {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.830160 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.iscsi_secondary_ip_addresses = [] {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.830656 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.iscsi_target_flags = {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.831186 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.iscsi_write_cache = on {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.831694 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.iser_helper = tgtadm {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.832386 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.iser_ip_address = 10.222.0.32 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.832897 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.iser_port = 3260 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.833401 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.833947 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.max_over_subscription_ratio = 20.0 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.834455 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.num_iser_scan_tries = 3 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.834951 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.num_shell_tries = 3 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.835437 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.num_volume_device_scan_tries = 3 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.835934 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.nvmet_ns_id = 10 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.836472 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.nvmet_port_id = 1 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.837025 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.rados_connect_timeout = -1 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.837733 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.rados_connection_interval = 5 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.838235 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.rados_connection_retries = 3 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.839024 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.rbd_ceph_conf = {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.839525 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.rbd_cluster_name = ceph {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.840024 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.rbd_exclusive_cinder_pool = True {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.840516 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.rbd_flatten_volume_from_snapshot = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.841041 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.rbd_max_clone_depth = 5 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.841556 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.rbd_pool = rbd {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.842062 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.rbd_secret_uuid = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.842562 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.rbd_store_chunk_size = 4 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.843039 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.rbd_user = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.844261 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.replication_connect_timeout = 5 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.844757 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.replication_device = **** {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.845275 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.report_discard_supported = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.845798 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.report_dynamic_total_capacity = True {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.846284 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.reserved_percentage = 0 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.846790 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.scst_target_driver = iscsi {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.847268 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.scst_target_iqn_name = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.847769 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.storage_protocol = iscsi {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.848287 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.target_helper = tgtadm {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.848931 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.target_ip_address = 10.222.0.32 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.849423 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.target_port = 3260 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.849929 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.target_prefix = iqn.2010-10.org.openstack: {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.850433 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.target_protocol = iscsi {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.850925 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.trace_flags = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.851600 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.unique_fqdn_network = True {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.852097 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.use_chap_auth = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.852581 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.use_multipath_for_image_xfer = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.853116 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.volume_backend_name = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.853613 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.volume_clear = zero {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.854108 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.volume_clear_ionice = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.854592 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.volume_clear_size = 0 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.855102 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.855585 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.volume_copy_bps_limit = 0 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.856083 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.volume_dd_blocksize = 1M {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.856676 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] backend_defaults.volumes_dir = /opt/stack/data/cinder/volumes {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.857255 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_reports.file_event_handler = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.857773 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_reports.file_event_handler_interval = 1 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.858277 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_reports.log_dir = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.858854 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.859976 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.860743 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.861257 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.861787 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.862275 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.862780 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_rabbit.heartbeat_in_pthread = True {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.863296 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_rabbit.heartbeat_rate = 2 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.863798 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.864284 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.864810 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.865343 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.865855 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.866668 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.867170 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.867672 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.868159 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.868680 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.869213 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.869729 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.870219 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.870754 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_rabbit.ssl = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.871257 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_rabbit.ssl_ca_file = {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.871770 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_rabbit.ssl_cert_file = {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.872267 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_rabbit.ssl_key_file = {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.872767 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_rabbit.ssl_version = {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.873365 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_notifications.driver = [] {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.873871 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_notifications.retry = -1 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.874540 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.875916 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] oslo_messaging_notifications.transport_url = **** {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.876487 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] key_manager.backend = cinder.keymgr.conf_key_mgr.ConfKeyManager {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.877113 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] barbican.auth_endpoint = http://localhost/identity/v3 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.877617 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] barbican.barbican_api_version = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.878181 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] barbican.barbican_endpoint = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.878737 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] barbican.barbican_endpoint_type = public {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.879226 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] barbican.barbican_region_name = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.879720 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] barbican.cafile = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.880197 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] barbican.certfile = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.880697 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] barbican.collect_timing = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.881224 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] barbican.insecure = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.881884 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] barbican.keyfile = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.882412 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] barbican.number_of_retries = 60 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.882916 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] barbican.retry_delay = 1 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.883402 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] barbican.send_service_user_token = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.883897 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] barbican.split_loggers = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.884387 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] barbican.timeout = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.884869 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] barbican.verify_ssl = True {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.885387 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] barbican.verify_ssl_path = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.885882 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] barbican_service_user.auth_section = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.886376 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] barbican_service_user.auth_type = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.886871 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] barbican_service_user.cafile = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.887347 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] barbican_service_user.certfile = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.887840 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] barbican_service_user.collect_timing = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.888328 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] barbican_service_user.insecure = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.888831 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] barbican_service_user.keyfile = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.889363 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] barbican_service_user.split_loggers = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.889838 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] barbican_service_user.timeout = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.891203 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] vault.approle_role_id = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.891704 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] vault.approle_secret_id = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.892215 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] vault.cafile = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.892693 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] vault.certfile = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.893279 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] vault.collect_timing = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.893769 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] vault.insecure = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.894259 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] vault.keyfile = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.894753 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] vault.kv_mountpoint = secret {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.895234 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] vault.kv_version = 2 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.895721 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] vault.namespace = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.896367 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] vault.root_token_id = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.896862 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] vault.split_loggers = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.897358 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] vault.ssl_ca_crt_file = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.897835 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] vault.timeout = None {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.898342 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] vault.use_ssl = False {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.898855 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] vault.vault_url = http://127.0.0.1:8200 {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Dec 25 20:57:56.899280 ncdv-master-1 cinder-backup[129359]: DEBUG oslo_service.service [None req-34322119-7e73-462c-a569-3f4d0a3245e9 None None] ******************************************************************************** {{(pid=129359) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Dec 25 20:57:56.919462 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_db.sqlalchemy.engines [None req-dc3d93dc-ed05-4ec8-9777-a4bb3fdd77f8 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129839) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 25 20:57:56.950770 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.lockutils [None req-dc3d93dc-ed05-4ec8-9777-a4bb3fdd77f8 None None] Lock "cleanup_incomplete_backups_129359" "released" by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: held 0.246s {{(pid=129839) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:400}} Dec 25 20:57:56.957527 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_db.sqlalchemy.engines [None req-dc3d93dc-ed05-4ec8-9777-a4bb3fdd77f8 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129839) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 25 20:57:56.970538 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-dc3d93dc-ed05-4ec8-9777-a4bb3fdd77f8 None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 20:57:56.991634 ncdv-master-1 cinder-backup[129839]: INFO cinder.keymgr.migration [None req-ec280529-5b9a-4827-a347-bdffc586ac9a None None] Not migrating encryption keys because the ConfKeyManager is still in use. Dec 25 20:57:57.019562 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.service [None req-dc3d93dc-ed05-4ec8-9777-a4bb3fdd77f8 None None] Creating RPC server for service cinder-backup {{(pid=129839) start /opt/stack/cinder/cinder/service.py:231}} Dec 25 20:57:57.030451 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.service [None req-dc3d93dc-ed05-4ec8-9777-a4bb3fdd77f8 None None] Pinning object versions for RPC server serializer to 1.39 {{(pid=129839) start /opt/stack/cinder/cinder/service.py:237}} Dec 25 20:59:43.081142 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 20:59:43.081905 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:00:43.086881 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:00:43.087554 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:01:43.083055 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:01:43.083753 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:02:43.088550 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:02:43.089220 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:03:43.084714 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:03:43.085351 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:04:43.090435 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:04:43.091111 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:05:43.095693 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:05:43.096269 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:06:43.100462 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:06:43.101187 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:07:43.109442 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:07:43.110067 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:08:43.113279 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:08:43.115163 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:09:43.114424 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:09:43.115897 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:10:43.110680 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:10:43.111374 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:11:43.113333 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:11:43.114037 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:12:43.118274 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:12:43.118969 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:13:43.123163 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:13:43.124773 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:14:43.122513 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:14:43.123233 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:15:43.121769 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:15:43.122484 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:16:43.131812 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:16:43.132395 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:17:43.130414 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:17:43.131747 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:18:43.132220 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:18:43.132853 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:19:43.133369 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:19:43.138375 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:20:43.135738 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:20:43.136350 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:21:43.134261 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:21:43.134896 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:22:43.131026 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:22:43.131725 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:23:43.136844 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:23:43.138302 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:24:43.136325 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:24:43.137152 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:25:43.132155 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:25:43.133078 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:26:43.136855 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:26:43.137605 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:27:43.136801 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:27:43.141197 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:28:43.144747 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:28:43.144747 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:29:43.136863 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:29:43.147533 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:30:43.133588 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:30:43.134739 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:31:43.132012 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:31:43.132551 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:32:43.128210 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:32:43.129468 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:33:17.729591 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] Create backup started, backup: 5889f359-d4ac-4426-8a9a-ea8f6b580c42 volume: 418b0ee7-9825-409e-a27f-ab8034c2c05f. Dec 25 21:33:17.824900 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.utils [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.32', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:33:17.828791 ncdv-master-1 cinder-backup[129839]: INFO oslo.privsep.daemon [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] Running privsep helper: ['sudo', 'cinder-rootwrap', '/etc/cinder/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/cinder/cinder.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmpidx8pm8i/privsep.sock'] Dec 25 21:33:17.882272 ncdv-master-1 sudo[161897]: ubuntu : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf privsep-helper --config-file /etc/cinder/cinder.conf --privsep_context os_brick.privileged.default --privsep_sock_path /tmp/tmpidx8pm8i/privsep.sock Dec 25 21:33:17.889522 ncdv-master-1 sudo[161897]: pam_unix(sudo:session): session opened for user root by (uid=0) Dec 25 21:33:19.928388 ncdv-master-1 sudo[161897]: pam_unix(sudo:session): session closed for user root Dec 25 21:33:19.939609 ncdv-master-1 cinder-backup[129839]: INFO oslo.privsep.daemon [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] Spawned new privsep daemon via rootwrap Dec 25 21:33:19.945124 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] Accepted privsep connection to /tmp/tmpidx8pm8i/privsep.sock {{(pid=129839) __init__ /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:363}} Dec 25 21:33:19.945124 ncdv-master-1 cinder-backup[129839]: INFO oslo.privsep.daemon [-] privsep daemon starting Dec 25 21:33:19.945124 ncdv-master-1 cinder-backup[129839]: INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 Dec 25 21:33:19.945124 ncdv-master-1 cinder-backup[129839]: INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none Dec 25 21:33:19.945124 ncdv-master-1 cinder-backup[129839]: INFO oslo.privsep.daemon [-] privsep daemon running as pid 161909 Dec 25 21:33:19.949470 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[b788c938-7635-4dfb-8f24-34af53e760bb]: (2,) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:33:20.321817 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:33:20.348744 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.027s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:33:20.349615 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[3e3bef6a-e20e-4d9e-96c2-5319165df541]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:c2a322773d0\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:33:20.351475 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:33:20.360159 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Dec 25 21:33:20.377744 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[833954fc-3b05-424e-a16e-13f0c435defc]: [Errno 2] No such file or directory: 'systool' {{(pid=161909) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}} Dec 25 21:33:20.377744 ncdv-master-1 cinder-backup[129839]: Traceback (most recent call last): Dec 25 21:33:20.377744 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd Dec 25 21:33:20.377744 ncdv-master-1 cinder-backup[129839]: ret = func(*f_args, **f_kwargs) Dec 25 21:33:20.377744 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Dec 25 21:33:20.377744 ncdv-master-1 cinder-backup[129839]: return func(*args, **kwargs) Dec 25 21:33:20.377744 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Dec 25 21:33:20.377744 ncdv-master-1 cinder-backup[129839]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Dec 25 21:33:20.377744 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Dec 25 21:33:20.377744 ncdv-master-1 cinder-backup[129839]: return putils.execute(on_execute=on_execute, Dec 25 21:33:20.377744 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Dec 25 21:33:20.377744 ncdv-master-1 cinder-backup[129839]: obj = subprocess.Popen(cmd, Dec 25 21:33:20.377744 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Dec 25 21:33:20.377744 ncdv-master-1 cinder-backup[129839]: self._execute_child(args, executable, preexec_fn, close_fds, Dec 25 21:33:20.377744 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Dec 25 21:33:20.377744 ncdv-master-1 cinder-backup[129839]: raise child_exception_type(errno_num, err_msg, err_filename) Dec 25 21:33:20.377744 ncdv-master-1 cinder-backup[129839]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Dec 25 21:33:20.378966 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[833954fc-3b05-424e-a16e-13f0c435defc]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:33:20.379656 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:33:20.386558 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Dec 25 21:33:20.387404 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[6b7b0e4b-783f-4bf2-a202-8600ca53a78a]: [Errno 2] No such file or directory: 'systool' {{(pid=161909) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}} Dec 25 21:33:20.387404 ncdv-master-1 cinder-backup[129839]: Traceback (most recent call last): Dec 25 21:33:20.387404 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd Dec 25 21:33:20.387404 ncdv-master-1 cinder-backup[129839]: ret = func(*f_args, **f_kwargs) Dec 25 21:33:20.387404 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Dec 25 21:33:20.387404 ncdv-master-1 cinder-backup[129839]: return func(*args, **kwargs) Dec 25 21:33:20.387404 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Dec 25 21:33:20.387404 ncdv-master-1 cinder-backup[129839]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Dec 25 21:33:20.387404 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Dec 25 21:33:20.387404 ncdv-master-1 cinder-backup[129839]: return putils.execute(on_execute=on_execute, Dec 25 21:33:20.387404 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Dec 25 21:33:20.387404 ncdv-master-1 cinder-backup[129839]: obj = subprocess.Popen(cmd, Dec 25 21:33:20.387404 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Dec 25 21:33:20.387404 ncdv-master-1 cinder-backup[129839]: self._execute_child(args, executable, preexec_fn, close_fds, Dec 25 21:33:20.387404 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Dec 25 21:33:20.387404 ncdv-master-1 cinder-backup[129839]: raise child_exception_type(errno_num, err_msg, err_filename) Dec 25 21:33:20.387404 ncdv-master-1 cinder-backup[129839]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Dec 25 21:33:20.391255 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[6b7b0e4b-783f-4bf2-a202-8600ca53a78a]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:33:20.391255 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:33:20.460910 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.070s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:33:20.461295 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[f639b821-331b-4beb-bc59-8dd68684fc33]: (4, ('/dev/sda1\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:33:20.462499 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:33:20.512980 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.050s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:33:20.513300 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[79b9a37b-35ea-4b5d-a231-bb2b2e40013d]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:33:20.514913 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:33:20.522286 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.007s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:33:20.525042 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[27258b8b-0f14-4872-a2eb-1f7c403947ee]: (4, ('0b661f24-87f6-9248-8553-e9ec840a31d0\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:33:20.525042 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] Running cmd (subprocess): nvme version {{(pid=129839) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:33:20.566329 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] CMD "nvme version" returned: 0 in 0.042s {{(pid=129839) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:33:20.571445 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.utils [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] <== get_connector_properties: return (2746ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.32', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:c2a322773d0', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '0b661f24-87f6-9248-8553-e9ec840a31d0', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:68cc0364-fba7-439f-bcd5-9f4beb715fab'} {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:33:21.689530 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connector [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] Factory for rbd on None {{(pid=129839) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}} Dec 25 21:33:21.705326 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-418b0ee7-9825-409e-a27f-ab8034c2c05f', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '418b0ee7-9825-409e-a27f-ab8034c2c05f', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:33:21.705326 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.linuxrbd [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] opening connection to ceph cluster (timeout=-1). {{(pid=129839) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Dec 25 21:33:21.951435 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] <== connect_volume: return (254ms) {'path': } {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:33:21.961392 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_db.sqlalchemy.engines [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129839) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 25 21:33:22.089918 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] Starting backup of volume='418b0ee7-9825-409e-a27f-ab8034c2c05f'. {{(pid=129839) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Dec 25 21:33:22.090907 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] Volume file is RBD: attempting optimized backup {{(pid=129839) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Dec 25 21:33:22.121522 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] Creating base image 'volume-418b0ee7-9825-409e-a27f-ab8034c2c05f.backup.5889f359-d4ac-4426-8a9a-ea8f6b580c42' {{(pid=129839) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Dec 25 21:33:22.122083 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129839) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Dec 25 21:33:25.937840 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] Using --from-snap 'None' for incremental backup of volume 418b0ee7-9825-409e-a27f-ab8034c2c05f. {{(pid=129839) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Dec 25 21:33:25.939776 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] Creating backup snapshot='backup.5889f359-d4ac-4426-8a9a-ea8f6b580c42.snap.1640468005.9393804' {{(pid=129839) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Dec 25 21:33:28.575458 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] Performing differential transfer from 'volume-418b0ee7-9825-409e-a27f-ab8034c2c05f' to 'volume-418b0ee7-9825-409e-a27f-ab8034c2c05f.backup.5889f359-d4ac-4426-8a9a-ea8f6b580c42' {{(pid=129839) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Dec 25 21:33:28.576070 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_qndsti3o --pool volumes volumes/volume-418b0ee7-9825-409e-a27f-ab8034c2c05f@backup.5889f359-d4ac-4426-8a9a-ea8f6b580c42.snap.1640468005.9393804 -' into... {{(pid=129839) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Dec 25 21:33:28.576289 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-418b0ee7-9825-409e-a27f-ab8034c2c05f.backup.5889f359-d4ac-4426-8a9a-ea8f6b580c42' {{(pid=129839) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Dec 25 21:33:32.512360 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] Differential backup transfer completed in 3.9363s {{(pid=129839) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Dec 25 21:33:32.514732 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] Getting metadata type 'volume-base-metadata' {{(pid=129839) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Dec 25 21:33:32.526763 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129839) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Dec 25 21:33:32.527168 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] Getting metadata type 'volume-metadata' {{(pid=129839) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Dec 25 21:33:32.538517 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] No metadata type 'volume-metadata' available {{(pid=129839) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Dec 25 21:33:32.543360 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] Getting metadata type 'volume-glance-metadata' {{(pid=129839) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Dec 25 21:33:32.547585 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] No metadata type 'volume-glance-metadata' available {{(pid=129839) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Dec 25 21:33:32.548157 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] Backing up metadata for volume 418b0ee7-9825-409e-a27f-ab8034c2c05f. {{(pid=129839) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Dec 25 21:33:32.841260 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] Backup '5889f359-d4ac-4426-8a9a-ea8f6b580c42' of volume 418b0ee7-9825-409e-a27f-ab8034c2c05f finished. {{(pid=129839) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Dec 25 21:33:32.842807 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-418b0ee7-9825-409e-a27f-ab8034c2c05f', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '418b0ee7-9825-409e-a27f-ab8034c2c05f', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, 'ignore_errors': True}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:33:33.559755 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] <== disconnect_volume: return (716ms) None {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:33:33.911931 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-8826072d-9c7d-40e0-8d21-863b3def84e5 tempest-VolumesBackupsV39Test-1766595062 None] Create backup finished. backup: 5889f359-d4ac-4426-8a9a-ea8f6b580c42. Dec 25 21:33:34.820360 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-3fb7f035-78a5-4eb3-b971-3a3e0d882a39 tempest-VolumesBackupsV39Test-1766595062 None] Delete backup started, backup: 5889f359-d4ac-4426-8a9a-ea8f6b580c42. Dec 25 21:33:34.822342 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-3fb7f035-78a5-4eb3-b971-3a3e0d882a39 tempest-VolumesBackupsV39Test-1766595062 None] Delete started for backup=5889f359-d4ac-4426-8a9a-ea8f6b580c42 {{(pid=129839) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Dec 25 21:33:34.823267 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-3fb7f035-78a5-4eb3-b971-3a3e0d882a39 tempest-VolumesBackupsV39Test-1766595062 None] Trying diff format basename='volume-418b0ee7-9825-409e-a27f-ab8034c2c05f.backup.5889f359-d4ac-4426-8a9a-ea8f6b580c42' for backup base image of volume 418b0ee7-9825-409e-a27f-ab8034c2c05f. {{(pid=129839) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Dec 25 21:33:35.238872 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-3fb7f035-78a5-4eb3-b971-3a3e0d882a39 tempest-VolumesBackupsV39Test-1766595062 None] Looking for snapshot of backup base 'volume-418b0ee7-9825-409e-a27f-ab8034c2c05f.backup.5889f359-d4ac-4426-8a9a-ea8f6b580c42' {{(pid=129839) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Dec 25 21:33:35.239251 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-3fb7f035-78a5-4eb3-b971-3a3e0d882a39 tempest-VolumesBackupsV39Test-1766595062 None] Found snapshot 'backup.5889f359-d4ac-4426-8a9a-ea8f6b580c42.snap.1640468005.9393804' {{(pid=129839) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Dec 25 21:33:35.239659 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-3fb7f035-78a5-4eb3-b971-3a3e0d882a39 tempest-VolumesBackupsV39Test-1766595062 None] Deleting backup snapshot='backup.5889f359-d4ac-4426-8a9a-ea8f6b580c42.snap.1640468005.9393804' {{(pid=129839) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Dec 25 21:33:37.248827 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_db.sqlalchemy.engines [None req-9b4bfa17-f8eb-43dd-9d64-d05efc7c4b62 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129839) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 25 21:33:39.568314 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.drivers.ceph [None req-3fb7f035-78a5-4eb3-b971-3a3e0d882a39 tempest-VolumesBackupsV39Test-1766595062 None] Deleting backup base image='volume-418b0ee7-9825-409e-a27f-ab8034c2c05f.backup.5889f359-d4ac-4426-8a9a-ea8f6b580c42' of volume 418b0ee7-9825-409e-a27f-ab8034c2c05f. Dec 25 21:33:43.124787 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:33:43.126460 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:33:50.203467 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-3fb7f035-78a5-4eb3-b971-3a3e0d882a39 tempest-VolumesBackupsV39Test-1766595062 None] Base backup image='volume-418b0ee7-9825-409e-a27f-ab8034c2c05f.backup.5889f359-d4ac-4426-8a9a-ea8f6b580c42' of volume 418b0ee7-9825-409e-a27f-ab8034c2c05f deleted. {{(pid=129839) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Dec 25 21:33:50.459271 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-3fb7f035-78a5-4eb3-b971-3a3e0d882a39 tempest-VolumesBackupsV39Test-1766595062 None] Delete of backup '5889f359-d4ac-4426-8a9a-ea8f6b580c42' for volume '418b0ee7-9825-409e-a27f-ab8034c2c05f' finished. {{(pid=129839) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Dec 25 21:33:50.683315 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.quota [None req-3fb7f035-78a5-4eb3-b971-3a3e0d882a39 tempest-VolumesBackupsV39Test-1766595062 None] Created reservations ['5252efd2-de05-4d14-bce5-69da96f4a05d', '0b465788-3571-414c-8e8b-3dd48964fa38'] {{(pid=129839) reserve /opt/stack/cinder/cinder/quota.py:755}} Dec 25 21:33:50.912633 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-3fb7f035-78a5-4eb3-b971-3a3e0d882a39 tempest-VolumesBackupsV39Test-1766595062 None] Delete backup finished, backup 5889f359-d4ac-4426-8a9a-ea8f6b580c42 deleted. Dec 25 21:34:37.309880 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] Create backup started, backup: caf4f268-3a04-4b21-814f-e0e2e5ca3d4f volume: f03e9745-84e8-42ce-aae1-83a861dbd629 snapshot: fa65ec2d-542f-4a76-8b2c-0f05d2f40f68. Dec 25 21:34:37.473495 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.utils [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.32', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:34:37.474938 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:34:37.481952 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.007s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:34:37.482281 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[56a75759-d9ab-406b-baf7-29467e10c989]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:c2a322773d0\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:34:37.483495 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:34:37.489384 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Dec 25 21:34:37.490095 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[0ec02f26-1094-4b42-ab26-2cd1f5030952]: [Errno 2] No such file or directory: 'systool' {{(pid=161909) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}} Dec 25 21:34:37.490095 ncdv-master-1 cinder-backup[129839]: Traceback (most recent call last): Dec 25 21:34:37.490095 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd Dec 25 21:34:37.490095 ncdv-master-1 cinder-backup[129839]: ret = func(*f_args, **f_kwargs) Dec 25 21:34:37.490095 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Dec 25 21:34:37.490095 ncdv-master-1 cinder-backup[129839]: return func(*args, **kwargs) Dec 25 21:34:37.490095 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Dec 25 21:34:37.490095 ncdv-master-1 cinder-backup[129839]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Dec 25 21:34:37.490095 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Dec 25 21:34:37.490095 ncdv-master-1 cinder-backup[129839]: return putils.execute(on_execute=on_execute, Dec 25 21:34:37.490095 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Dec 25 21:34:37.490095 ncdv-master-1 cinder-backup[129839]: obj = subprocess.Popen(cmd, Dec 25 21:34:37.490095 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Dec 25 21:34:37.490095 ncdv-master-1 cinder-backup[129839]: self._execute_child(args, executable, preexec_fn, close_fds, Dec 25 21:34:37.490095 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Dec 25 21:34:37.490095 ncdv-master-1 cinder-backup[129839]: raise child_exception_type(errno_num, err_msg, err_filename) Dec 25 21:34:37.490095 ncdv-master-1 cinder-backup[129839]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Dec 25 21:34:37.491380 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[0ec02f26-1094-4b42-ab26-2cd1f5030952]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:34:37.491727 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:34:37.497113 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Dec 25 21:34:37.497732 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[36737388-fa7e-4ec5-8839-710ec4e1a8a6]: [Errno 2] No such file or directory: 'systool' {{(pid=161909) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}} Dec 25 21:34:37.497732 ncdv-master-1 cinder-backup[129839]: Traceback (most recent call last): Dec 25 21:34:37.497732 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd Dec 25 21:34:37.497732 ncdv-master-1 cinder-backup[129839]: ret = func(*f_args, **f_kwargs) Dec 25 21:34:37.497732 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Dec 25 21:34:37.497732 ncdv-master-1 cinder-backup[129839]: return func(*args, **kwargs) Dec 25 21:34:37.497732 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Dec 25 21:34:37.497732 ncdv-master-1 cinder-backup[129839]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Dec 25 21:34:37.497732 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Dec 25 21:34:37.497732 ncdv-master-1 cinder-backup[129839]: return putils.execute(on_execute=on_execute, Dec 25 21:34:37.497732 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Dec 25 21:34:37.497732 ncdv-master-1 cinder-backup[129839]: obj = subprocess.Popen(cmd, Dec 25 21:34:37.497732 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Dec 25 21:34:37.497732 ncdv-master-1 cinder-backup[129839]: self._execute_child(args, executable, preexec_fn, close_fds, Dec 25 21:34:37.497732 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Dec 25 21:34:37.497732 ncdv-master-1 cinder-backup[129839]: raise child_exception_type(errno_num, err_msg, err_filename) Dec 25 21:34:37.497732 ncdv-master-1 cinder-backup[129839]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Dec 25 21:34:37.499904 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[36737388-fa7e-4ec5-8839-710ec4e1a8a6]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:34:37.499904 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:34:37.512475 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.013s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:34:37.512837 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[3f2dde1d-b89b-44c3-922d-f26e389d687b]: (4, ('/dev/sda1\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:34:37.514003 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:34:37.526320 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.012s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:34:37.527148 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[390987cf-5a12-4ea8-a987-938fa035b11a]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:34:37.528709 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:34:37.542117 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.013s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:34:37.542965 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[2ac7d8ff-6812-410e-b775-5789e404b9ca]: (4, ('0b661f24-87f6-9248-8553-e9ec840a31d0\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:34:37.545214 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] Running cmd (subprocess): nvme version {{(pid=129839) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:34:37.559590 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] CMD "nvme version" returned: 0 in 0.016s {{(pid=129839) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:34:37.560635 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.utils [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] <== get_connector_properties: return (86ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.32', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:c2a322773d0', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '0b661f24-87f6-9248-8553-e9ec840a31d0', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:68cc0364-fba7-439f-bcd5-9f4beb715fab'} {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:34:41.657375 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connector [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] Factory for rbd on None {{(pid=129839) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}} Dec 25 21:34:41.658787 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-69fd46b7-dacb-409d-88c5-27d9657b39f5', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '69fd46b7-dacb-409d-88c5-27d9657b39f5', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:34:41.659815 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.linuxrbd [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] opening connection to ceph cluster (timeout=-1). {{(pid=129839) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Dec 25 21:34:41.938982 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] <== connect_volume: return (279ms) {'path': } {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:34:41.947413 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_db.sqlalchemy.engines [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129839) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 25 21:34:42.032383 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] Starting backup of volume='f03e9745-84e8-42ce-aae1-83a861dbd629'. {{(pid=129839) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Dec 25 21:34:42.193209 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129839) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Dec 25 21:34:42.193603 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] Creating backup base image='volume-f03e9745-84e8-42ce-aae1-83a861dbd629.backup.base' for volume f03e9745-84e8-42ce-aae1-83a861dbd629. {{(pid=129839) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:830}} Dec 25 21:34:43.122216 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:34:43.122861 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:34:43.601840 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] Copying data from volume f03e9745-84e8-42ce-aae1-83a861dbd629. {{(pid=129839) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:842}} Dec 25 21:34:43.884061 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] Transferring data between 'volume-f03e9745-84e8-42ce-aae1-83a861dbd629' and 'volume-f03e9745-84e8-42ce-aae1-83a861dbd629.backup.base' {{(pid=129839) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:399}} Dec 25 21:34:43.884725 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] 8 chunks of 134217728 bytes to be transferred {{(pid=129839) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:403}} Dec 25 21:34:57.575626 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] Transferred chunk 1 of 8 (9574K/s) {{(pid=129839) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Dec 25 21:35:08.975471 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] Transferred chunk 2 of 8 (11499K/s) {{(pid=129839) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Dec 25 21:35:20.745425 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] Transferred chunk 3 of 8 (11136K/s) {{(pid=129839) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Dec 25 21:35:32.640973 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] Transferred chunk 4 of 8 (11019K/s) {{(pid=129839) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Dec 25 21:35:43.118701 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:35:43.119343 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:35:44.226675 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] Transferred chunk 5 of 8 (11313K/s) {{(pid=129839) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Dec 25 21:35:56.998483 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] Transferred chunk 6 of 8 (10263K/s) {{(pid=129839) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Dec 25 21:36:09.617616 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] Transferred chunk 7 of 8 (10387K/s) {{(pid=129839) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Dec 25 21:36:20.354271 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] Transferred chunk 8 of 8 (12208K/s) {{(pid=129839) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:422}} Dec 25 21:36:21.158983 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] Getting metadata type 'volume-base-metadata' {{(pid=129839) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Dec 25 21:36:21.169898 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129839) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Dec 25 21:36:21.170308 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] Getting metadata type 'volume-metadata' {{(pid=129839) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Dec 25 21:36:21.186353 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] No metadata type 'volume-metadata' available {{(pid=129839) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Dec 25 21:36:21.186854 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] Getting metadata type 'volume-glance-metadata' {{(pid=129839) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Dec 25 21:36:21.193336 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] No metadata type 'volume-glance-metadata' available {{(pid=129839) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Dec 25 21:36:21.193902 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] Backing up metadata for volume f03e9745-84e8-42ce-aae1-83a861dbd629. {{(pid=129839) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Dec 25 21:36:21.383565 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] Backup 'caf4f268-3a04-4b21-814f-e0e2e5ca3d4f' of volume f03e9745-84e8-42ce-aae1-83a861dbd629 finished. {{(pid=129839) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Dec 25 21:36:21.385548 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-69fd46b7-dacb-409d-88c5-27d9657b39f5', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '69fd46b7-dacb-409d-88c5-27d9657b39f5', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, 'ignore_errors': True}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:36:21.709727 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] <== disconnect_volume: return (323ms) None {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:36:24.809526 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-1125206f-c370-4113-9b5e-cc680b2bd33a tempest-VolumesSnapshotTestJSON-639273304 None] Create backup finished. backup: caf4f268-3a04-4b21-814f-e0e2e5ca3d4f. Dec 25 21:36:25.769967 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-79672ec2-d521-41f1-97aa-6d3946847c1d tempest-VolumesSnapshotTestJSON-639273304 None] Delete backup started, backup: caf4f268-3a04-4b21-814f-e0e2e5ca3d4f. Dec 25 21:36:25.771974 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-79672ec2-d521-41f1-97aa-6d3946847c1d tempest-VolumesSnapshotTestJSON-639273304 None] Delete started for backup=caf4f268-3a04-4b21-814f-e0e2e5ca3d4f {{(pid=129839) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Dec 25 21:36:25.772508 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-79672ec2-d521-41f1-97aa-6d3946847c1d tempest-VolumesSnapshotTestJSON-639273304 None] Trying diff format basename='volume-f03e9745-84e8-42ce-aae1-83a861dbd629.backup.caf4f268-3a04-4b21-814f-e0e2e5ca3d4f' for backup base image of volume f03e9745-84e8-42ce-aae1-83a861dbd629. {{(pid=129839) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Dec 25 21:36:25.808397 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-79672ec2-d521-41f1-97aa-6d3946847c1d tempest-VolumesSnapshotTestJSON-639273304 None] Image 'volume-f03e9745-84e8-42ce-aae1-83a861dbd629.backup.caf4f268-3a04-4b21-814f-e0e2e5ca3d4f' not found - trying diff format name {{(pid=129839) _rbd_image_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:661}} Dec 25 21:36:25.975936 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-79672ec2-d521-41f1-97aa-6d3946847c1d tempest-VolumesSnapshotTestJSON-639273304 None] Looking for snapshot of backup base 'volume-f03e9745-84e8-42ce-aae1-83a861dbd629.backup.base' {{(pid=129839) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Dec 25 21:36:25.976733 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-79672ec2-d521-41f1-97aa-6d3946847c1d tempest-VolumesSnapshotTestJSON-639273304 None] Backup base 'volume-f03e9745-84e8-42ce-aae1-83a861dbd629.backup.base' has no snapshots {{(pid=129839) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:908}} Dec 25 21:36:25.977220 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-79672ec2-d521-41f1-97aa-6d3946847c1d tempest-VolumesSnapshotTestJSON-639273304 None] No backup snapshot to delete {{(pid=129839) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:476}} Dec 25 21:36:26.216409 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.drivers.ceph [None req-79672ec2-d521-41f1-97aa-6d3946847c1d tempest-VolumesSnapshotTestJSON-639273304 None] Deleting backup base image='volume-f03e9745-84e8-42ce-aae1-83a861dbd629.backup.base' of volume f03e9745-84e8-42ce-aae1-83a861dbd629. Dec 25 21:36:27.270257 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_db.sqlalchemy.engines [None req-9b4bfa17-f8eb-43dd-9d64-d05efc7c4b62 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129839) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 25 21:36:42.291593 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-79672ec2-d521-41f1-97aa-6d3946847c1d tempest-VolumesSnapshotTestJSON-639273304 None] Base backup image='volume-f03e9745-84e8-42ce-aae1-83a861dbd629.backup.base' of volume f03e9745-84e8-42ce-aae1-83a861dbd629 deleted. {{(pid=129839) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Dec 25 21:36:42.463664 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-79672ec2-d521-41f1-97aa-6d3946847c1d tempest-VolumesSnapshotTestJSON-639273304 None] Delete of backup 'caf4f268-3a04-4b21-814f-e0e2e5ca3d4f' for volume 'f03e9745-84e8-42ce-aae1-83a861dbd629' finished. {{(pid=129839) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Dec 25 21:36:42.541815 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.quota [None req-79672ec2-d521-41f1-97aa-6d3946847c1d tempest-VolumesSnapshotTestJSON-639273304 None] Created reservations ['060dffac-8ed3-42cb-863c-92858d717027', 'e5d417a9-470b-4814-8c41-0fe3889f01d2'] {{(pid=129839) reserve /opt/stack/cinder/cinder/quota.py:755}} Dec 25 21:36:42.730557 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-79672ec2-d521-41f1-97aa-6d3946847c1d tempest-VolumesSnapshotTestJSON-639273304 None] Delete backup finished, backup caf4f268-3a04-4b21-814f-e0e2e5ca3d4f deleted. Dec 25 21:36:43.114306 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:36:43.114631 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:37:43.120032 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:37:43.121589 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:38:43.118310 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:38:43.119034 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:39:20.530001 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] Create backup started, backup: 7c3cc723-7f3f-422b-b14b-8ba52dfd371f volume: 26e57931-5e2e-498f-907c-ba032fee0597. Dec 25 21:39:20.617470 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.utils [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.32', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:39:20.619062 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:39:20.630058 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.009s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:39:20.630553 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[cc60807e-39e7-4412-a851-20dd4e98294a]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:c2a322773d0\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:39:20.643985 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:39:20.665651 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Dec 25 21:39:20.666089 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[5a31227b-dcfa-4075-8fa9-5feee5ea7e59]: [Errno 2] No such file or directory: 'systool' {{(pid=161909) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}} Dec 25 21:39:20.666089 ncdv-master-1 cinder-backup[129839]: Traceback (most recent call last): Dec 25 21:39:20.666089 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd Dec 25 21:39:20.666089 ncdv-master-1 cinder-backup[129839]: ret = func(*f_args, **f_kwargs) Dec 25 21:39:20.666089 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Dec 25 21:39:20.666089 ncdv-master-1 cinder-backup[129839]: return func(*args, **kwargs) Dec 25 21:39:20.666089 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Dec 25 21:39:20.666089 ncdv-master-1 cinder-backup[129839]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Dec 25 21:39:20.666089 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Dec 25 21:39:20.666089 ncdv-master-1 cinder-backup[129839]: return putils.execute(on_execute=on_execute, Dec 25 21:39:20.666089 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Dec 25 21:39:20.666089 ncdv-master-1 cinder-backup[129839]: obj = subprocess.Popen(cmd, Dec 25 21:39:20.666089 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Dec 25 21:39:20.666089 ncdv-master-1 cinder-backup[129839]: self._execute_child(args, executable, preexec_fn, close_fds, Dec 25 21:39:20.666089 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Dec 25 21:39:20.666089 ncdv-master-1 cinder-backup[129839]: raise child_exception_type(errno_num, err_msg, err_filename) Dec 25 21:39:20.666089 ncdv-master-1 cinder-backup[129839]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Dec 25 21:39:20.666886 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[5a31227b-dcfa-4075-8fa9-5feee5ea7e59]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:39:20.670513 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:39:20.679825 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Dec 25 21:39:20.680403 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[b27c0f09-c0bb-47ca-b123-ff2632e0e315]: [Errno 2] No such file or directory: 'systool' {{(pid=161909) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}} Dec 25 21:39:20.680403 ncdv-master-1 cinder-backup[129839]: Traceback (most recent call last): Dec 25 21:39:20.680403 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd Dec 25 21:39:20.680403 ncdv-master-1 cinder-backup[129839]: ret = func(*f_args, **f_kwargs) Dec 25 21:39:20.680403 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Dec 25 21:39:20.680403 ncdv-master-1 cinder-backup[129839]: return func(*args, **kwargs) Dec 25 21:39:20.680403 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Dec 25 21:39:20.680403 ncdv-master-1 cinder-backup[129839]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Dec 25 21:39:20.680403 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Dec 25 21:39:20.680403 ncdv-master-1 cinder-backup[129839]: return putils.execute(on_execute=on_execute, Dec 25 21:39:20.680403 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Dec 25 21:39:20.680403 ncdv-master-1 cinder-backup[129839]: obj = subprocess.Popen(cmd, Dec 25 21:39:20.680403 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Dec 25 21:39:20.680403 ncdv-master-1 cinder-backup[129839]: self._execute_child(args, executable, preexec_fn, close_fds, Dec 25 21:39:20.680403 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Dec 25 21:39:20.680403 ncdv-master-1 cinder-backup[129839]: raise child_exception_type(errno_num, err_msg, err_filename) Dec 25 21:39:20.680403 ncdv-master-1 cinder-backup[129839]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Dec 25 21:39:20.682645 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[b27c0f09-c0bb-47ca-b123-ff2632e0e315]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:39:20.684091 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:39:20.701425 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.017s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:39:20.701773 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[aee64bf8-ef83-4873-afca-a890e7281efc]: (4, ('/dev/sda1\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:39:20.707280 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:39:20.715158 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.012s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:39:20.715472 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[d313dff0-caba-4dbf-a332-80aee4556247]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:39:20.716700 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:39:20.729614 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.012s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:39:20.729974 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[96f78272-8b3a-43fe-8cfb-246035b74402]: (4, ('0b661f24-87f6-9248-8553-e9ec840a31d0\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:39:20.731120 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] Running cmd (subprocess): nvme version {{(pid=129839) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:39:20.757095 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] CMD "nvme version" returned: 0 in 0.026s {{(pid=129839) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:39:20.758393 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.utils [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] <== get_connector_properties: return (140ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.32', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:c2a322773d0', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '0b661f24-87f6-9248-8553-e9ec840a31d0', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:68cc0364-fba7-439f-bcd5-9f4beb715fab'} {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:39:21.666250 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connector [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] Factory for rbd on None {{(pid=129839) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}} Dec 25 21:39:21.667963 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-26e57931-5e2e-498f-907c-ba032fee0597', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '26e57931-5e2e-498f-907c-ba032fee0597', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:39:21.669024 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.linuxrbd [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] opening connection to ceph cluster (timeout=-1). {{(pid=129839) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Dec 25 21:39:22.089543 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] <== connect_volume: return (421ms) {'path': } {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:39:22.098043 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_db.sqlalchemy.engines [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129839) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 25 21:39:22.188161 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] Starting backup of volume='26e57931-5e2e-498f-907c-ba032fee0597'. {{(pid=129839) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Dec 25 21:39:22.189478 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] Volume file is RBD: attempting optimized backup {{(pid=129839) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Dec 25 21:39:22.232543 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] Creating base image 'volume-26e57931-5e2e-498f-907c-ba032fee0597.backup.7c3cc723-7f3f-422b-b14b-8ba52dfd371f' {{(pid=129839) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Dec 25 21:39:22.233225 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129839) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Dec 25 21:39:23.057226 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] Using --from-snap 'None' for incremental backup of volume 26e57931-5e2e-498f-907c-ba032fee0597. {{(pid=129839) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Dec 25 21:39:23.057882 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] Creating backup snapshot='backup.7c3cc723-7f3f-422b-b14b-8ba52dfd371f.snap.1640468363.0572503' {{(pid=129839) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Dec 25 21:39:25.861559 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] Performing differential transfer from 'volume-26e57931-5e2e-498f-907c-ba032fee0597' to 'volume-26e57931-5e2e-498f-907c-ba032fee0597.backup.7c3cc723-7f3f-422b-b14b-8ba52dfd371f' {{(pid=129839) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Dec 25 21:39:25.862157 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_kwlk3y_c --pool volumes volumes/volume-26e57931-5e2e-498f-907c-ba032fee0597@backup.7c3cc723-7f3f-422b-b14b-8ba52dfd371f.snap.1640468363.0572503 -' into... {{(pid=129839) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Dec 25 21:39:25.862656 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-26e57931-5e2e-498f-907c-ba032fee0597.backup.7c3cc723-7f3f-422b-b14b-8ba52dfd371f' {{(pid=129839) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Dec 25 21:39:43.115733 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:39:43.116369 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:40:16.903381 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] Create backup started, backup: 996a92c5-2024-48b9-a849-2d57b581a05d volume: 2e560f62-0022-4854-a99f-ce9ce208b3c1. Dec 25 21:40:16.982109 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.utils [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.32', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:40:16.983564 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:40:16.990676 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.007s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:40:16.991373 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[ddb8228f-b5b7-422d-aca7-16e02b6797fe]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:c2a322773d0\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:40:16.992307 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:40:16.997960 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Dec 25 21:40:16.998622 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[523866fb-f8b4-4773-87a0-a880dae47410]: [Errno 2] No such file or directory: 'systool' {{(pid=161909) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}} Dec 25 21:40:16.998622 ncdv-master-1 cinder-backup[129839]: Traceback (most recent call last): Dec 25 21:40:16.998622 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd Dec 25 21:40:16.998622 ncdv-master-1 cinder-backup[129839]: ret = func(*f_args, **f_kwargs) Dec 25 21:40:16.998622 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Dec 25 21:40:16.998622 ncdv-master-1 cinder-backup[129839]: return func(*args, **kwargs) Dec 25 21:40:16.998622 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Dec 25 21:40:16.998622 ncdv-master-1 cinder-backup[129839]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Dec 25 21:40:16.998622 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Dec 25 21:40:16.998622 ncdv-master-1 cinder-backup[129839]: return putils.execute(on_execute=on_execute, Dec 25 21:40:16.998622 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Dec 25 21:40:16.998622 ncdv-master-1 cinder-backup[129839]: obj = subprocess.Popen(cmd, Dec 25 21:40:16.998622 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Dec 25 21:40:16.998622 ncdv-master-1 cinder-backup[129839]: self._execute_child(args, executable, preexec_fn, close_fds, Dec 25 21:40:16.998622 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Dec 25 21:40:16.998622 ncdv-master-1 cinder-backup[129839]: raise child_exception_type(errno_num, err_msg, err_filename) Dec 25 21:40:16.998622 ncdv-master-1 cinder-backup[129839]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Dec 25 21:40:16.999644 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[523866fb-f8b4-4773-87a0-a880dae47410]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:40:17.000078 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:40:17.005494 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Dec 25 21:40:17.006079 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[a138c5a6-3d57-45a1-87b1-458ccc1b0de0]: [Errno 2] No such file or directory: 'systool' {{(pid=161909) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}} Dec 25 21:40:17.006079 ncdv-master-1 cinder-backup[129839]: Traceback (most recent call last): Dec 25 21:40:17.006079 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd Dec 25 21:40:17.006079 ncdv-master-1 cinder-backup[129839]: ret = func(*f_args, **f_kwargs) Dec 25 21:40:17.006079 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Dec 25 21:40:17.006079 ncdv-master-1 cinder-backup[129839]: return func(*args, **kwargs) Dec 25 21:40:17.006079 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Dec 25 21:40:17.006079 ncdv-master-1 cinder-backup[129839]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Dec 25 21:40:17.006079 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Dec 25 21:40:17.006079 ncdv-master-1 cinder-backup[129839]: return putils.execute(on_execute=on_execute, Dec 25 21:40:17.006079 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Dec 25 21:40:17.006079 ncdv-master-1 cinder-backup[129839]: obj = subprocess.Popen(cmd, Dec 25 21:40:17.006079 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Dec 25 21:40:17.006079 ncdv-master-1 cinder-backup[129839]: self._execute_child(args, executable, preexec_fn, close_fds, Dec 25 21:40:17.006079 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Dec 25 21:40:17.006079 ncdv-master-1 cinder-backup[129839]: raise child_exception_type(errno_num, err_msg, err_filename) Dec 25 21:40:17.006079 ncdv-master-1 cinder-backup[129839]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Dec 25 21:40:17.006973 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[a138c5a6-3d57-45a1-87b1-458ccc1b0de0]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:40:17.007807 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:40:17.016673 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.009s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:40:17.017218 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[4b8cd39d-62bf-4482-ae6f-bcbf3ad48f68]: (4, ('/dev/sda1\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:40:17.018743 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:40:17.027782 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.009s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:40:17.028121 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[ba7dba15-1dd3-43e2-8476-6e78c2a33963]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:40:17.029246 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:40:17.035489 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.006s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:40:17.035837 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[ea60f18c-d365-4e5e-bef2-ef8c5ec8b882]: (4, ('0b661f24-87f6-9248-8553-e9ec840a31d0\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:40:17.036639 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] Running cmd (subprocess): nvme version {{(pid=129839) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:40:17.053497 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] CMD "nvme version" returned: 0 in 0.016s {{(pid=129839) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:40:17.054563 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.utils [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] <== get_connector_properties: return (71ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.32', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:c2a322773d0', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '0b661f24-87f6-9248-8553-e9ec840a31d0', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:68cc0364-fba7-439f-bcd5-9f4beb715fab'} {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:40:17.291157 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_db.sqlalchemy.engines [None req-9b4bfa17-f8eb-43dd-9d64-d05efc7c4b62 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129839) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 25 21:40:19.912460 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connector [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] Factory for rbd on None {{(pid=129839) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}} Dec 25 21:40:19.914680 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-2e560f62-0022-4854-a99f-ce9ce208b3c1', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '2e560f62-0022-4854-a99f-ce9ce208b3c1', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:40:19.916090 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.linuxrbd [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] opening connection to ceph cluster (timeout=-1). {{(pid=129839) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Dec 25 21:40:20.100289 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] <== connect_volume: return (184ms) {'path': } {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:40:20.197617 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] Starting backup of volume='2e560f62-0022-4854-a99f-ce9ce208b3c1'. {{(pid=129839) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Dec 25 21:40:20.201314 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] Volume file is RBD: attempting optimized backup {{(pid=129839) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Dec 25 21:40:20.232992 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] Creating base image 'volume-2e560f62-0022-4854-a99f-ce9ce208b3c1.backup.996a92c5-2024-48b9-a849-2d57b581a05d' {{(pid=129839) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Dec 25 21:40:20.233507 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129839) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Dec 25 21:40:21.441192 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] Using --from-snap 'None' for incremental backup of volume 2e560f62-0022-4854-a99f-ce9ce208b3c1. {{(pid=129839) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Dec 25 21:40:21.441876 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] Creating backup snapshot='backup.996a92c5-2024-48b9-a849-2d57b581a05d.snap.1640468421.4411354' {{(pid=129839) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Dec 25 21:40:21.898197 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] Differential backup transfer completed in 56.0367s {{(pid=129839) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Dec 25 21:40:21.899016 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] Getting metadata type 'volume-base-metadata' {{(pid=129839) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Dec 25 21:40:21.911630 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129839) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Dec 25 21:40:21.912048 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] Getting metadata type 'volume-metadata' {{(pid=129839) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Dec 25 21:40:21.919259 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] No metadata type 'volume-metadata' available {{(pid=129839) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Dec 25 21:40:21.919641 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] Getting metadata type 'volume-glance-metadata' {{(pid=129839) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Dec 25 21:40:21.929592 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=129839) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}} Dec 25 21:40:21.930303 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] Backing up metadata for volume 26e57931-5e2e-498f-907c-ba032fee0597. {{(pid=129839) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Dec 25 21:40:22.120788 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] Backup '7c3cc723-7f3f-422b-b14b-8ba52dfd371f' of volume 26e57931-5e2e-498f-907c-ba032fee0597 finished. {{(pid=129839) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Dec 25 21:40:22.122155 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-26e57931-5e2e-498f-907c-ba032fee0597', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '26e57931-5e2e-498f-907c-ba032fee0597', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, 'ignore_errors': True}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:40:22.426341 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] <== disconnect_volume: return (302ms) None {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:40:22.647824 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-d0931f01-3f80-4d85-9ceb-e41f7a1f22d7 tempest-TestVolumeBackupRestore-1733305645 None] Create backup finished. backup: 7c3cc723-7f3f-422b-b14b-8ba52dfd371f. Dec 25 21:40:23.396676 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] Performing differential transfer from 'volume-2e560f62-0022-4854-a99f-ce9ce208b3c1' to 'volume-2e560f62-0022-4854-a99f-ce9ce208b3c1.backup.996a92c5-2024-48b9-a849-2d57b581a05d' {{(pid=129839) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Dec 25 21:40:23.397132 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_0djy4nhb --pool volumes volumes/volume-2e560f62-0022-4854-a99f-ce9ce208b3c1@backup.996a92c5-2024-48b9-a849-2d57b581a05d.snap.1640468421.4411354 -' into... {{(pid=129839) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Dec 25 21:40:23.397490 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-2e560f62-0022-4854-a99f-ce9ce208b3c1.backup.996a92c5-2024-48b9-a849-2d57b581a05d' {{(pid=129839) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Dec 25 21:40:25.641131 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] Differential backup transfer completed in 2.2440s {{(pid=129839) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Dec 25 21:40:25.641752 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] Getting metadata type 'volume-base-metadata' {{(pid=129839) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Dec 25 21:40:25.657353 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129839) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Dec 25 21:40:25.657756 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] Getting metadata type 'volume-metadata' {{(pid=129839) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Dec 25 21:40:25.664031 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] No metadata type 'volume-metadata' available {{(pid=129839) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Dec 25 21:40:25.664403 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] Getting metadata type 'volume-glance-metadata' {{(pid=129839) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Dec 25 21:40:25.670418 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] No metadata type 'volume-glance-metadata' available {{(pid=129839) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Dec 25 21:40:25.670973 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] Backing up metadata for volume 2e560f62-0022-4854-a99f-ce9ce208b3c1. {{(pid=129839) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Dec 25 21:40:25.835574 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] Backup '996a92c5-2024-48b9-a849-2d57b581a05d' of volume 2e560f62-0022-4854-a99f-ce9ce208b3c1 finished. {{(pid=129839) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Dec 25 21:40:25.837433 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-2e560f62-0022-4854-a99f-ce9ce208b3c1', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '2e560f62-0022-4854-a99f-ce9ce208b3c1', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, 'ignore_errors': True}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:40:25.940407 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] <== disconnect_volume: return (102ms) None {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:40:26.106989 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] Restore backup started, backup: 7c3cc723-7f3f-422b-b14b-8ba52dfd371f volume: beb1f33b-8577-4e27-b087-711ea8d893c7. Dec 25 21:40:26.130893 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.utils [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.32', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:40:26.130893 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:40:26.134120 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_db.sqlalchemy.engines [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129839) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 25 21:40:26.140064 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.013s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:40:26.140402 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[ac4964e8-4eef-4916-8a06-5ed5ed558073]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:c2a322773d0\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:40:26.151025 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:40:26.151025 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Dec 25 21:40:26.151025 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[9201478e-ad75-4454-b3ba-56186c4cb799]: [Errno 2] No such file or directory: 'systool' {{(pid=161909) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}} Dec 25 21:40:26.151025 ncdv-master-1 cinder-backup[129839]: Traceback (most recent call last): Dec 25 21:40:26.151025 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd Dec 25 21:40:26.151025 ncdv-master-1 cinder-backup[129839]: ret = func(*f_args, **f_kwargs) Dec 25 21:40:26.151025 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Dec 25 21:40:26.151025 ncdv-master-1 cinder-backup[129839]: return func(*args, **kwargs) Dec 25 21:40:26.151025 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Dec 25 21:40:26.151025 ncdv-master-1 cinder-backup[129839]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Dec 25 21:40:26.151025 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Dec 25 21:40:26.151025 ncdv-master-1 cinder-backup[129839]: return putils.execute(on_execute=on_execute, Dec 25 21:40:26.151025 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Dec 25 21:40:26.151025 ncdv-master-1 cinder-backup[129839]: obj = subprocess.Popen(cmd, Dec 25 21:40:26.151025 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Dec 25 21:40:26.151025 ncdv-master-1 cinder-backup[129839]: self._execute_child(args, executable, preexec_fn, close_fds, Dec 25 21:40:26.151025 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Dec 25 21:40:26.151025 ncdv-master-1 cinder-backup[129839]: raise child_exception_type(errno_num, err_msg, err_filename) Dec 25 21:40:26.151025 ncdv-master-1 cinder-backup[129839]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Dec 25 21:40:26.152419 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[9201478e-ad75-4454-b3ba-56186c4cb799]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:40:26.152419 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:40:26.158480 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Dec 25 21:40:26.158998 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[bcc9bb6e-3492-4828-9a85-5fc557d50794]: [Errno 2] No such file or directory: 'systool' {{(pid=161909) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}} Dec 25 21:40:26.158998 ncdv-master-1 cinder-backup[129839]: Traceback (most recent call last): Dec 25 21:40:26.158998 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd Dec 25 21:40:26.158998 ncdv-master-1 cinder-backup[129839]: ret = func(*f_args, **f_kwargs) Dec 25 21:40:26.158998 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Dec 25 21:40:26.158998 ncdv-master-1 cinder-backup[129839]: return func(*args, **kwargs) Dec 25 21:40:26.158998 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Dec 25 21:40:26.158998 ncdv-master-1 cinder-backup[129839]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Dec 25 21:40:26.158998 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Dec 25 21:40:26.158998 ncdv-master-1 cinder-backup[129839]: return putils.execute(on_execute=on_execute, Dec 25 21:40:26.158998 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Dec 25 21:40:26.158998 ncdv-master-1 cinder-backup[129839]: obj = subprocess.Popen(cmd, Dec 25 21:40:26.158998 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Dec 25 21:40:26.158998 ncdv-master-1 cinder-backup[129839]: self._execute_child(args, executable, preexec_fn, close_fds, Dec 25 21:40:26.158998 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Dec 25 21:40:26.158998 ncdv-master-1 cinder-backup[129839]: raise child_exception_type(errno_num, err_msg, err_filename) Dec 25 21:40:26.158998 ncdv-master-1 cinder-backup[129839]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Dec 25 21:40:26.159929 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[bcc9bb6e-3492-4828-9a85-5fc557d50794]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:40:26.160838 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:40:26.173285 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.012s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:40:26.174144 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[73069731-be89-4b22-a958-65563bc63494]: (4, ('/dev/sda1\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:40:26.178208 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:40:26.190258 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.014s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:40:26.191319 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[d59770f0-cc9f-4a38-b772-d302e1090de4]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:40:26.192552 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:40:26.204251 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-986f1f11-2cc3-4f93-b996-95854c87a1e9 tempest-VolumesBackupsTest-596628470 None] Create backup finished. backup: 996a92c5-2024-48b9-a849-2d57b581a05d. Dec 25 21:40:26.205154 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.011s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:40:26.205724 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[ccf5f71c-23e8-46f1-a96e-d3baaea4d3ac]: (4, ('0b661f24-87f6-9248-8553-e9ec840a31d0\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:40:26.206672 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] Running cmd (subprocess): nvme version {{(pid=129839) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:40:26.224944 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] CMD "nvme version" returned: 0 in 0.018s {{(pid=129839) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:40:26.226071 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.utils [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] <== get_connector_properties: return (99ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.32', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:c2a322773d0', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '0b661f24-87f6-9248-8553-e9ec840a31d0', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:68cc0364-fba7-439f-bcd5-9f4beb715fab'} {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:40:26.942790 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connector [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] Factory for rbd on None {{(pid=129839) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}} Dec 25 21:40:26.944460 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-beb1f33b-8577-4e27-b087-711ea8d893c7', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'beb1f33b-8577-4e27-b087-711ea8d893c7', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:40:26.947026 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.linuxrbd [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] opening connection to ceph cluster (timeout=-1). {{(pid=129839) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Dec 25 21:40:27.066481 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] <== connect_volume: return (121ms) {'path': } {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:40:27.080444 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] Starting restore from Ceph backup=7c3cc723-7f3f-422b-b14b-8ba52dfd371f to volume=volume-beb1f33b-8577-4e27-b087-711ea8d893c7 {{(pid=129839) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1274}} Dec 25 21:40:27.250112 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] Looking for snapshot of backup base 'volume-26e57931-5e2e-498f-907c-ba032fee0597.backup.7c3cc723-7f3f-422b-b14b-8ba52dfd371f' {{(pid=129839) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Dec 25 21:40:27.250591 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] Found snapshot 'backup.7c3cc723-7f3f-422b-b14b-8ba52dfd371f.snap.1640468363.0572503' {{(pid=129839) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Dec 25 21:40:27.255219 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] Volume file is RBD. {{(pid=129839) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1184}} Dec 25 21:40:27.330594 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-e5e5a503-3fc4-499f-9d29-83e2ed84ffbc tempest-VolumesBackupsTest-596628470 None] Delete backup started, backup: 996a92c5-2024-48b9-a849-2d57b581a05d. Dec 25 21:40:27.332800 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-e5e5a503-3fc4-499f-9d29-83e2ed84ffbc tempest-VolumesBackupsTest-596628470 None] Delete started for backup=996a92c5-2024-48b9-a849-2d57b581a05d {{(pid=129839) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Dec 25 21:40:27.333257 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-e5e5a503-3fc4-499f-9d29-83e2ed84ffbc tempest-VolumesBackupsTest-596628470 None] Trying diff format basename='volume-2e560f62-0022-4854-a99f-ce9ce208b3c1.backup.996a92c5-2024-48b9-a849-2d57b581a05d' for backup base image of volume 2e560f62-0022-4854-a99f-ce9ce208b3c1. {{(pid=129839) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Dec 25 21:40:27.347612 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] Attempting differential restore. {{(pid=129839) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1234}} Dec 25 21:40:27.348083 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] Attempting incremental restore from base='volume-26e57931-5e2e-498f-907c-ba032fee0597.backup.7c3cc723-7f3f-422b-b14b-8ba52dfd371f' snap='backup.7c3cc723-7f3f-422b-b14b-8ba52dfd371f.snap.1640468363.0572503' {{(pid=129839) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1089}} Dec 25 21:40:27.348470 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] Performing differential transfer from 'volume-26e57931-5e2e-498f-907c-ba032fee0597.backup.7c3cc723-7f3f-422b-b14b-8ba52dfd371f' to 'volume-beb1f33b-8577-4e27-b087-711ea8d893c7' {{(pid=129839) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Dec 25 21:40:27.348864 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-26e57931-5e2e-498f-907c-ba032fee0597.backup.7c3cc723-7f3f-422b-b14b-8ba52dfd371f@backup.7c3cc723-7f3f-422b-b14b-8ba52dfd371f.snap.1640468363.0572503 -' into... {{(pid=129839) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Dec 25 21:40:27.349268 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_91ae5ndl --pool volumes - volumes/volume-beb1f33b-8577-4e27-b087-711ea8d893c7' {{(pid=129839) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Dec 25 21:40:27.390935 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_db.sqlalchemy.engines [None req-9b4bfa17-f8eb-43dd-9d64-d05efc7c4b62 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129839) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 25 21:40:27.521931 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-e5e5a503-3fc4-499f-9d29-83e2ed84ffbc tempest-VolumesBackupsTest-596628470 None] Looking for snapshot of backup base 'volume-2e560f62-0022-4854-a99f-ce9ce208b3c1.backup.996a92c5-2024-48b9-a849-2d57b581a05d' {{(pid=129839) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Dec 25 21:40:27.522385 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-e5e5a503-3fc4-499f-9d29-83e2ed84ffbc tempest-VolumesBackupsTest-596628470 None] Found snapshot 'backup.996a92c5-2024-48b9-a849-2d57b581a05d.snap.1640468421.4411354' {{(pid=129839) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Dec 25 21:40:27.522763 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-e5e5a503-3fc4-499f-9d29-83e2ed84ffbc tempest-VolumesBackupsTest-596628470 None] Deleting backup snapshot='backup.996a92c5-2024-48b9-a849-2d57b581a05d.snap.1640468421.4411354' {{(pid=129839) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Dec 25 21:40:31.141200 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.drivers.ceph [None req-e5e5a503-3fc4-499f-9d29-83e2ed84ffbc tempest-VolumesBackupsTest-596628470 None] Deleting backup base image='volume-2e560f62-0022-4854-a99f-ce9ce208b3c1.backup.996a92c5-2024-48b9-a849-2d57b581a05d' of volume 2e560f62-0022-4854-a99f-ce9ce208b3c1. Dec 25 21:40:42.233023 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-e5e5a503-3fc4-499f-9d29-83e2ed84ffbc tempest-VolumesBackupsTest-596628470 None] Base backup image='volume-2e560f62-0022-4854-a99f-ce9ce208b3c1.backup.996a92c5-2024-48b9-a849-2d57b581a05d' of volume 2e560f62-0022-4854-a99f-ce9ce208b3c1 deleted. {{(pid=129839) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Dec 25 21:40:42.536827 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-e5e5a503-3fc4-499f-9d29-83e2ed84ffbc tempest-VolumesBackupsTest-596628470 None] Delete of backup '996a92c5-2024-48b9-a849-2d57b581a05d' for volume '2e560f62-0022-4854-a99f-ce9ce208b3c1' finished. {{(pid=129839) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Dec 25 21:40:42.626444 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.quota [None req-e5e5a503-3fc4-499f-9d29-83e2ed84ffbc tempest-VolumesBackupsTest-596628470 None] Created reservations ['eb48c6d2-19da-4695-92cd-2475e177296f', '9ee42d55-a2cc-47f9-acea-3917d02775b2'] {{(pid=129839) reserve /opt/stack/cinder/cinder/quota.py:755}} Dec 25 21:40:42.855847 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-e5e5a503-3fc4-499f-9d29-83e2ed84ffbc tempest-VolumesBackupsTest-596628470 None] Delete backup finished, backup 996a92c5-2024-48b9-a849-2d57b581a05d deleted. Dec 25 21:40:43.117463 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:40:43.117819 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:41:14.742861 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] Restore transfer completed in 47.3941s {{(pid=129839) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1111}} Dec 25 21:41:14.743866 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=129839) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1285}} Dec 25 21:41:14.772255 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] Restoring volume base metadata {{(pid=129839) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} Dec 25 21:41:14.922675 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] No metadata of type 'volume-metadata' to restore {{(pid=129839) put /opt/stack/cinder/cinder/backup/driver.py:345}} Dec 25 21:41:14.923074 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] Restoring volume glance metadata {{(pid=129839) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}} Dec 25 21:41:15.680838 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] Create backup started, backup: 9572109d-ce58-4d2d-8b06-f838523c3b76 volume: e2767895-8ea1-400c-9c10-00e5832d20fb. Dec 25 21:41:15.769875 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.utils [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.32', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:41:15.771690 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:41:15.783247 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.012s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:41:15.783574 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[8d00d324-231f-41bd-84eb-51fd630cac04]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:c2a322773d0\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:41:15.785010 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_db.sqlalchemy.engines [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129839) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 25 21:41:15.785574 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:41:15.792552 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Dec 25 21:41:15.792837 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[f3950c7a-3d7c-49f5-98cf-8adde9135a17]: [Errno 2] No such file or directory: 'systool' {{(pid=161909) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}} Dec 25 21:41:15.792837 ncdv-master-1 cinder-backup[129839]: Traceback (most recent call last): Dec 25 21:41:15.792837 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd Dec 25 21:41:15.792837 ncdv-master-1 cinder-backup[129839]: ret = func(*f_args, **f_kwargs) Dec 25 21:41:15.792837 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Dec 25 21:41:15.792837 ncdv-master-1 cinder-backup[129839]: return func(*args, **kwargs) Dec 25 21:41:15.792837 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Dec 25 21:41:15.792837 ncdv-master-1 cinder-backup[129839]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Dec 25 21:41:15.792837 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Dec 25 21:41:15.792837 ncdv-master-1 cinder-backup[129839]: return putils.execute(on_execute=on_execute, Dec 25 21:41:15.792837 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Dec 25 21:41:15.792837 ncdv-master-1 cinder-backup[129839]: obj = subprocess.Popen(cmd, Dec 25 21:41:15.792837 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Dec 25 21:41:15.792837 ncdv-master-1 cinder-backup[129839]: self._execute_child(args, executable, preexec_fn, close_fds, Dec 25 21:41:15.792837 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Dec 25 21:41:15.792837 ncdv-master-1 cinder-backup[129839]: raise child_exception_type(errno_num, err_msg, err_filename) Dec 25 21:41:15.792837 ncdv-master-1 cinder-backup[129839]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Dec 25 21:41:15.794027 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[f3950c7a-3d7c-49f5-98cf-8adde9135a17]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:41:15.797038 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:41:15.801565 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Dec 25 21:41:15.802575 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[56d0d0a4-3c9e-40e4-9d2f-be62a322fc7d]: [Errno 2] No such file or directory: 'systool' {{(pid=161909) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}} Dec 25 21:41:15.802575 ncdv-master-1 cinder-backup[129839]: Traceback (most recent call last): Dec 25 21:41:15.802575 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd Dec 25 21:41:15.802575 ncdv-master-1 cinder-backup[129839]: ret = func(*f_args, **f_kwargs) Dec 25 21:41:15.802575 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Dec 25 21:41:15.802575 ncdv-master-1 cinder-backup[129839]: return func(*args, **kwargs) Dec 25 21:41:15.802575 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Dec 25 21:41:15.802575 ncdv-master-1 cinder-backup[129839]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Dec 25 21:41:15.802575 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Dec 25 21:41:15.802575 ncdv-master-1 cinder-backup[129839]: return putils.execute(on_execute=on_execute, Dec 25 21:41:15.802575 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Dec 25 21:41:15.802575 ncdv-master-1 cinder-backup[129839]: obj = subprocess.Popen(cmd, Dec 25 21:41:15.802575 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Dec 25 21:41:15.802575 ncdv-master-1 cinder-backup[129839]: self._execute_child(args, executable, preexec_fn, close_fds, Dec 25 21:41:15.802575 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Dec 25 21:41:15.802575 ncdv-master-1 cinder-backup[129839]: raise child_exception_type(errno_num, err_msg, err_filename) Dec 25 21:41:15.802575 ncdv-master-1 cinder-backup[129839]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Dec 25 21:41:15.803924 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[56d0d0a4-3c9e-40e4-9d2f-be62a322fc7d]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:41:15.807574 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:41:15.814279 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.009s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:41:15.814775 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[e847d603-c490-4a09-875a-5c72fca04daa]: (4, ('/dev/sda1\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:41:15.816353 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:41:15.825812 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.009s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:41:15.826137 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[261dc06d-e280-4be1-a47f-69f92ca89a7d]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:41:15.827901 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:41:15.834330 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.006s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:41:15.834663 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[dc72b5d0-f1a1-4a23-ad3b-bbf3d898ccc2]: (4, ('0b661f24-87f6-9248-8553-e9ec840a31d0\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:41:15.835411 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] Running cmd (subprocess): nvme version {{(pid=129839) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:41:15.854119 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] CMD "nvme version" returned: 0 in 0.018s {{(pid=129839) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:41:15.855189 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.utils [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] <== get_connector_properties: return (85ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.32', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:c2a322773d0', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '0b661f24-87f6-9248-8553-e9ec840a31d0', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:68cc0364-fba7-439f-bcd5-9f4beb715fab'} {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:41:16.086252 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] Restore to volume beb1f33b-8577-4e27-b087-711ea8d893c7 finished successfully. {{(pid=129839) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1292}} Dec 25 21:41:16.087846 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-beb1f33b-8577-4e27-b087-711ea8d893c7', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'beb1f33b-8577-4e27-b087-711ea8d893c7', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, 'ignore_errors': False}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:41:16.167586 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] <== disconnect_volume: return (79ms) None {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:41:16.218288 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.manager [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] Encryption key ID for volume beb1f33b-8577-4e27-b087-711ea8d893c7 already matches encryption key ID in backup 7c3cc723-7f3f-422b-b14b-8ba52dfd371f. {{(pid=129839) _run_restore /opt/stack/cinder/cinder/backup/manager.py:801}} Dec 25 21:41:16.420218 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-0dab36ad-1b77-4ca1-91c2-35e149ac0796 tempest-TestVolumeBackupRestore-1733305645 None] Finished restoring backup 7c3cc723-7f3f-422b-b14b-8ba52dfd371f to volume beb1f33b-8577-4e27-b087-711ea8d893c7. Dec 25 21:41:16.611991 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connector [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] Factory for rbd on None {{(pid=129839) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}} Dec 25 21:41:16.613137 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-e2767895-8ea1-400c-9c10-00e5832d20fb', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'e2767895-8ea1-400c-9c10-00e5832d20fb', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:41:16.614160 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.linuxrbd [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] opening connection to ceph cluster (timeout=-1). {{(pid=129839) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Dec 25 21:41:16.791536 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] <== connect_volume: return (177ms) {'path': } {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:41:16.882750 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] Starting backup of volume='e2767895-8ea1-400c-9c10-00e5832d20fb'. {{(pid=129839) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Dec 25 21:41:16.883625 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] Volume file is RBD: attempting optimized backup {{(pid=129839) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Dec 25 21:41:16.903488 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] Creating base image 'volume-e2767895-8ea1-400c-9c10-00e5832d20fb.backup.9572109d-ce58-4d2d-8b06-f838523c3b76' {{(pid=129839) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Dec 25 21:41:16.903899 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129839) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Dec 25 21:41:17.320110 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] Using --from-snap 'None' for incremental backup of volume e2767895-8ea1-400c-9c10-00e5832d20fb. {{(pid=129839) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Dec 25 21:41:17.320519 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] Creating backup snapshot='backup.9572109d-ce58-4d2d-8b06-f838523c3b76.snap.1640468477.3201368' {{(pid=129839) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Dec 25 21:41:18.690727 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] Performing differential transfer from 'volume-e2767895-8ea1-400c-9c10-00e5832d20fb' to 'volume-e2767895-8ea1-400c-9c10-00e5832d20fb.backup.9572109d-ce58-4d2d-8b06-f838523c3b76' {{(pid=129839) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Dec 25 21:41:18.691737 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_3xauzco8 --pool volumes volumes/volume-e2767895-8ea1-400c-9c10-00e5832d20fb@backup.9572109d-ce58-4d2d-8b06-f838523c3b76.snap.1640468477.3201368 -' into... {{(pid=129839) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Dec 25 21:41:18.692282 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-e2767895-8ea1-400c-9c10-00e5832d20fb.backup.9572109d-ce58-4d2d-8b06-f838523c3b76' {{(pid=129839) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Dec 25 21:41:43.114454 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:41:43.116485 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:42:04.104305 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] Differential backup transfer completed in 45.4135s {{(pid=129839) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Dec 25 21:42:04.104991 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] Getting metadata type 'volume-base-metadata' {{(pid=129839) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Dec 25 21:42:04.117168 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129839) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Dec 25 21:42:04.117518 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] Getting metadata type 'volume-metadata' {{(pid=129839) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Dec 25 21:42:04.123476 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] No metadata type 'volume-metadata' available {{(pid=129839) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Dec 25 21:42:04.123851 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] Getting metadata type 'volume-glance-metadata' {{(pid=129839) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Dec 25 21:42:04.130820 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=129839) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}} Dec 25 21:42:04.131386 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] Backing up metadata for volume e2767895-8ea1-400c-9c10-00e5832d20fb. {{(pid=129839) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Dec 25 21:42:04.331876 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] Backup '9572109d-ce58-4d2d-8b06-f838523c3b76' of volume e2767895-8ea1-400c-9c10-00e5832d20fb finished. {{(pid=129839) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Dec 25 21:42:04.333594 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-e2767895-8ea1-400c-9c10-00e5832d20fb', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'e2767895-8ea1-400c-9c10-00e5832d20fb', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, 'ignore_errors': True}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:42:05.225599 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] <== disconnect_volume: return (891ms) None {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:42:05.408785 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-4e5e29d2-86fa-49b7-82f4-be229232b47d tempest-VolumesBackupsTest-596628470 None] Create backup finished. backup: 9572109d-ce58-4d2d-8b06-f838523c3b76. Dec 25 21:42:08.637287 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] Restore backup started, backup: 9572109d-ce58-4d2d-8b06-f838523c3b76 volume: 3fcdc80c-5f5d-49b4-9b24-27ff806ca9a5. Dec 25 21:42:08.652062 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.utils [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.32', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:42:08.661900 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:42:08.662383 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.009s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:42:08.662753 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[b6b8f61e-b1cc-4696-829b-a7929f7f8889]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:c2a322773d0\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:42:08.664010 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:42:08.670481 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Dec 25 21:42:08.671058 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[d63662df-16f5-4ed1-adaa-2a9e522378a1]: [Errno 2] No such file or directory: 'systool' {{(pid=161909) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}} Dec 25 21:42:08.671058 ncdv-master-1 cinder-backup[129839]: Traceback (most recent call last): Dec 25 21:42:08.671058 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd Dec 25 21:42:08.671058 ncdv-master-1 cinder-backup[129839]: ret = func(*f_args, **f_kwargs) Dec 25 21:42:08.671058 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Dec 25 21:42:08.671058 ncdv-master-1 cinder-backup[129839]: return func(*args, **kwargs) Dec 25 21:42:08.671058 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Dec 25 21:42:08.671058 ncdv-master-1 cinder-backup[129839]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Dec 25 21:42:08.671058 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Dec 25 21:42:08.671058 ncdv-master-1 cinder-backup[129839]: return putils.execute(on_execute=on_execute, Dec 25 21:42:08.671058 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Dec 25 21:42:08.671058 ncdv-master-1 cinder-backup[129839]: obj = subprocess.Popen(cmd, Dec 25 21:42:08.671058 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Dec 25 21:42:08.671058 ncdv-master-1 cinder-backup[129839]: self._execute_child(args, executable, preexec_fn, close_fds, Dec 25 21:42:08.671058 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Dec 25 21:42:08.671058 ncdv-master-1 cinder-backup[129839]: raise child_exception_type(errno_num, err_msg, err_filename) Dec 25 21:42:08.671058 ncdv-master-1 cinder-backup[129839]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Dec 25 21:42:08.672258 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[d63662df-16f5-4ed1-adaa-2a9e522378a1]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:42:08.673064 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:42:08.679117 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Dec 25 21:42:08.679655 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[3a2702d3-e0c3-426d-a4d4-829fc4f58258]: [Errno 2] No such file or directory: 'systool' {{(pid=161909) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}} Dec 25 21:42:08.679655 ncdv-master-1 cinder-backup[129839]: Traceback (most recent call last): Dec 25 21:42:08.679655 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd Dec 25 21:42:08.679655 ncdv-master-1 cinder-backup[129839]: ret = func(*f_args, **f_kwargs) Dec 25 21:42:08.679655 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Dec 25 21:42:08.679655 ncdv-master-1 cinder-backup[129839]: return func(*args, **kwargs) Dec 25 21:42:08.679655 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Dec 25 21:42:08.679655 ncdv-master-1 cinder-backup[129839]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Dec 25 21:42:08.679655 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Dec 25 21:42:08.679655 ncdv-master-1 cinder-backup[129839]: return putils.execute(on_execute=on_execute, Dec 25 21:42:08.679655 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Dec 25 21:42:08.679655 ncdv-master-1 cinder-backup[129839]: obj = subprocess.Popen(cmd, Dec 25 21:42:08.679655 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Dec 25 21:42:08.679655 ncdv-master-1 cinder-backup[129839]: self._execute_child(args, executable, preexec_fn, close_fds, Dec 25 21:42:08.679655 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Dec 25 21:42:08.679655 ncdv-master-1 cinder-backup[129839]: raise child_exception_type(errno_num, err_msg, err_filename) Dec 25 21:42:08.679655 ncdv-master-1 cinder-backup[129839]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Dec 25 21:42:08.680563 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[3a2702d3-e0c3-426d-a4d4-829fc4f58258]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:42:08.681344 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:42:08.689196 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.008s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:42:08.689522 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[38daa3eb-ae85-4301-9126-2d5a7da0e94c]: (4, ('/dev/sda1\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:42:08.693287 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:42:08.699082 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.008s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:42:08.699427 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[4a4a66b0-5669-47d0-a144-33c1e1ec2eeb]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:42:08.700688 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:42:08.709659 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.008s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:42:08.711496 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[26035d5a-c72a-41bc-bd0f-6d5f187de127]: (4, ('0b661f24-87f6-9248-8553-e9ec840a31d0\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:42:08.711496 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] Running cmd (subprocess): nvme version {{(pid=129839) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:42:08.730377 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] CMD "nvme version" returned: 0 in 0.019s {{(pid=129839) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:42:08.731401 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.utils [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] <== get_connector_properties: return (78ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.32', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:c2a322773d0', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '0b661f24-87f6-9248-8553-e9ec840a31d0', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:68cc0364-fba7-439f-bcd5-9f4beb715fab'} {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:42:09.431681 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connector [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] Factory for rbd on None {{(pid=129839) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}} Dec 25 21:42:09.433393 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-3fcdc80c-5f5d-49b4-9b24-27ff806ca9a5', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '3fcdc80c-5f5d-49b4-9b24-27ff806ca9a5', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:42:09.434162 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.linuxrbd [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] opening connection to ceph cluster (timeout=-1). {{(pid=129839) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Dec 25 21:42:09.762341 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] <== connect_volume: return (328ms) {'path': } {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:42:09.770575 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_db.sqlalchemy.engines [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129839) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 25 21:42:09.780362 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] Starting restore from Ceph backup=9572109d-ce58-4d2d-8b06-f838523c3b76 to volume=volume-3fcdc80c-5f5d-49b4-9b24-27ff806ca9a5 {{(pid=129839) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1274}} Dec 25 21:42:09.958948 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] Looking for snapshot of backup base 'volume-e2767895-8ea1-400c-9c10-00e5832d20fb.backup.9572109d-ce58-4d2d-8b06-f838523c3b76' {{(pid=129839) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Dec 25 21:42:09.959533 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] Found snapshot 'backup.9572109d-ce58-4d2d-8b06-f838523c3b76.snap.1640468477.3201368' {{(pid=129839) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Dec 25 21:42:09.965020 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] Volume file is RBD. {{(pid=129839) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1184}} Dec 25 21:42:10.097450 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] Attempting differential restore. {{(pid=129839) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1234}} Dec 25 21:42:10.097933 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] Attempting incremental restore from base='volume-e2767895-8ea1-400c-9c10-00e5832d20fb.backup.9572109d-ce58-4d2d-8b06-f838523c3b76' snap='backup.9572109d-ce58-4d2d-8b06-f838523c3b76.snap.1640468477.3201368' {{(pid=129839) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1089}} Dec 25 21:42:10.098328 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] Performing differential transfer from 'volume-e2767895-8ea1-400c-9c10-00e5832d20fb.backup.9572109d-ce58-4d2d-8b06-f838523c3b76' to 'volume-3fcdc80c-5f5d-49b4-9b24-27ff806ca9a5' {{(pid=129839) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Dec 25 21:42:10.098730 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-e2767895-8ea1-400c-9c10-00e5832d20fb.backup.9572109d-ce58-4d2d-8b06-f838523c3b76@backup.9572109d-ce58-4d2d-8b06-f838523c3b76.snap.1640468477.3201368 -' into... {{(pid=129839) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Dec 25 21:42:10.099101 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_5pdeqnr5 --pool volumes - volumes/volume-3fcdc80c-5f5d-49b4-9b24-27ff806ca9a5' {{(pid=129839) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Dec 25 21:42:43.112234 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:42:43.112905 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:43:03.807643 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] Restore transfer completed in 53.7090s {{(pid=129839) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1111}} Dec 25 21:43:03.808605 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=129839) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1285}} Dec 25 21:43:03.848667 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] Restoring volume base metadata {{(pid=129839) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} Dec 25 21:43:03.972740 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] No metadata of type 'volume-metadata' to restore {{(pid=129839) put /opt/stack/cinder/cinder/backup/driver.py:345}} Dec 25 21:43:03.973376 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] Restoring volume glance metadata {{(pid=129839) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}} Dec 25 21:43:05.264062 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] Restore to volume 3fcdc80c-5f5d-49b4-9b24-27ff806ca9a5 finished successfully. {{(pid=129839) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1292}} Dec 25 21:43:05.265653 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-3fcdc80c-5f5d-49b4-9b24-27ff806ca9a5', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '3fcdc80c-5f5d-49b4-9b24-27ff806ca9a5', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, 'ignore_errors': False}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:43:05.511911 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] <== disconnect_volume: return (245ms) None {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:43:05.573772 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.manager [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] Encryption key ID for volume 3fcdc80c-5f5d-49b4-9b24-27ff806ca9a5 already matches encryption key ID in backup 9572109d-ce58-4d2d-8b06-f838523c3b76. {{(pid=129839) _run_restore /opt/stack/cinder/cinder/backup/manager.py:801}} Dec 25 21:43:05.883764 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-0a9faf64-891d-4d7d-9537-8eea7c0f36a0 tempest-VolumesBackupsTest-596628470 None] Finished restoring backup 9572109d-ce58-4d2d-8b06-f838523c3b76 to volume 3fcdc80c-5f5d-49b4-9b24-27ff806ca9a5. Dec 25 21:43:24.958532 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-4579516f-ac6d-428b-8b50-e31ea7857f93 tempest-VolumesBackupsTest-596628470 None] Delete backup started, backup: 9572109d-ce58-4d2d-8b06-f838523c3b76. Dec 25 21:43:24.963790 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4579516f-ac6d-428b-8b50-e31ea7857f93 tempest-VolumesBackupsTest-596628470 None] Delete started for backup=9572109d-ce58-4d2d-8b06-f838523c3b76 {{(pid=129839) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Dec 25 21:43:24.964258 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4579516f-ac6d-428b-8b50-e31ea7857f93 tempest-VolumesBackupsTest-596628470 None] Trying diff format basename='volume-e2767895-8ea1-400c-9c10-00e5832d20fb.backup.9572109d-ce58-4d2d-8b06-f838523c3b76' for backup base image of volume e2767895-8ea1-400c-9c10-00e5832d20fb. {{(pid=129839) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Dec 25 21:43:25.250033 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4579516f-ac6d-428b-8b50-e31ea7857f93 tempest-VolumesBackupsTest-596628470 None] Looking for snapshot of backup base 'volume-e2767895-8ea1-400c-9c10-00e5832d20fb.backup.9572109d-ce58-4d2d-8b06-f838523c3b76' {{(pid=129839) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Dec 25 21:43:25.250842 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4579516f-ac6d-428b-8b50-e31ea7857f93 tempest-VolumesBackupsTest-596628470 None] Found snapshot 'backup.9572109d-ce58-4d2d-8b06-f838523c3b76.snap.1640468477.3201368' {{(pid=129839) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Dec 25 21:43:25.251246 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4579516f-ac6d-428b-8b50-e31ea7857f93 tempest-VolumesBackupsTest-596628470 None] Deleting backup snapshot='backup.9572109d-ce58-4d2d-8b06-f838523c3b76.snap.1640468477.3201368' {{(pid=129839) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Dec 25 21:43:27.358302 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_db.sqlalchemy.engines [None req-9b4bfa17-f8eb-43dd-9d64-d05efc7c4b62 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129839) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 25 21:43:28.251691 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.drivers.ceph [None req-4579516f-ac6d-428b-8b50-e31ea7857f93 tempest-VolumesBackupsTest-596628470 None] Deleting backup base image='volume-e2767895-8ea1-400c-9c10-00e5832d20fb.backup.9572109d-ce58-4d2d-8b06-f838523c3b76' of volume e2767895-8ea1-400c-9c10-00e5832d20fb. Dec 25 21:43:42.401873 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4579516f-ac6d-428b-8b50-e31ea7857f93 tempest-VolumesBackupsTest-596628470 None] Base backup image='volume-e2767895-8ea1-400c-9c10-00e5832d20fb.backup.9572109d-ce58-4d2d-8b06-f838523c3b76' of volume e2767895-8ea1-400c-9c10-00e5832d20fb deleted. {{(pid=129839) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Dec 25 21:43:42.705113 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4579516f-ac6d-428b-8b50-e31ea7857f93 tempest-VolumesBackupsTest-596628470 None] Delete of backup '9572109d-ce58-4d2d-8b06-f838523c3b76' for volume 'e2767895-8ea1-400c-9c10-00e5832d20fb' finished. {{(pid=129839) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Dec 25 21:43:42.923050 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.quota [None req-4579516f-ac6d-428b-8b50-e31ea7857f93 tempest-VolumesBackupsTest-596628470 None] Created reservations ['217c5148-de76-4a5d-9f25-fd92f150dcf2', '6048cac4-35a7-47af-adc9-d42bacdf8a01'] {{(pid=129839) reserve /opt/stack/cinder/cinder/quota.py:755}} Dec 25 21:43:43.107871 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:43:43.114690 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:43:43.131931 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-4579516f-ac6d-428b-8b50-e31ea7857f93 tempest-VolumesBackupsTest-596628470 None] Delete backup finished, backup 9572109d-ce58-4d2d-8b06-f838523c3b76 deleted. Dec 25 21:43:49.178268 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-e19a46b2-5357-41cb-8ee9-a6b2e36ba964 tempest-TestVolumeBackupRestore-1733305645 None] Delete backup started, backup: 7c3cc723-7f3f-422b-b14b-8ba52dfd371f. Dec 25 21:43:49.180010 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-e19a46b2-5357-41cb-8ee9-a6b2e36ba964 tempest-TestVolumeBackupRestore-1733305645 None] Delete started for backup=7c3cc723-7f3f-422b-b14b-8ba52dfd371f {{(pid=129839) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Dec 25 21:43:49.180380 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-e19a46b2-5357-41cb-8ee9-a6b2e36ba964 tempest-TestVolumeBackupRestore-1733305645 None] Trying diff format basename='volume-26e57931-5e2e-498f-907c-ba032fee0597.backup.7c3cc723-7f3f-422b-b14b-8ba52dfd371f' for backup base image of volume 26e57931-5e2e-498f-907c-ba032fee0597. {{(pid=129839) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Dec 25 21:43:49.570871 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-e19a46b2-5357-41cb-8ee9-a6b2e36ba964 tempest-TestVolumeBackupRestore-1733305645 None] Looking for snapshot of backup base 'volume-26e57931-5e2e-498f-907c-ba032fee0597.backup.7c3cc723-7f3f-422b-b14b-8ba52dfd371f' {{(pid=129839) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Dec 25 21:43:49.571059 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-e19a46b2-5357-41cb-8ee9-a6b2e36ba964 tempest-TestVolumeBackupRestore-1733305645 None] Found snapshot 'backup.7c3cc723-7f3f-422b-b14b-8ba52dfd371f.snap.1640468363.0572503' {{(pid=129839) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Dec 25 21:43:49.571372 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-e19a46b2-5357-41cb-8ee9-a6b2e36ba964 tempest-TestVolumeBackupRestore-1733305645 None] Deleting backup snapshot='backup.7c3cc723-7f3f-422b-b14b-8ba52dfd371f.snap.1640468363.0572503' {{(pid=129839) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Dec 25 21:43:53.793780 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.drivers.ceph [None req-e19a46b2-5357-41cb-8ee9-a6b2e36ba964 tempest-TestVolumeBackupRestore-1733305645 None] Deleting backup base image='volume-26e57931-5e2e-498f-907c-ba032fee0597.backup.7c3cc723-7f3f-422b-b14b-8ba52dfd371f' of volume 26e57931-5e2e-498f-907c-ba032fee0597. Dec 25 21:43:57.364510 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_db.sqlalchemy.engines [None req-9b4bfa17-f8eb-43dd-9d64-d05efc7c4b62 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129839) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 25 21:44:07.126987 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-e19a46b2-5357-41cb-8ee9-a6b2e36ba964 tempest-TestVolumeBackupRestore-1733305645 None] Base backup image='volume-26e57931-5e2e-498f-907c-ba032fee0597.backup.7c3cc723-7f3f-422b-b14b-8ba52dfd371f' of volume 26e57931-5e2e-498f-907c-ba032fee0597 deleted. {{(pid=129839) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Dec 25 21:44:07.524569 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-e19a46b2-5357-41cb-8ee9-a6b2e36ba964 tempest-TestVolumeBackupRestore-1733305645 None] Delete of backup '7c3cc723-7f3f-422b-b14b-8ba52dfd371f' for volume '26e57931-5e2e-498f-907c-ba032fee0597' finished. {{(pid=129839) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Dec 25 21:44:07.628821 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.quota [None req-e19a46b2-5357-41cb-8ee9-a6b2e36ba964 tempest-TestVolumeBackupRestore-1733305645 None] Created reservations ['cb978ffc-0479-4587-9431-5d2bbac0a97a', '7ee1f0f8-4f53-4408-8b6e-9a9c7f4f8983'] {{(pid=129839) reserve /opt/stack/cinder/cinder/quota.py:755}} Dec 25 21:44:07.862203 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-e19a46b2-5357-41cb-8ee9-a6b2e36ba964 tempest-TestVolumeBackupRestore-1733305645 None] Delete backup finished, backup 7c3cc723-7f3f-422b-b14b-8ba52dfd371f deleted. Dec 25 21:44:43.104592 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:44:43.104592 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:45:43.110136 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:45:43.110720 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:46:15.145972 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] Create backup started, backup: db8899c3-364f-425e-837b-0faa75339ff4 volume: 5b13f37b-b766-43c1-8d43-9ce551127c45. Dec 25 21:46:15.250110 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.utils [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.32', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:46:15.251585 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:46:15.259162 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.007s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:46:15.259502 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[8ff28ed9-031c-4f7a-97c6-1f7e3bce7cc6]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:c2a322773d0\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:46:15.261191 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:46:15.267127 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Dec 25 21:46:15.267719 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[2a7a852d-0fed-4ada-a15f-f4144c07c40f]: [Errno 2] No such file or directory: 'systool' {{(pid=161909) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}} Dec 25 21:46:15.267719 ncdv-master-1 cinder-backup[129839]: Traceback (most recent call last): Dec 25 21:46:15.267719 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd Dec 25 21:46:15.267719 ncdv-master-1 cinder-backup[129839]: ret = func(*f_args, **f_kwargs) Dec 25 21:46:15.267719 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Dec 25 21:46:15.267719 ncdv-master-1 cinder-backup[129839]: return func(*args, **kwargs) Dec 25 21:46:15.267719 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Dec 25 21:46:15.267719 ncdv-master-1 cinder-backup[129839]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Dec 25 21:46:15.267719 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Dec 25 21:46:15.267719 ncdv-master-1 cinder-backup[129839]: return putils.execute(on_execute=on_execute, Dec 25 21:46:15.267719 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Dec 25 21:46:15.267719 ncdv-master-1 cinder-backup[129839]: obj = subprocess.Popen(cmd, Dec 25 21:46:15.267719 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Dec 25 21:46:15.267719 ncdv-master-1 cinder-backup[129839]: self._execute_child(args, executable, preexec_fn, close_fds, Dec 25 21:46:15.267719 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Dec 25 21:46:15.267719 ncdv-master-1 cinder-backup[129839]: raise child_exception_type(errno_num, err_msg, err_filename) Dec 25 21:46:15.267719 ncdv-master-1 cinder-backup[129839]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Dec 25 21:46:15.269027 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[2a7a852d-0fed-4ada-a15f-f4144c07c40f]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:46:15.269992 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:46:15.275998 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Dec 25 21:46:15.276983 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[04afad65-9110-4fae-9c70-3c3193d038e0]: [Errno 2] No such file or directory: 'systool' {{(pid=161909) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}} Dec 25 21:46:15.276983 ncdv-master-1 cinder-backup[129839]: Traceback (most recent call last): Dec 25 21:46:15.276983 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd Dec 25 21:46:15.276983 ncdv-master-1 cinder-backup[129839]: ret = func(*f_args, **f_kwargs) Dec 25 21:46:15.276983 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Dec 25 21:46:15.276983 ncdv-master-1 cinder-backup[129839]: return func(*args, **kwargs) Dec 25 21:46:15.276983 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Dec 25 21:46:15.276983 ncdv-master-1 cinder-backup[129839]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Dec 25 21:46:15.276983 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Dec 25 21:46:15.276983 ncdv-master-1 cinder-backup[129839]: return putils.execute(on_execute=on_execute, Dec 25 21:46:15.276983 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Dec 25 21:46:15.276983 ncdv-master-1 cinder-backup[129839]: obj = subprocess.Popen(cmd, Dec 25 21:46:15.276983 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Dec 25 21:46:15.276983 ncdv-master-1 cinder-backup[129839]: self._execute_child(args, executable, preexec_fn, close_fds, Dec 25 21:46:15.276983 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Dec 25 21:46:15.276983 ncdv-master-1 cinder-backup[129839]: raise child_exception_type(errno_num, err_msg, err_filename) Dec 25 21:46:15.276983 ncdv-master-1 cinder-backup[129839]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Dec 25 21:46:15.281648 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[04afad65-9110-4fae-9c70-3c3193d038e0]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:46:15.281648 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:46:15.287288 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.008s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:46:15.287697 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[41688e24-2cd1-4a59-a0b5-bca0b570863d]: (4, ('/dev/sda1\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:46:15.288829 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:46:15.298504 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.009s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:46:15.298854 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[79f478f5-fd82-437d-ab93-27d4060e86e7]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:46:15.300087 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:46:15.307338 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.006s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:46:15.308982 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[66a21c14-a877-42d0-9866-ba107e1b90b5]: (4, ('0b661f24-87f6-9248-8553-e9ec840a31d0\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:46:15.308982 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] Running cmd (subprocess): nvme version {{(pid=129839) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:46:15.326128 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] CMD "nvme version" returned: 0 in 0.017s {{(pid=129839) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:46:15.327210 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.utils [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] <== get_connector_properties: return (76ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.32', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:c2a322773d0', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '0b661f24-87f6-9248-8553-e9ec840a31d0', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:68cc0364-fba7-439f-bcd5-9f4beb715fab'} {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:46:16.044846 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connector [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] Factory for rbd on None {{(pid=129839) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}} Dec 25 21:46:16.050557 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-5b13f37b-b766-43c1-8d43-9ce551127c45', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '5b13f37b-b766-43c1-8d43-9ce551127c45', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:46:16.051612 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.linuxrbd [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] opening connection to ceph cluster (timeout=-1). {{(pid=129839) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Dec 25 21:46:16.162443 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] <== connect_volume: return (111ms) {'path': } {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:46:16.171344 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_db.sqlalchemy.engines [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129839) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 25 21:46:16.248768 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] Starting backup of volume='5b13f37b-b766-43c1-8d43-9ce551127c45'. {{(pid=129839) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Dec 25 21:46:16.250300 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] Volume file is RBD: attempting optimized backup {{(pid=129839) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Dec 25 21:46:16.275611 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] Creating base image 'volume-5b13f37b-b766-43c1-8d43-9ce551127c45.backup.db8899c3-364f-425e-837b-0faa75339ff4' {{(pid=129839) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Dec 25 21:46:16.278793 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129839) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Dec 25 21:46:16.658730 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] Using --from-snap 'None' for incremental backup of volume 5b13f37b-b766-43c1-8d43-9ce551127c45. {{(pid=129839) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Dec 25 21:46:16.659633 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] Creating backup snapshot='backup.db8899c3-364f-425e-837b-0faa75339ff4.snap.1640468776.6592145' {{(pid=129839) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Dec 25 21:46:17.747483 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] Performing differential transfer from 'volume-5b13f37b-b766-43c1-8d43-9ce551127c45' to 'volume-5b13f37b-b766-43c1-8d43-9ce551127c45.backup.db8899c3-364f-425e-837b-0faa75339ff4' {{(pid=129839) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Dec 25 21:46:17.759454 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_jurc7uas --pool volumes volumes/volume-5b13f37b-b766-43c1-8d43-9ce551127c45@backup.db8899c3-364f-425e-837b-0faa75339ff4.snap.1640468776.6592145 -' into... {{(pid=129839) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Dec 25 21:46:17.759454 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-5b13f37b-b766-43c1-8d43-9ce551127c45.backup.db8899c3-364f-425e-837b-0faa75339ff4' {{(pid=129839) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Dec 25 21:46:19.314404 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] Differential backup transfer completed in 1.5666s {{(pid=129839) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Dec 25 21:46:19.314994 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] Getting metadata type 'volume-base-metadata' {{(pid=129839) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Dec 25 21:46:19.328778 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129839) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Dec 25 21:46:19.329248 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] Getting metadata type 'volume-metadata' {{(pid=129839) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Dec 25 21:46:19.335593 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] No metadata type 'volume-metadata' available {{(pid=129839) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Dec 25 21:46:19.335978 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] Getting metadata type 'volume-glance-metadata' {{(pid=129839) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Dec 25 21:46:19.342603 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] No metadata type 'volume-glance-metadata' available {{(pid=129839) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Dec 25 21:46:19.343156 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] Backing up metadata for volume 5b13f37b-b766-43c1-8d43-9ce551127c45. {{(pid=129839) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Dec 25 21:46:19.481718 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] Backup 'db8899c3-364f-425e-837b-0faa75339ff4' of volume 5b13f37b-b766-43c1-8d43-9ce551127c45 finished. {{(pid=129839) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Dec 25 21:46:19.484100 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-5b13f37b-b766-43c1-8d43-9ce551127c45', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '5b13f37b-b766-43c1-8d43-9ce551127c45', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, 'ignore_errors': True}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:46:19.934941 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] <== disconnect_volume: return (450ms) None {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:46:20.096612 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-4bc3ad18-a239-4469-bf70-595fcd5e8bbd tempest-VolumesBackupsAdminTest-869564114 None] Create backup finished. backup: db8899c3-364f-425e-837b-0faa75339ff4. Dec 25 21:46:20.422831 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-e052dd34-8625-42f1-9554-b24bd4e119fe tempest-VolumesBackupsAdminTest-1316713765 None] Export record started, backup: db8899c3-364f-425e-837b-0faa75339ff4. Dec 25 21:46:20.425301 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-e052dd34-8625-42f1-9554-b24bd4e119fe tempest-VolumesBackupsAdminTest-1316713765 None] Export record finished, backup db8899c3-364f-425e-837b-0faa75339ff4 exported. Dec 25 21:46:20.961485 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-994d374b-7335-411e-8715-614a2d1914e8 tempest-VolumesBackupsAdminTest-1316713765 None] Import record started, backup_url: eyJkcml2ZXJfaW5mbyI6IHt9LCAiaWQiOiAiZDE2MjM0MDAtMjZjYS00MGYzLWJmY2UtOGZiNzVkZTFiNTQwIiwgInVzZXJfaWQiOiAiYjQ1NjMzZjJhNzgwNDlmNmI3M2MxZjQ2ZTVlY2IxYTQiLCAicHJvamVjdF9pZCI6ICI5ZmE4NjMwZDJlNDM0OTdiYTJiNzY4YjQ4MGQ4M2E5OCIsICJ2b2x1bWVfaWQiOiAiNWIxM2YzN2ItYjc2Ni00M2MxLThkNDMtOWNlNTUxMTI3YzQ1IiwgImhvc3QiOiAibmNkdi1tYXN0ZXItMSIsICJhdmFpbGFiaWxpdHlfem9uZSI6IG51bGwsICJjb250YWluZXIiOiAiYmFja3VwcyIsICJwYXJlbnRfaWQiOiBudWxsLCAic3RhdHVzIjogImF2YWlsYWJsZSIsICJmYWlsX3JlYXNvbiI6IG51bGwsICJzaXplIjogMSwgImRpc3BsYXlfbmFtZSI6ICJ0ZW1wZXN0LVZvbHVtZXNCYWNrdXBzQWRtaW5UZXN0LUJhY2t1cC0xOTI0ODk0OTg1IiwgImRpc3BsYXlfZGVzY3JpcHRpb24iOiBudWxsLCAic2VydmljZV9tZXRhZGF0YSI6ICJ7XCJiYXNlXCI6IFwidm9sdW1lLTViMTNmMzdiLWI3NjYtNDNjMS04ZDQzLTljZTU1MTEyN2M0NS5iYWNrdXAuZGI4ODk5YzMtMzY0Zi00MjVlLTgzN2ItMGZhYTc1MzM5ZmY0XCJ9IiwgInNlcnZpY2UiOiAiY2luZGVyLmJhY2t1cC5kcml2ZXJzLmNlcGguQ2VwaEJhY2t1cERyaXZlciIsICJvYmplY3RfY291bnQiOiAwLCAidGVtcF92b2x1bWVfaWQiOiBudWxsLCAidGVtcF9zbmFwc2hvdF9pZCI6IG51bGwsICJudW1fZGVwZW5kZW50X2JhY2t1cHMiOiAwLCAic25hcHNob3RfaWQiOiBudWxsLCAiZGF0YV90aW1lc3RhbXAiOiAiMjAyMS0xMi0yNVQyMTo0NjoxNVoiLCAicmVzdG9yZV92b2x1bWVfaWQiOiBudWxsLCAibWV0YWRhdGEiOiB7fSwgImVuY3J5cHRpb25fa2V5X2lkIjogbnVsbCwgImNyZWF0ZWRfYXQiOiAiMjAyMS0xMi0yNVQyMTo0NjoxNVoiLCAidXBkYXRlZF9hdCI6ICIyMDIxLTEyLTI1VDIxOjQ2OjIwWiIsICJkZWxldGVkX2F0IjogbnVsbCwgImRlbGV0ZWQiOiBmYWxzZX0=. Dec 25 21:46:20.974319 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_db.sqlalchemy.engines [None req-994d374b-7335-411e-8715-614a2d1914e8 tempest-VolumesBackupsAdminTest-1316713765 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129839) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 25 21:46:21.151717 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-994d374b-7335-411e-8715-614a2d1914e8 tempest-VolumesBackupsAdminTest-1316713765 None] Import record id d1623400-26ca-40f3-bfce-8fb75de1b540 metadata from driver finished. Dec 25 21:46:23.685177 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] Restore backup started, backup: db8899c3-364f-425e-837b-0faa75339ff4 volume: fbff126a-a78a-41f1-84fc-eb484dd711c7. Dec 25 21:46:23.697334 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.utils [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.32', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:46:23.699467 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:46:23.716781 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.018s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:46:23.717160 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[ef184ffb-846b-4917-82d4-961a7ad58146]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:c2a322773d0\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:46:23.722414 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:46:23.726333 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Dec 25 21:46:23.727122 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[a51195e2-cf82-4796-9dd4-46b2b3a35d06]: [Errno 2] No such file or directory: 'systool' {{(pid=161909) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}} Dec 25 21:46:23.727122 ncdv-master-1 cinder-backup[129839]: Traceback (most recent call last): Dec 25 21:46:23.727122 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd Dec 25 21:46:23.727122 ncdv-master-1 cinder-backup[129839]: ret = func(*f_args, **f_kwargs) Dec 25 21:46:23.727122 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Dec 25 21:46:23.727122 ncdv-master-1 cinder-backup[129839]: return func(*args, **kwargs) Dec 25 21:46:23.727122 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Dec 25 21:46:23.727122 ncdv-master-1 cinder-backup[129839]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Dec 25 21:46:23.727122 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Dec 25 21:46:23.727122 ncdv-master-1 cinder-backup[129839]: return putils.execute(on_execute=on_execute, Dec 25 21:46:23.727122 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Dec 25 21:46:23.727122 ncdv-master-1 cinder-backup[129839]: obj = subprocess.Popen(cmd, Dec 25 21:46:23.727122 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Dec 25 21:46:23.727122 ncdv-master-1 cinder-backup[129839]: self._execute_child(args, executable, preexec_fn, close_fds, Dec 25 21:46:23.727122 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Dec 25 21:46:23.727122 ncdv-master-1 cinder-backup[129839]: raise child_exception_type(errno_num, err_msg, err_filename) Dec 25 21:46:23.727122 ncdv-master-1 cinder-backup[129839]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Dec 25 21:46:23.728352 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[a51195e2-cf82-4796-9dd4-46b2b3a35d06]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:46:23.731472 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:46:23.737305 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Dec 25 21:46:23.737599 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[cbf31964-9cd8-406b-8c56-7927fa6c0b59]: [Errno 2] No such file or directory: 'systool' {{(pid=161909) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}} Dec 25 21:46:23.737599 ncdv-master-1 cinder-backup[129839]: Traceback (most recent call last): Dec 25 21:46:23.737599 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd Dec 25 21:46:23.737599 ncdv-master-1 cinder-backup[129839]: ret = func(*f_args, **f_kwargs) Dec 25 21:46:23.737599 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Dec 25 21:46:23.737599 ncdv-master-1 cinder-backup[129839]: return func(*args, **kwargs) Dec 25 21:46:23.737599 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Dec 25 21:46:23.737599 ncdv-master-1 cinder-backup[129839]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Dec 25 21:46:23.737599 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Dec 25 21:46:23.737599 ncdv-master-1 cinder-backup[129839]: return putils.execute(on_execute=on_execute, Dec 25 21:46:23.737599 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Dec 25 21:46:23.737599 ncdv-master-1 cinder-backup[129839]: obj = subprocess.Popen(cmd, Dec 25 21:46:23.737599 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Dec 25 21:46:23.737599 ncdv-master-1 cinder-backup[129839]: self._execute_child(args, executable, preexec_fn, close_fds, Dec 25 21:46:23.737599 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Dec 25 21:46:23.737599 ncdv-master-1 cinder-backup[129839]: raise child_exception_type(errno_num, err_msg, err_filename) Dec 25 21:46:23.737599 ncdv-master-1 cinder-backup[129839]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Dec 25 21:46:23.738877 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[cbf31964-9cd8-406b-8c56-7927fa6c0b59]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:46:23.739765 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:46:23.754015 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.013s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:46:23.754236 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[24fc7201-4bb5-4765-a198-df51f189e72f]: (4, ('/dev/sda1\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:46:23.756069 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:46:23.769619 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.012s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:46:23.770012 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[d562ce9c-d7d1-438a-ad7e-409668a90e84]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:46:23.771631 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:46:23.780530 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.009s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:46:23.780969 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[970290bc-c2aa-434b-b966-b7c9effb9922]: (4, ('0b661f24-87f6-9248-8553-e9ec840a31d0\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:46:23.781843 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] Running cmd (subprocess): nvme version {{(pid=129839) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:46:23.801570 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] CMD "nvme version" returned: 0 in 0.019s {{(pid=129839) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:46:23.802753 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.utils [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] <== get_connector_properties: return (104ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.32', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:c2a322773d0', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '0b661f24-87f6-9248-8553-e9ec840a31d0', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:68cc0364-fba7-439f-bcd5-9f4beb715fab'} {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:46:24.521403 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connector [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] Factory for rbd on None {{(pid=129839) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}} Dec 25 21:46:24.521906 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-fbff126a-a78a-41f1-84fc-eb484dd711c7', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'fbff126a-a78a-41f1-84fc-eb484dd711c7', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:46:24.523051 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.linuxrbd [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] opening connection to ceph cluster (timeout=-1). {{(pid=129839) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Dec 25 21:46:24.729152 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] <== connect_volume: return (207ms) {'path': } {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:46:24.736949 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_db.sqlalchemy.engines [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129839) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 25 21:46:24.754455 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] Starting restore from Ceph backup=db8899c3-364f-425e-837b-0faa75339ff4 to volume=volume-fbff126a-a78a-41f1-84fc-eb484dd711c7 {{(pid=129839) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1274}} Dec 25 21:46:24.867526 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] Looking for snapshot of backup base 'volume-5b13f37b-b766-43c1-8d43-9ce551127c45.backup.db8899c3-364f-425e-837b-0faa75339ff4' {{(pid=129839) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Dec 25 21:46:24.867936 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] Found snapshot 'backup.db8899c3-364f-425e-837b-0faa75339ff4.snap.1640468776.6592145' {{(pid=129839) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Dec 25 21:46:24.875300 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] Volume file is RBD. {{(pid=129839) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1184}} Dec 25 21:46:24.918612 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] Attempting differential restore. {{(pid=129839) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1234}} Dec 25 21:46:24.922224 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] Attempting incremental restore from base='volume-5b13f37b-b766-43c1-8d43-9ce551127c45.backup.db8899c3-364f-425e-837b-0faa75339ff4' snap='backup.db8899c3-364f-425e-837b-0faa75339ff4.snap.1640468776.6592145' {{(pid=129839) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1089}} Dec 25 21:46:24.922224 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] Performing differential transfer from 'volume-5b13f37b-b766-43c1-8d43-9ce551127c45.backup.db8899c3-364f-425e-837b-0faa75339ff4' to 'volume-fbff126a-a78a-41f1-84fc-eb484dd711c7' {{(pid=129839) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Dec 25 21:46:24.922224 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-5b13f37b-b766-43c1-8d43-9ce551127c45.backup.db8899c3-364f-425e-837b-0faa75339ff4@backup.db8899c3-364f-425e-837b-0faa75339ff4.snap.1640468776.6592145 -' into... {{(pid=129839) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Dec 25 21:46:24.922224 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_kvd4zxhf --pool volumes - volumes/volume-fbff126a-a78a-41f1-84fc-eb484dd711c7' {{(pid=129839) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Dec 25 21:46:27.069887 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] Restore transfer completed in 2.1497s {{(pid=129839) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1111}} Dec 25 21:46:27.070821 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=129839) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1285}} Dec 25 21:46:27.105971 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] Restoring volume base metadata {{(pid=129839) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} Dec 25 21:46:27.162883 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] No metadata of type 'volume-metadata' to restore {{(pid=129839) put /opt/stack/cinder/cinder/backup/driver.py:345}} Dec 25 21:46:27.163268 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] No metadata of type 'volume-glance-metadata' to restore {{(pid=129839) put /opt/stack/cinder/cinder/backup/driver.py:345}} Dec 25 21:46:27.167204 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] Restore to volume fbff126a-a78a-41f1-84fc-eb484dd711c7 finished successfully. {{(pid=129839) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1292}} Dec 25 21:46:27.168797 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-fbff126a-a78a-41f1-84fc-eb484dd711c7', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'fbff126a-a78a-41f1-84fc-eb484dd711c7', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, 'ignore_errors': False}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:46:27.250150 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] <== disconnect_volume: return (80ms) None {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:46:27.301573 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.manager [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] Encryption key ID for volume fbff126a-a78a-41f1-84fc-eb484dd711c7 already matches encryption key ID in backup db8899c3-364f-425e-837b-0faa75339ff4. {{(pid=129839) _run_restore /opt/stack/cinder/cinder/backup/manager.py:801}} Dec 25 21:46:27.546182 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-09d8e2da-4a5b-4d01-8cc8-657637f6875c tempest-VolumesBackupsAdminTest-869564114 None] Finished restoring backup db8899c3-364f-425e-837b-0faa75339ff4 to volume fbff126a-a78a-41f1-84fc-eb484dd711c7. Dec 25 21:46:28.307297 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-564394ef-d397-42d7-a801-74c7f6226268 tempest-VolumesBackupsAdminTest-1316713765 None] Delete backup started, backup: d1623400-26ca-40f3-bfce-8fb75de1b540. Dec 25 21:46:28.309860 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-564394ef-d397-42d7-a801-74c7f6226268 tempest-VolumesBackupsAdminTest-1316713765 None] Delete started for backup=d1623400-26ca-40f3-bfce-8fb75de1b540 {{(pid=129839) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Dec 25 21:46:28.310512 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-564394ef-d397-42d7-a801-74c7f6226268 tempest-VolumesBackupsAdminTest-1316713765 None] Trying diff format basename='volume-5b13f37b-b766-43c1-8d43-9ce551127c45.backup.db8899c3-364f-425e-837b-0faa75339ff4' for backup base image of volume 5b13f37b-b766-43c1-8d43-9ce551127c45. {{(pid=129839) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Dec 25 21:46:28.563491 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-564394ef-d397-42d7-a801-74c7f6226268 tempest-VolumesBackupsAdminTest-1316713765 None] Looking for snapshot of backup base 'volume-5b13f37b-b766-43c1-8d43-9ce551127c45.backup.db8899c3-364f-425e-837b-0faa75339ff4' {{(pid=129839) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Dec 25 21:46:28.564457 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-564394ef-d397-42d7-a801-74c7f6226268 tempest-VolumesBackupsAdminTest-1316713765 None] Backup 'd1623400-26ca-40f3-bfce-8fb75de1b540' has no snapshot {{(pid=129839) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:915}} Dec 25 21:46:28.565051 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-564394ef-d397-42d7-a801-74c7f6226268 tempest-VolumesBackupsAdminTest-1316713765 None] No backup snapshot to delete {{(pid=129839) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:476}} Dec 25 21:46:28.757707 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.drivers.ceph [None req-564394ef-d397-42d7-a801-74c7f6226268 tempest-VolumesBackupsAdminTest-1316713765 None] Backup base image of volume 5b13f37b-b766-43c1-8d43-9ce551127c45 still has 1 snapshots so skipping base image delete. Dec 25 21:46:28.932321 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-564394ef-d397-42d7-a801-74c7f6226268 tempest-VolumesBackupsAdminTest-1316713765 None] Metadata backup object 'backup.d1623400-26ca-40f3-bfce-8fb75de1b540.meta' not found - ignoring {{(pid=129839) remove_if_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:160}} Dec 25 21:46:28.936857 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-564394ef-d397-42d7-a801-74c7f6226268 tempest-VolumesBackupsAdminTest-1316713765 None] Delete of backup 'd1623400-26ca-40f3-bfce-8fb75de1b540' for volume '5b13f37b-b766-43c1-8d43-9ce551127c45' finished. {{(pid=129839) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Dec 25 21:46:28.956059 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_db.sqlalchemy.engines [None req-564394ef-d397-42d7-a801-74c7f6226268 tempest-VolumesBackupsAdminTest-1316713765 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129839) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 25 21:46:29.054990 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.quota [None req-564394ef-d397-42d7-a801-74c7f6226268 tempest-VolumesBackupsAdminTest-1316713765 None] Created reservations ['1ae58dcf-5202-4505-87ee-4fd40195c859', 'd7980a08-30e0-40c2-ac5f-078bfa7b4aca'] {{(pid=129839) reserve /opt/stack/cinder/cinder/quota.py:755}} Dec 25 21:46:29.202741 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-564394ef-d397-42d7-a801-74c7f6226268 tempest-VolumesBackupsAdminTest-1316713765 None] Delete backup finished, backup d1623400-26ca-40f3-bfce-8fb75de1b540 deleted. Dec 25 21:46:29.519099 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-e488668d-74e5-45c5-af76-4ddc0cc8a053 tempest-VolumesBackupsAdminTest-869564114 None] Delete backup started, backup: db8899c3-364f-425e-837b-0faa75339ff4. Dec 25 21:46:29.520950 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-e488668d-74e5-45c5-af76-4ddc0cc8a053 tempest-VolumesBackupsAdminTest-869564114 None] Delete started for backup=db8899c3-364f-425e-837b-0faa75339ff4 {{(pid=129839) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Dec 25 21:46:29.521711 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-e488668d-74e5-45c5-af76-4ddc0cc8a053 tempest-VolumesBackupsAdminTest-869564114 None] Trying diff format basename='volume-5b13f37b-b766-43c1-8d43-9ce551127c45.backup.db8899c3-364f-425e-837b-0faa75339ff4' for backup base image of volume 5b13f37b-b766-43c1-8d43-9ce551127c45. {{(pid=129839) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Dec 25 21:46:29.705983 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-e488668d-74e5-45c5-af76-4ddc0cc8a053 tempest-VolumesBackupsAdminTest-869564114 None] Looking for snapshot of backup base 'volume-5b13f37b-b766-43c1-8d43-9ce551127c45.backup.db8899c3-364f-425e-837b-0faa75339ff4' {{(pid=129839) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Dec 25 21:46:29.706850 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-e488668d-74e5-45c5-af76-4ddc0cc8a053 tempest-VolumesBackupsAdminTest-869564114 None] Found snapshot 'backup.db8899c3-364f-425e-837b-0faa75339ff4.snap.1640468776.6592145' {{(pid=129839) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Dec 25 21:46:29.707255 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-e488668d-74e5-45c5-af76-4ddc0cc8a053 tempest-VolumesBackupsAdminTest-869564114 None] Deleting backup snapshot='backup.db8899c3-364f-425e-837b-0faa75339ff4.snap.1640468776.6592145' {{(pid=129839) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Dec 25 21:46:32.475583 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.drivers.ceph [None req-e488668d-74e5-45c5-af76-4ddc0cc8a053 tempest-VolumesBackupsAdminTest-869564114 None] Deleting backup base image='volume-5b13f37b-b766-43c1-8d43-9ce551127c45.backup.db8899c3-364f-425e-837b-0faa75339ff4' of volume 5b13f37b-b766-43c1-8d43-9ce551127c45. Dec 25 21:46:37.380181 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_db.sqlalchemy.engines [None req-9b4bfa17-f8eb-43dd-9d64-d05efc7c4b62 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129839) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 25 21:46:37.719298 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-e488668d-74e5-45c5-af76-4ddc0cc8a053 tempest-VolumesBackupsAdminTest-869564114 None] Base backup image='volume-5b13f37b-b766-43c1-8d43-9ce551127c45.backup.db8899c3-364f-425e-837b-0faa75339ff4' of volume 5b13f37b-b766-43c1-8d43-9ce551127c45 deleted. {{(pid=129839) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Dec 25 21:46:37.887588 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-e488668d-74e5-45c5-af76-4ddc0cc8a053 tempest-VolumesBackupsAdminTest-869564114 None] Delete of backup 'db8899c3-364f-425e-837b-0faa75339ff4' for volume '5b13f37b-b766-43c1-8d43-9ce551127c45' finished. {{(pid=129839) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Dec 25 21:46:38.019829 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.quota [None req-e488668d-74e5-45c5-af76-4ddc0cc8a053 tempest-VolumesBackupsAdminTest-869564114 None] Created reservations ['1bf1b090-1b0d-4074-88fd-248518cb7759', 'a1afaaec-d5b3-4cb9-91a7-1daa5f6b2b15'] {{(pid=129839) reserve /opt/stack/cinder/cinder/quota.py:755}} Dec 25 21:46:38.168386 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-e488668d-74e5-45c5-af76-4ddc0cc8a053 tempest-VolumesBackupsAdminTest-869564114 None] Delete backup finished, backup db8899c3-364f-425e-837b-0faa75339ff4 deleted. Dec 25 21:46:42.373010 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] Create backup started, backup: ad45f069-5558-4ff8-973d-d216ab3fb269 volume: cd13dd2c-fc3f-43c6-bd53-e46cb8d21cd6. Dec 25 21:46:42.450273 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.utils [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.32', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:46:42.457370 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:46:42.484946 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.032s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:46:42.486041 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[a2f4468b-a7d1-444a-94c2-85f7d0412fc5]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:c2a322773d0\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:46:42.488691 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:46:42.501089 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Dec 25 21:46:42.510987 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[b6b142a3-8f78-436b-ad7b-df535ad0538c]: [Errno 2] No such file or directory: 'systool' {{(pid=161909) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}} Dec 25 21:46:42.510987 ncdv-master-1 cinder-backup[129839]: Traceback (most recent call last): Dec 25 21:46:42.510987 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd Dec 25 21:46:42.510987 ncdv-master-1 cinder-backup[129839]: ret = func(*f_args, **f_kwargs) Dec 25 21:46:42.510987 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Dec 25 21:46:42.510987 ncdv-master-1 cinder-backup[129839]: return func(*args, **kwargs) Dec 25 21:46:42.510987 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Dec 25 21:46:42.510987 ncdv-master-1 cinder-backup[129839]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Dec 25 21:46:42.510987 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Dec 25 21:46:42.510987 ncdv-master-1 cinder-backup[129839]: return putils.execute(on_execute=on_execute, Dec 25 21:46:42.510987 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Dec 25 21:46:42.510987 ncdv-master-1 cinder-backup[129839]: obj = subprocess.Popen(cmd, Dec 25 21:46:42.510987 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Dec 25 21:46:42.510987 ncdv-master-1 cinder-backup[129839]: self._execute_child(args, executable, preexec_fn, close_fds, Dec 25 21:46:42.510987 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Dec 25 21:46:42.510987 ncdv-master-1 cinder-backup[129839]: raise child_exception_type(errno_num, err_msg, err_filename) Dec 25 21:46:42.510987 ncdv-master-1 cinder-backup[129839]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Dec 25 21:46:42.510987 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[b6b142a3-8f78-436b-ad7b-df535ad0538c]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:46:42.513507 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:46:42.521363 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} Dec 25 21:46:42.521897 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[d442c755-d66f-4f8a-9471-3037d5d09d83]: [Errno 2] No such file or directory: 'systool' {{(pid=161909) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:480}} Dec 25 21:46:42.521897 ncdv-master-1 cinder-backup[129839]: Traceback (most recent call last): Dec 25 21:46:42.521897 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 477, in _process_cmd Dec 25 21:46:42.521897 ncdv-master-1 cinder-backup[129839]: ret = func(*f_args, **f_kwargs) Dec 25 21:46:42.521897 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap Dec 25 21:46:42.521897 ncdv-master-1 cinder-backup[129839]: return func(*args, **kwargs) Dec 25 21:46:42.521897 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root Dec 25 21:46:42.521897 ncdv-master-1 cinder-backup[129839]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) Dec 25 21:46:42.521897 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute Dec 25 21:46:42.521897 ncdv-master-1 cinder-backup[129839]: return putils.execute(on_execute=on_execute, Dec 25 21:46:42.521897 ncdv-master-1 cinder-backup[129839]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute Dec 25 21:46:42.521897 ncdv-master-1 cinder-backup[129839]: obj = subprocess.Popen(cmd, Dec 25 21:46:42.521897 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ Dec 25 21:46:42.521897 ncdv-master-1 cinder-backup[129839]: self._execute_child(args, executable, preexec_fn, close_fds, Dec 25 21:46:42.521897 ncdv-master-1 cinder-backup[129839]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child Dec 25 21:46:42.521897 ncdv-master-1 cinder-backup[129839]: raise child_exception_type(errno_num, err_msg, err_filename) Dec 25 21:46:42.521897 ncdv-master-1 cinder-backup[129839]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' Dec 25 21:46:42.523612 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[d442c755-d66f-4f8a-9471-3037d5d09d83]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:46:42.536553 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:46:42.593417 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.066s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:46:42.596446 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[f168363c-e0cd-42db-98ba-ba895640c25e]: (4, ('/dev/sda1\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:46:42.597324 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:46:42.623548 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.028s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:46:42.624775 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[5200c427-eec0-4018-822b-e5df3390cca1]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:46:42.627586 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:46:42.658315 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.030s {{(pid=161909) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:46:42.659262 ncdv-master-1 cinder-backup[129839]: DEBUG oslo.privsep.daemon [-] privsep: reply[d2a13672-8161-4996-a75a-3a502712f0f4]: (4, ('0b661f24-87f6-9248-8553-e9ec840a31d0\n', '')) {{(pid=161909) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:502}} Dec 25 21:46:42.660178 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] Running cmd (subprocess): nvme version {{(pid=129839) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} Dec 25 21:46:42.730351 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_concurrency.processutils [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] CMD "nvme version" returned: 0 in 0.069s {{(pid=129839) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} Dec 25 21:46:42.731712 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.utils [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] <== get_connector_properties: return (280ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.32', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:c2a322773d0', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '0b661f24-87f6-9248-8553-e9ec840a31d0', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:68cc0364-fba7-439f-bcd5-9f4beb715fab'} {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:46:43.106608 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:46:43.107580 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:46:43.882153 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connector [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] Factory for rbd on None {{(pid=129839) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:281}} Dec 25 21:46:43.882153 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-cd13dd2c-fc3f-43c6-bd53-e46cb8d21cd6', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'cd13dd2c-fc3f-43c6-bd53-e46cb8d21cd6', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:46:43.882770 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.linuxrbd [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] opening connection to ceph cluster (timeout=-1). {{(pid=129839) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} Dec 25 21:46:44.083619 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] <== connect_volume: return (201ms) {'path': } {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:46:44.092927 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_db.sqlalchemy.engines [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129839) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 25 21:46:44.203661 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] Starting backup of volume='cd13dd2c-fc3f-43c6-bd53-e46cb8d21cd6'. {{(pid=129839) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:974}} Dec 25 21:46:44.204720 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] Volume file is RBD: attempting optimized backup {{(pid=129839) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:985}} Dec 25 21:46:44.301437 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] Creating base image 'volume-cd13dd2c-fc3f-43c6-bd53-e46cb8d21cd6.backup.ad45f069-5558-4ff8-973d-d216ab3fb269' {{(pid=129839) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} Dec 25 21:46:44.301864 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=129839) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:272}} Dec 25 21:46:44.806692 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] Using --from-snap 'None' for incremental backup of volume cd13dd2c-fc3f-43c6-bd53-e46cb8d21cd6. {{(pid=129839) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:764}} Dec 25 21:46:44.807468 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] Creating backup snapshot='backup.ad45f069-5558-4ff8-973d-d216ab3fb269.snap.1640468804.8070328' {{(pid=129839) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:770}} Dec 25 21:46:46.707143 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] Performing differential transfer from 'volume-cd13dd2c-fc3f-43c6-bd53-e46cb8d21cd6' to 'volume-cd13dd2c-fc3f-43c6-bd53-e46cb8d21cd6.backup.ad45f069-5558-4ff8-973d-d216ab3fb269' {{(pid=129839) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:624}} Dec 25 21:46:46.707855 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_q8nu8xbw --pool volumes volumes/volume-cd13dd2c-fc3f-43c6-bd53-e46cb8d21cd6@backup.ad45f069-5558-4ff8-973d-d216ab3fb269.snap.1640468804.8070328 -' into... {{(pid=129839) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:585}} Dec 25 21:46:46.708496 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-cd13dd2c-fc3f-43c6-bd53-e46cb8d21cd6.backup.ad45f069-5558-4ff8-973d-d216ab3fb269' {{(pid=129839) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:586}} Dec 25 21:46:49.308567 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] Differential backup transfer completed in 2.6012s {{(pid=129839) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:790}} Dec 25 21:46:49.309300 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] Getting metadata type 'volume-base-metadata' {{(pid=129839) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} Dec 25 21:46:49.321821 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=129839) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} Dec 25 21:46:49.322458 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] Getting metadata type 'volume-metadata' {{(pid=129839) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} Dec 25 21:46:49.329695 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] No metadata type 'volume-metadata' available {{(pid=129839) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} Dec 25 21:46:49.330240 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] Getting metadata type 'volume-glance-metadata' {{(pid=129839) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} Dec 25 21:46:49.338658 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.driver [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] No metadata type 'volume-glance-metadata' available {{(pid=129839) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} Dec 25 21:46:49.339626 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] Backing up metadata for volume cd13dd2c-fc3f-43c6-bd53-e46cb8d21cd6. {{(pid=129839) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:951}} Dec 25 21:46:49.540092 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] Backup 'ad45f069-5558-4ff8-973d-d216ab3fb269' of volume cd13dd2c-fc3f-43c6-bd53-e46cb8d21cd6 finished. {{(pid=129839) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1015}} Dec 25 21:46:49.541661 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-cd13dd2c-fc3f-43c6-bd53-e46cb8d21cd6', 'hosts': ['10.222.0.32'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'cd13dd2c-fc3f-43c6-bd53-e46cb8d21cd6', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, 'ignore_errors': True}" {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:157}} Dec 25 21:46:49.882785 ncdv-master-1 cinder-backup[129839]: DEBUG os_brick.initiator.connectors.rbd [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] <== disconnect_volume: return (340ms) None {{(pid=129839) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:184}} Dec 25 21:46:50.110908 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-0b6715ab-57ca-431e-b787-a828aacc0fc2 tempest-VolumesBackupsAdminTest-869564114 None] Create backup finished. backup: ad45f069-5558-4ff8-973d-d216ab3fb269. Dec 25 21:46:50.718177 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-bcaace05-f2e9-4245-8132-59f91b09ef64 tempest-VolumesBackupsAdminTest-1316713765 None] Reset backup status started, backup_id: ad45f069-5558-4ff8-973d-d216ab3fb269, status: error. Dec 25 21:46:50.718742 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-bcaace05-f2e9-4245-8132-59f91b09ef64 tempest-VolumesBackupsAdminTest-1316713765 None] Backup service: cinder.backup.drivers.ceph.CephBackupDriver. Dec 25 21:46:51.922875 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-10714222-7b39-45a9-aa13-28403a20e2c6 tempest-VolumesBackupsAdminTest-869564114 None] Delete backup started, backup: ad45f069-5558-4ff8-973d-d216ab3fb269. Dec 25 21:46:51.937748 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-10714222-7b39-45a9-aa13-28403a20e2c6 tempest-VolumesBackupsAdminTest-869564114 None] Delete started for backup=ad45f069-5558-4ff8-973d-d216ab3fb269 {{(pid=129839) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1301}} Dec 25 21:46:51.937748 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-10714222-7b39-45a9-aa13-28403a20e2c6 tempest-VolumesBackupsAdminTest-869564114 None] Trying diff format basename='volume-cd13dd2c-fc3f-43c6-bd53-e46cb8d21cd6.backup.ad45f069-5558-4ff8-973d-d216ab3fb269' for backup base image of volume cd13dd2c-fc3f-43c6-bd53-e46cb8d21cd6. {{(pid=129839) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:513}} Dec 25 21:46:52.111564 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-10714222-7b39-45a9-aa13-28403a20e2c6 tempest-VolumesBackupsAdminTest-869564114 None] Looking for snapshot of backup base 'volume-cd13dd2c-fc3f-43c6-bd53-e46cb8d21cd6.backup.ad45f069-5558-4ff8-973d-d216ab3fb269' {{(pid=129839) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:905}} Dec 25 21:46:52.112699 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-10714222-7b39-45a9-aa13-28403a20e2c6 tempest-VolumesBackupsAdminTest-869564114 None] Found snapshot 'backup.ad45f069-5558-4ff8-973d-d216ab3fb269.snap.1640468804.8070328' {{(pid=129839) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:924}} Dec 25 21:46:52.113648 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-10714222-7b39-45a9-aa13-28403a20e2c6 tempest-VolumesBackupsAdminTest-869564114 None] Deleting backup snapshot='backup.ad45f069-5558-4ff8-973d-d216ab3fb269.snap.1640468804.8070328' {{(pid=129839) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:473}} Dec 25 21:46:53.843289 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.drivers.ceph [None req-10714222-7b39-45a9-aa13-28403a20e2c6 tempest-VolumesBackupsAdminTest-869564114 None] Deleting backup base image='volume-cd13dd2c-fc3f-43c6-bd53-e46cb8d21cd6.backup.ad45f069-5558-4ff8-973d-d216ab3fb269' of volume cd13dd2c-fc3f-43c6-bd53-e46cb8d21cd6. Dec 25 21:46:56.849365 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-10714222-7b39-45a9-aa13-28403a20e2c6 tempest-VolumesBackupsAdminTest-869564114 None] Base backup image='volume-cd13dd2c-fc3f-43c6-bd53-e46cb8d21cd6.backup.ad45f069-5558-4ff8-973d-d216ab3fb269' of volume cd13dd2c-fc3f-43c6-bd53-e46cb8d21cd6 deleted. {{(pid=129839) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:561}} Dec 25 21:46:57.000074 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.backup.drivers.ceph [None req-10714222-7b39-45a9-aa13-28403a20e2c6 tempest-VolumesBackupsAdminTest-869564114 None] Delete of backup 'ad45f069-5558-4ff8-973d-d216ab3fb269' for volume 'cd13dd2c-fc3f-43c6-bd53-e46cb8d21cd6' finished. {{(pid=129839) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1329}} Dec 25 21:46:57.005895 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_db.sqlalchemy.engines [None req-10714222-7b39-45a9-aa13-28403a20e2c6 tempest-VolumesBackupsAdminTest-869564114 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=129839) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Dec 25 21:46:57.105257 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.quota [None req-10714222-7b39-45a9-aa13-28403a20e2c6 tempest-VolumesBackupsAdminTest-869564114 None] Created reservations ['fe6f0a5b-32b2-47ac-adad-17502d79bc62', 'c03de5e8-c00f-42a2-8d17-8dd5f251c153'] {{(pid=129839) reserve /opt/stack/cinder/cinder/quota.py:755}} Dec 25 21:46:57.258654 ncdv-master-1 cinder-backup[129839]: INFO cinder.backup.manager [None req-10714222-7b39-45a9-aa13-28403a20e2c6 tempest-VolumesBackupsAdminTest-869564114 None] Delete backup finished, backup ad45f069-5558-4ff8-973d-d216ab3fb269 deleted. Dec 25 21:48:43.104412 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:48:43.105149 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:49:43.109401 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:49:43.110156 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:50:43.114471 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:50:43.115133 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:51:43.120532 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:51:43.121213 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:52:43.125706 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:52:43.126658 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:53:43.130651 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:53:43.131317 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:54:43.135831 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:54:43.136471 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:55:43.141890 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:55:43.142592 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:56:43.137442 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:56:43.138050 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:57:43.142221 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:57:43.142944 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:58:43.138768 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:58:43.139468 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 21:59:43.144693 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 21:59:43.145262 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:00:43.140609 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:00:43.141295 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:01:43.145523 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:01:43.146135 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:02:43.150728 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:02:43.151273 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:03:43.155855 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:03:43.156504 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:04:43.160215 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:04:43.160948 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:05:43.165268 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:05:43.165936 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:06:43.169969 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:06:43.170725 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:07:43.175646 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:07:43.176398 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:08:43.179912 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:08:43.180514 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:09:43.184800 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:09:43.185493 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:10:43.190937 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:10:43.191700 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:11:43.195284 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:11:43.195992 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:12:43.200052 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:12:43.200787 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:13:43.205000 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:13:43.205648 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:14:43.202751 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:14:43.203389 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:15:43.207791 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:15:43.208429 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:16:43.208787 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:16:43.213061 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:17:43.214289 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:17:43.214865 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:18:43.210483 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:18:43.211087 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:19:43.205897 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:19:43.206590 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:20:43.203569 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:20:43.204192 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:21:43.199940 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:21:43.200570 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:22:43.205434 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:22:43.206051 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:23:43.210387 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:23:43.211023 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:24:43.206849 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:24:43.207540 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:25:43.202459 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:25:43.203021 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:26:43.199047 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:26:43.199594 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:27:43.204524 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:27:43.205069 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:28:43.200550 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:28:43.201110 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:29:43.205281 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:29:43.214193 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:30:43.209932 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:30:43.210500 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:31:43.214534 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:31:43.215158 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:32:43.219562 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:32:43.220197 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:33:43.223986 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:33:43.224661 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:34:43.219637 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:34:43.220302 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:35:43.225127 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:35:43.225771 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:36:43.230237 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:36:43.230878 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} Dec 25 22:37:43.234626 ncdv-master-1 cinder-backup[129839]: DEBUG oslo_service.periodic_task [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=129839) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} Dec 25 22:37:43.235367 ncdv-master-1 cinder-backup[129839]: DEBUG cinder.manager [None req-6e4b981b-ea37-403e-ad8b-0f68182c3a5e None None] Notifying Schedulers of capabilities ... {{(pid=129839) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}