-- Logs begin at Mon 2022-05-02 09:16:05 UTC, end at Mon 2022-05-02 11:53:34 UTC. -- May 02 10:09:35.663254 ncdv-master-1 systemd[1]: Started Devstack devstack@c-bak.service. May 02 10:09:39.056388 ncdv-master-1 cinder-backup[123252]: INFO cinder.cmd.backup [-] Backup running with 1 processes. May 02 10:09:39.068183 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_concurrency.lockutils [-] Acquired lock "singleton_lock" {{(pid=123252) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:294}} May 02 10:09:39.068183 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_concurrency.lockutils [-] Releasing lock "singleton_lock" {{(pid=123252) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:312}} May 02 10:09:39.258546 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_db.sqlalchemy.engines [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123252) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 02 10:09:39.487950 ncdv-master-1 cinder-backup[123252]: INFO cinder.rpc [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] Automatically selected cinder-backup objects version 1.39 as minimum service version. May 02 10:09:39.492773 ncdv-master-1 cinder-backup[123252]: INFO cinder.rpc [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] Automatically selected cinder-backup RPC version 2.3 as minimum service version. May 02 10:09:39.497357 ncdv-master-1 cinder-backup[123252]: INFO cinder.rpc [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] Automatically selected cinder-volume objects version 1.39 as minimum service version. May 02 10:09:39.501952 ncdv-master-1 cinder-backup[123252]: INFO cinder.rpc [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] Automatically selected cinder-volume RPC version 3.18 as minimum service version. May 02 10:09:39.507184 ncdv-master-1 cinder-backup[123252]: INFO cinder.rpc [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] Automatically selected cinder-scheduler objects version 1.39 as minimum service version. May 02 10:09:39.514430 ncdv-master-1 cinder-backup[123252]: INFO cinder.rpc [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] Automatically selected cinder-scheduler RPC version 3.12 as minimum service version. May 02 10:09:39.571250 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_db.sqlalchemy.engines [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123252) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 02 10:09:39.728146 ncdv-master-1 cinder-backup[123252]: INFO oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] Starting 1 workers May 02 10:09:39.734774 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] Started child 123698 {{(pid=123252) _start_child /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:575}} May 02 10:09:39.736202 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] Full set of CONF: {{(pid=123252) wait /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:649}} May 02 10:09:39.741755 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] ******************************************************************************** {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} May 02 10:09:39.741755 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] Configuration options gathered from: {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} May 02 10:09:39.741755 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] command line args: ['--config-file', '/etc/cinder/cinder.conf'] {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} May 02 10:09:39.741755 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] config files: ['/etc/cinder/cinder.conf'] {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} May 02 10:09:39.741755 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] ================================================================================ {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} May 02 10:09:39.741755 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] allow_availability_zone_fallback = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.744179 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] allow_compression_on_image_upload = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.744179 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] allowed_direct_url_schemes = [] {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.744179 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] api_rate_limit = True {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.744179 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] auth_strategy = keystone {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.744179 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_availability_zone = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.744179 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backup_api_class = cinder.backup.api.API {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.744179 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backup_ceph_chunk_size = 134217728 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.745624 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backup_ceph_conf = /etc/ceph/ceph.conf {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.747844 ncdv-master-1 cinder-backup[123698]: INFO cinder.service [-] Starting cinder-backup node (version 20.1.0) May 02 10:09:39.748378 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backup_ceph_image_journals = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.748378 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backup_ceph_pool = backups {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.748378 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backup_ceph_stripe_count = 0 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.748378 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backup_ceph_stripe_unit = 0 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.748378 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backup_ceph_user = cinder-bak {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.748378 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backup_driver = cinder.backup.drivers.ceph.CephBackupDriver {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.753956 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backup_driver_init_check_interval = 60 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.753956 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backup_driver_stats_polling_interval = 60 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.753956 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backup_manager = cinder.backup.manager.BackupManager {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.753956 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backup_max_operations = 15 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.753956 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backup_metadata_version = 2 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.753956 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backup_name_template = backup-%s {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.754366 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backup_native_threads_pool_size = 60 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.754366 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backup_object_number_per_notification = 10 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.754366 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backup_service_inithost_offload = True {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.754366 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backup_timer_interval = 120 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.754366 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backup_use_same_host = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.754366 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backup_use_temp_snapshot = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.754366 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backup_workers = 1 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.754952 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] chap_password = **** {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.754952 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] chap_username = {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.754952 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.754952 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cinder_internal_tenant_project_id = 972cae0aa9694d6cad22fbdbcf916ff7 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.754952 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cinder_internal_tenant_user_id = 8eab371b02c34a13b629eacdf504dbdb {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.754952 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] compression_format = gzip {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.755360 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] config_dir = [] {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.755360 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] config_file = ['/etc/cinder/cinder.conf'] {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.755524 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] config_source = [] {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.755990 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] consistencygroup_api_class = cinder.consistencygroup.api.API {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.756784 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] control_exchange = openstack {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.757386 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] debug = True {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.757901 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] default_availability_zone = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.758526 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] default_group_type = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.759391 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'swiftclient=WARN', 'botocore=WARN'] {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.760991 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] default_volume_type = ceph {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.761530 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] driver_client_cert = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.762251 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] driver_client_cert_key = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.763467 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] driver_data_namespace = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.764052 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] driver_ssl_cert_path = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.764673 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] driver_ssl_cert_verify = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.765227 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] driver_use_ssl = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.765726 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] enable_new_services = True {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.766537 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] enable_unsupported_driver = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.767128 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] enabled_backends = ['ceph'] {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.767669 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] enforce_multipath_for_image_xfer = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.768185 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] filter_function = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.768801 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] glance_api_insecure = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.769362 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] glance_api_servers = ['http://10.222.0.16/image'] {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.769901 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] glance_api_ssl_compression = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.770835 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] glance_ca_certificates_file = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.771231 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] glance_catalog_info = image:glance:publicURL {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.771641 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] glance_certfile = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.772410 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] glance_core_properties = ['checksum', 'container_format', 'disk_format', 'image_name', 'image_id', 'min_disk', 'min_ram', 'name', 'size'] {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.772949 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] glance_keyfile = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.776584 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] glance_num_retries = 3 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.776584 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] glance_request_timeout = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.776584 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] goodness_function = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.776584 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] graceful_shutdown_timeout = 5 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.776584 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] group_api_class = cinder.group.api.API {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.776584 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] host = ncdv-master-1 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.783693 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] iet_conf = /etc/iet/ietd.conf {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.783693 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] image_compress_on_upload = True {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.783693 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] image_conversion_address_space_limit = 1 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.783693 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] image_conversion_cpu_limit = 60 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.783693 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] image_conversion_dir = /opt/stack/data/cinder/conversion {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.783693 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] image_upload_use_cinder_backend = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.786925 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] image_upload_use_internal_tenant = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.786925 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] image_volume_cache_enabled = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.786925 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] image_volume_cache_max_count = 0 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.786925 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] image_volume_cache_max_size_gb = 0 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.786925 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] instance_format = [instance: %(uuid)s] {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.786925 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] instance_uuid_format = [instance: %(uuid)s] {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.792745 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] iscsi_iotype = fileio {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.792745 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] iscsi_secondary_ip_addresses = [] {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.792745 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] iscsi_target_flags = {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.792745 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] iscsi_write_cache = on {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.792745 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] iser_helper = tgtadm {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.792745 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] iser_ip_address = 10.222.0.16 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.792745 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] iser_port = 3260 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.903302 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.903302 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] log_config_append = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.903302 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.903302 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] log_dir = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.903302 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] log_file = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.903302 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] log_options = True {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.903302 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] log_rotate_interval = 1 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.916689 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] log_rotate_interval_type = days {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.916689 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] log_rotation_type = none {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.916689 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.916689 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.916689 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.917143 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.917143 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] logging_user_identity_format = %(user)s %(project)s %(domain)s %(user_domain)s %(project_domain)s {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.917143 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] max_age = 0 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.917143 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] max_logfile_count = 30 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.917143 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] max_logfile_size_mb = 200 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.917143 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] max_over_subscription_ratio = 20.0 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.917611 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] message_reap_interval = 86400 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.917611 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] message_ttl = 2592000 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.917611 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] monkey_patch = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.917611 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] monkey_patch_modules = [] {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.917611 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] my_ip = 10.222.0.16 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.917611 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] no_snapshot_gb_quota = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.917611 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] num_iser_scan_tries = 3 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.918169 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] num_shell_tries = 3 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.918169 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] num_volume_device_scan_tries = 3 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.918169 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] nvmet_ns_id = 10 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.918169 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] nvmet_port_id = 1 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.918169 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] osapi_max_limit = 1000 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.918169 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] osapi_volume_ext_list = [] {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.918169 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] osapi_volume_extension = ['cinder.api.contrib.standard_extensions'] {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.918673 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] osapi_volume_listen = 0.0.0.0 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.918673 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] osapi_volume_listen_port = 8776 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.918673 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] osapi_volume_use_ssl = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.918673 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] osapi_volume_workers = 2 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.918673 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] per_volume_size_limit = -1 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.918673 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] periodic_fuzzy_delay = 60 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.918673 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] periodic_interval = 60 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.919138 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] project_id_regex = [0-9a-f\-]+ {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.919138 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] public_endpoint = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.919138 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] publish_errors = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.919138 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] quota_backup_gigabytes = 1000 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.919405 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.lockutils [None req-8d710a85-3490-4f35-bab8-84b1c2c94066 None None] Lock "cleanup_incomplete_backups_123252" acquired by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: waited 0.007s {{(pid=123698) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:386}} May 02 10:09:39.919503 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] quota_backups = 10 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.919586 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-8d710a85-3490-4f35-bab8-84b1c2c94066 None None] Cleaning up incomplete backup operations. May 02 10:09:39.919666 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] quota_consistencygroups = 10 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.919666 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] quota_driver = cinder.quota.DbQuotaDriver {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.919666 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] quota_gigabytes = 1000 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.919666 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] quota_groups = 10 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.919666 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] quota_snapshots = 10 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.919666 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] quota_volumes = 10 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.919666 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] rate_limit_burst = 0 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.920737 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] rate_limit_except_level = CRITICAL {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.920737 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] rate_limit_interval = 0 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.920737 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] replication_device = **** {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.920737 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] report_discard_supported = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.920737 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] report_interval = 10 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.920737 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] reservation_clean_interval = 86400 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.920737 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] reservation_expire = 86400 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.924752 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] reserved_percentage = 0 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.924752 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] resource_query_filters_file = /etc/cinder/resource_filters.json {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.924752 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] restore_discard_excess_bytes = True {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.924752 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] rootwrap_config = /etc/cinder/rootwrap.conf {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.924752 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] rpc_response_timeout = 120 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.924752 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] run_external_periodic_tasks = True {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.925175 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] scheduler_manager = cinder.scheduler.manager.SchedulerManager {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.925175 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] scst_target_driver = iscsi {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.925175 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] scst_target_iqn_name = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.925175 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] service_down_time = 60 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.925175 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] snapshot_name_template = snapshot-%s {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.925175 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] split_loggers = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.925638 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] state_path = /opt/stack/data/cinder {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.925638 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] storage_availability_zone = nova {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.925638 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] storage_protocol = iSCSI {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.925638 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] syslog_log_facility = LOG_USER {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.925638 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] target_helper = lioadm {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.925638 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] target_ip_address = 10.222.0.16 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.925638 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] target_port = 3260 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.926095 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] target_prefix = iqn.2010-10.org.openstack: {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.926095 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] target_protocol = iscsi {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.926095 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] trace_flags = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.926095 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] transfer_api_class = cinder.transfer.api.API {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.926095 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] transport_url = **** {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.926095 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] until_refresh = 0 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.926535 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] use_chap_auth = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.926535 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] use_default_quota_class = True {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.926535 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] use_eventlog = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.926535 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] use_journal = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.926535 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] use_json = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.926535 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] use_multipath_for_image_xfer = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.926535 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] use_stderr = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.926994 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] use_syslog = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.926994 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] verify_glance_signatures = enabled {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.926994 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] volume_api_class = cinder.volume.api.API {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.926994 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] volume_backend_name = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.926994 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] volume_clear = zero {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.926994 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] volume_clear_ionice = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.926994 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] volume_clear_size = 0 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.927480 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.927480 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] volume_copy_bps_limit = 0 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.927480 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] volume_dd_blocksize = 1M {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.927480 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] volume_manager = cinder.volume.manager.VolumeManager {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.927480 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] volume_name_template = volume-%s {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.927480 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] volume_usage_audit_period = month {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.927859 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] volumes_dir = /opt/stack/data/cinder/volumes {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.927859 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] watch_log_file = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 02 10:09:39.927859 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.927859 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.927859 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_policy.enforce_new_defaults = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.927859 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_policy.enforce_scope = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.928228 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_policy.policy_default_rule = default {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.928228 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_policy.policy_dirs = ['policy.d'] {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.928228 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_policy.policy_file = policy.yaml {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.928228 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.928228 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.928228 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.929413 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.929413 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.929413 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_concurrency.disable_process_locking = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.929413 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_concurrency.lock_path = /opt/stack/data/cinder {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.929413 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.admin_password = **** {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.929413 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.admin_tenant_name = admin {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.930606 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.admin_token = **** {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.930606 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.admin_user = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.930606 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.auth_admin_prefix = {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.930606 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.930606 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.auth_port = 35357 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.930606 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.auth_protocol = https {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.930606 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.auth_section = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.931607 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.auth_type = password {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.931607 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.auth_uri = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.931607 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.auth_version = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.931607 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.cache = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.931607 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.cafile = /opt/stack/data/ca-bundle.pem {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.931607 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.certfile = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.932780 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.delay_auth_decision = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.932780 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.enforce_token_bind = permissive {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.932780 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.http_connect_timeout = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.932780 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.http_request_max_retries = 3 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.932780 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.identity_uri = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.932780 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.include_service_catalog = True {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.933698 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.insecure = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.933698 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.interface = public {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.933698 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.keyfile = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.933698 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.933698 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.933698 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.934512 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.934512 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.934512 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.memcache_secret_key = **** {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.934512 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.memcache_security_strategy = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.934512 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.934512 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.memcached_servers = ['localhost:11211'] {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.935328 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.region_name = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.935328 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.service_token_roles = ['service'] {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.935328 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.service_token_roles_required = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.935328 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.service_type = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.935328 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.token_cache_time = 300 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.935328 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] keystone_authtoken.www_authenticate_uri = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.936842 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.backend = dogpile.cache.null {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.936842 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.backend_argument = **** {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.936842 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.config_prefix = cache.oslo {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.936842 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.dead_timeout = 60.0 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.936842 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.debug_cache_backend = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.936842 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.enable_retry_client = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.936842 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.enable_socket_keepalive = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.937836 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.enabled = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.937836 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.expiration_time = 600 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.937836 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.hashclient_retry_attempts = 2 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.937836 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.hashclient_retry_delay = 1.0 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.937836 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.memcache_dead_retry = 300 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.937836 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.memcache_pool_connection_get_timeout = 10 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.937836 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.memcache_pool_flush_on_reconnect = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.938780 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.memcache_pool_maxsize = 10 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.938780 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.memcache_pool_unused_timeout = 60 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.938780 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.memcache_servers = ['localhost:11211'] {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.938780 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.memcache_socket_timeout = 1.0 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.938780 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.proxies = [] {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.938780 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.retry_attempts = 2 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.938780 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.retry_delay = 0.0 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.939723 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.socket_keepalive_count = 1 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.939723 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.socket_keepalive_idle = 1 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.939723 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.socket_keepalive_interval = 1 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.939723 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.tls_allowed_ciphers = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.939723 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.tls_cafile = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.939723 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.tls_certfile = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.939723 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.tls_enabled = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.940781 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cache.tls_keyfile = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.940781 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] coordination.backend_url = etcd3+http://10.222.0.16:2379 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.940781 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] database.backend = sqlalchemy {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.940781 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] database.connection = **** {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.940781 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] database.connection_debug = 0 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.940781 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] database.connection_parameters = {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.940781 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] database.connection_recycle_time = 3600 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.941869 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] database.connection_trace = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.941869 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] database.db_inc_retry_interval = True {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.941869 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] database.db_max_retries = 20 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.941869 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] database.db_max_retry_interval = 10 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.941869 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] database.db_retry_interval = 1 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.941869 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] database.max_overflow = 50 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.941869 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] database.max_pool_size = 5 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.943095 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] database.max_retries = 10 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.943095 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] database.mysql_enable_ndb = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.943095 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] database.mysql_sql_mode = TRADITIONAL {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.943095 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] database.pool_timeout = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.943095 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] database.retry_interval = 10 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.943095 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] database.slave_connection = **** {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.943095 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] database.sqlite_synchronous = True {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.945215 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] database.use_db_reconnect = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.945215 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.945215 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.945215 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_metrics.metrics_process_name = {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.945215 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.945215 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.946708 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] profiler.connection_string = messaging:// {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.946708 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] profiler.enabled = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.946708 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] profiler.es_doc_type = notification {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.946708 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] profiler.es_scroll_size = 10000 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.946708 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] profiler.es_scroll_time = 2m {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.946708 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] profiler.filter_error_trace = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.946708 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] profiler.hmac_keys = SECRET_KEY {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.947678 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] profiler.sentinel_service_name = mymaster {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.947678 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] profiler.socket_timeout = 0.1 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.947678 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] profiler.trace_sqlalchemy = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.947678 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] privsep_osbrick.capabilities = [21] {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.947678 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] privsep_osbrick.group = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.947678 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] privsep_osbrick.helper_command = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.947678 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] privsep_osbrick.logger_name = oslo_privsep.daemon {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.948670 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] privsep_osbrick.thread_pool_size = 4 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.948670 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] privsep_osbrick.user = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.948670 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cinder_sys_admin.capabilities = [0, 1, 2, 3, 12, 21] {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.948670 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cinder_sys_admin.group = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.948670 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cinder_sys_admin.helper_command = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.948670 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cinder_sys_admin.logger_name = oslo_privsep.daemon {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.948670 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cinder_sys_admin.thread_pool_size = 4 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.949665 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] cinder_sys_admin.user = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.949665 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] service_user.auth_section = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.949665 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] service_user.auth_type = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.949665 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] service_user.cafile = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.949665 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] service_user.certfile = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.949665 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] service_user.collect_timing = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.949665 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] service_user.insecure = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.950625 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] service_user.keyfile = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.950625 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] service_user.send_service_user_token = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.950625 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] service_user.split_loggers = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.950625 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] service_user.timeout = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.950625 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.backend_availability_zone = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.950625 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.chap_password = **** {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.950625 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.chap_username = {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.951709 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.951709 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.deferred_deletion_delay = 0 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.951709 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.deferred_deletion_purge_interval = 60 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.951709 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.driver_client_cert = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.951709 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.driver_client_cert_key = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.951709 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.driver_data_namespace = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.952548 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.driver_ssl_cert_path = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.952548 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.driver_ssl_cert_verify = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.952548 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.driver_use_ssl = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.952548 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.enable_deferred_deletion = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.977616 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_db.sqlalchemy.engines [None req-8d710a85-3490-4f35-bab8-84b1c2c94066 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123698) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 02 10:09:39.977820 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.enable_unsupported_driver = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.977820 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.enforce_multipath_for_image_xfer = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.977820 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.filter_function = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.977820 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.goodness_function = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.977820 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.iet_conf = /etc/iet/ietd.conf {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.977820 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.image_upload_use_cinder_backend = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.978264 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.image_upload_use_internal_tenant = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.978264 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.image_volume_cache_enabled = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.978264 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.image_volume_cache_max_count = 0 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.978264 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.image_volume_cache_max_size_gb = 0 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.978264 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.iscsi_iotype = fileio {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.978264 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.iscsi_secondary_ip_addresses = [] {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.978639 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.iscsi_target_flags = {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.978639 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.iscsi_write_cache = on {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.978639 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.iser_helper = tgtadm {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.978639 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.iser_ip_address = 10.222.0.16 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.978639 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.iser_port = 3260 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.978639 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.979054 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.max_over_subscription_ratio = 20.0 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.979163 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.lockutils [None req-8d710a85-3490-4f35-bab8-84b1c2c94066 None None] Lock "cleanup_incomplete_backups_123252" "released" by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: held 0.134s {{(pid=123698) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:400}} May 02 10:09:39.979240 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.num_iser_scan_tries = 3 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.979240 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.num_shell_tries = 3 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.979240 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.num_volume_device_scan_tries = 3 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.979240 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.nvmet_ns_id = 10 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.979240 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.nvmet_port_id = 1 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.979240 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.rados_connect_timeout = -1 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.979240 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.rados_connection_interval = 5 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.979653 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.rados_connection_retries = 3 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.979653 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.rbd_ceph_conf = {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.979653 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.rbd_cluster_name = ceph {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.979653 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.rbd_exclusive_cinder_pool = True {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.979653 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.rbd_flatten_volume_from_snapshot = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.979653 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.rbd_max_clone_depth = 5 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.980083 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.rbd_pool = rbd {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.980083 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.rbd_secret_uuid = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.980083 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.rbd_store_chunk_size = 4 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.980083 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.rbd_user = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.980083 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.replication_connect_timeout = 5 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.980083 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.replication_device = **** {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.980461 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.report_discard_supported = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.980461 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.report_dynamic_total_capacity = True {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.980461 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.reserved_percentage = 0 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.980685 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_db.sqlalchemy.engines [None req-8d710a85-3490-4f35-bab8-84b1c2c94066 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123698) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 02 10:09:39.980758 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.scst_target_driver = iscsi {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.980758 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.scst_target_iqn_name = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.980758 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.storage_protocol = iSCSI {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.980758 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.target_helper = tgtadm {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.980758 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.target_ip_address = 10.222.0.16 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.980758 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.target_port = 3260 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.981186 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.target_prefix = iqn.2010-10.org.openstack: {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.981186 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.target_protocol = iscsi {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.981186 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.trace_flags = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.981186 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.unique_fqdn_network = True {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.981186 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.use_chap_auth = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.981186 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.use_multipath_for_image_xfer = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.981554 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.volume_backend_name = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.981554 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.volume_clear = zero {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.981554 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.volume_clear_ionice = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.981554 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.volume_clear_size = 0 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.981554 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.981554 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.volume_copy_bps_limit = 0 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.981951 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.volume_dd_blocksize = 1M {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.981951 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] backend_defaults.volumes_dir = /opt/stack/data/cinder/volumes {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.981951 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_reports.file_event_handler = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.981951 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_reports.file_event_handler_interval = 1 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.981951 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_reports.log_dir = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.981951 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.982333 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.982333 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.982333 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.982333 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.982333 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.982333 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_rabbit.heartbeat_in_pthread = True {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.982720 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-8d710a85-3490-4f35-bab8-84b1c2c94066 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:09:39.988684 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_rabbit.heartbeat_rate = 2 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.995389 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.995389 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.995389 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.995389 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.995389 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.995389 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.995922 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.997640 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:39.998417 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.002887 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.003514 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.004050 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.004659 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.005833 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.007525 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_rabbit.ssl = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.010423 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_rabbit.ssl_ca_file = {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.011438 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_rabbit.ssl_cert_file = {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.013489 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_rabbit.ssl_key_file = {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.014061 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_rabbit.ssl_version = {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.014659 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_notifications.driver = [] {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.016155 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_notifications.retry = -1 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.016757 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.019324 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] oslo_messaging_notifications.transport_url = **** {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.019324 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] key_manager.backend = cinder.keymgr.conf_key_mgr.ConfKeyManager {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.019782 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] barbican.auth_endpoint = http://localhost/identity/v3 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.021037 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] barbican.barbican_api_version = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.022660 ncdv-master-1 cinder-backup[123698]: INFO cinder.keymgr.migration [None req-f5aaa076-dab9-4c68-a853-85e1fb71cdf5 None None] Not migrating encryption keys because the ConfKeyManager is still in use. May 02 10:09:40.023235 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] barbican.barbican_endpoint = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.023806 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] barbican.barbican_endpoint_type = public {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.024338 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] barbican.barbican_region_name = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.024888 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] barbican.cafile = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.026582 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] barbican.certfile = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.027118 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] barbican.collect_timing = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.028316 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] barbican.insecure = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.033245 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] barbican.keyfile = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.033934 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] barbican.number_of_retries = 60 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.035092 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] barbican.retry_delay = 1 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.035630 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] barbican.send_service_user_token = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.036170 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] barbican.split_loggers = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.036724 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] barbican.timeout = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.037263 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] barbican.verify_ssl = True {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.037830 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] barbican.verify_ssl_path = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.038356 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] barbican_service_user.auth_section = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.038878 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] barbican_service_user.auth_type = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.039385 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] barbican_service_user.cafile = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.039900 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] barbican_service_user.certfile = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.040407 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] barbican_service_user.collect_timing = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.041335 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] barbican_service_user.insecure = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.041848 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] barbican_service_user.keyfile = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.042347 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] barbican_service_user.split_loggers = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.043206 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] barbican_service_user.timeout = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.043804 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] vault.approle_role_id = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.044305 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] vault.approle_secret_id = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.047544 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] vault.cafile = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.058349 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] vault.certfile = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.060441 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.service [None req-8d710a85-3490-4f35-bab8-84b1c2c94066 None None] Creating RPC server for service cinder-backup {{(pid=123698) start /opt/stack/cinder/cinder/service.py:231}} May 02 10:09:40.068991 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] vault.collect_timing = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.069605 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] vault.insecure = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.070140 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] vault.keyfile = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.070698 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] vault.kv_mountpoint = secret {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.071706 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] vault.kv_version = 2 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.072224 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] vault.namespace = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.075687 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] vault.root_token_id = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.076290 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] vault.split_loggers = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.076860 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] vault.ssl_ca_crt_file = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.077379 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] vault.timeout = None {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.077965 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] vault.use_ssl = False {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.080120 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.service [None req-8d710a85-3490-4f35-bab8-84b1c2c94066 None None] Pinning object versions for RPC server serializer to 1.39 {{(pid=123698) start /opt/stack/cinder/cinder/service.py:237}} May 02 10:09:40.080758 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] vault.vault_url = http://127.0.0.1:8200 {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 02 10:09:40.081232 ncdv-master-1 cinder-backup[123252]: DEBUG oslo_service.service [None req-9610c0db-2b39-4cd9-9259-7d427ba37b06 None None] ******************************************************************************** {{(pid=123252) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} May 02 10:11:32.128914 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:11:32.129401 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:12:32.133537 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:12:32.134160 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:13:32.139050 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:13:32.139666 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:14:32.144080 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:14:32.144702 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:15:32.148541 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:15:32.149183 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:16:32.153643 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:16:32.154252 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:17:32.159087 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:17:32.159711 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:18:32.163990 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:18:32.164648 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:19:32.168524 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:19:32.169170 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:20:32.173855 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:20:32.174566 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:21:32.169930 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:21:32.170576 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:22:32.176186 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:22:32.176186 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:23:32.173998 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:23:32.174761 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:24:32.173987 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:24:32.175210 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:25:32.170644 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:25:32.171304 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:26:32.170214 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:26:32.171367 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:27:32.303327 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:27:32.303327 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:28:32.165284 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:28:32.166417 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:29:32.162586 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:29:32.163599 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:30:32.162909 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:30:32.163571 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:31:32.162290 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:31:32.162828 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:32:32.167479 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:32:32.168117 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:33:32.167975 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:33:32.169423 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:34:32.173376 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:34:32.186383 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:35:09.790068 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] Create backup started, backup: d14c165d-aab9-4c37-ab08-1b28760dea9b volume: c1451bbe-299e-4929-a8c1-a7b89b9e9f18. May 02 10:35:09.903809 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-02T10:35:09Z,data_timestamp=2022-05-02T10:35:09Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsTest-Backup-558585939',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=d14c165d-aab9-4c37-ab08-1b28760dea9b,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='56a059bb653d412f81c66dfd5c50806c',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='f70906b656804455a60b4be9f825869d',volume_id=c1451bbe-299e-4929-a8c1-a7b89b9e9f18) May 02 10:35:10.042763 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.utils [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.16', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 10:35:10.044956 ncdv-master-1 cinder-backup[123698]: INFO oslo.privsep.daemon [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] Running privsep helper: ['sudo', 'cinder-rootwrap', '/etc/cinder/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/cinder/cinder.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmpn55nqvcd/privsep.sock'] May 02 10:35:10.238754 ncdv-master-1 sudo[154367]: ubuntu : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf privsep-helper --config-file /etc/cinder/cinder.conf --privsep_context os_brick.privileged.default --privsep_sock_path /tmp/tmpn55nqvcd/privsep.sock May 02 10:35:10.240094 ncdv-master-1 sudo[154367]: pam_unix(sudo:session): session opened for user root by (uid=0) May 02 10:35:10.307046 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_db.sqlalchemy.engines [None req-e2e69ca3-23ac-4614-a31d-4f46fe2b66d3 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123698) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 02 10:35:15.677120 ncdv-master-1 sudo[154367]: pam_unix(sudo:session): session closed for user root May 02 10:35:15.686337 ncdv-master-1 cinder-backup[123698]: INFO oslo.privsep.daemon [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] Spawned new privsep daemon via rootwrap May 02 10:35:15.686980 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] Accepted privsep connection to /tmp/tmpn55nqvcd/privsep.sock {{(pid=123698) __init__ /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:362}} May 02 10:35:15.688149 ncdv-master-1 cinder-backup[123698]: INFO oslo.privsep.daemon [-] privsep daemon starting May 02 10:35:15.688432 ncdv-master-1 cinder-backup[123698]: INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 May 02 10:35:15.689517 ncdv-master-1 cinder-backup[123698]: INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none May 02 10:35:15.689799 ncdv-master-1 cinder-backup[123698]: INFO oslo.privsep.daemon [-] privsep daemon running as pid 154446 May 02 10:35:15.712338 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[b001db08-3ba0-4c92-b2cd-82d7e160976c]: (2,) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:35:16.304134 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:35:16.370679 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.067s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:35:16.371008 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[9e9291b3-b0a3-4005-8808-3ca0fc218fcb]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:9d66b0da6823\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:35:16.375007 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:35:16.385775 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 02 10:35:16.414051 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[36aec7d8-4c5e-4170-8a9d-8d703f27ebec]: [Errno 2] No such file or directory: 'systool' {{(pid=154446) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 02 10:35:16.414051 ncdv-master-1 cinder-backup[123698]: Traceback (most recent call last): May 02 10:35:16.414051 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 02 10:35:16.414051 ncdv-master-1 cinder-backup[123698]: ret = func(*f_args, **f_kwargs) May 02 10:35:16.414051 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 02 10:35:16.414051 ncdv-master-1 cinder-backup[123698]: return func(*args, **kwargs) May 02 10:35:16.414051 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 02 10:35:16.414051 ncdv-master-1 cinder-backup[123698]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 02 10:35:16.414051 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 02 10:35:16.414051 ncdv-master-1 cinder-backup[123698]: return putils.execute(on_execute=on_execute, May 02 10:35:16.414051 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 02 10:35:16.414051 ncdv-master-1 cinder-backup[123698]: obj = subprocess.Popen(cmd, May 02 10:35:16.414051 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 02 10:35:16.414051 ncdv-master-1 cinder-backup[123698]: self._execute_child(args, executable, preexec_fn, close_fds, May 02 10:35:16.414051 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 02 10:35:16.414051 ncdv-master-1 cinder-backup[123698]: raise child_exception_type(errno_num, err_msg, err_filename) May 02 10:35:16.414051 ncdv-master-1 cinder-backup[123698]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 02 10:35:16.414051 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[36aec7d8-4c5e-4170-8a9d-8d703f27ebec]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:35:16.415315 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:35:16.438501 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 02 10:35:16.438501 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[f48abe77-7af1-4add-b254-87ef0fb3a076]: [Errno 2] No such file or directory: 'systool' {{(pid=154446) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 02 10:35:16.438501 ncdv-master-1 cinder-backup[123698]: Traceback (most recent call last): May 02 10:35:16.438501 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 02 10:35:16.438501 ncdv-master-1 cinder-backup[123698]: ret = func(*f_args, **f_kwargs) May 02 10:35:16.438501 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 02 10:35:16.438501 ncdv-master-1 cinder-backup[123698]: return func(*args, **kwargs) May 02 10:35:16.438501 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 02 10:35:16.438501 ncdv-master-1 cinder-backup[123698]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 02 10:35:16.438501 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 02 10:35:16.438501 ncdv-master-1 cinder-backup[123698]: return putils.execute(on_execute=on_execute, May 02 10:35:16.438501 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 02 10:35:16.438501 ncdv-master-1 cinder-backup[123698]: obj = subprocess.Popen(cmd, May 02 10:35:16.438501 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 02 10:35:16.438501 ncdv-master-1 cinder-backup[123698]: self._execute_child(args, executable, preexec_fn, close_fds, May 02 10:35:16.438501 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 02 10:35:16.438501 ncdv-master-1 cinder-backup[123698]: raise child_exception_type(errno_num, err_msg, err_filename) May 02 10:35:16.438501 ncdv-master-1 cinder-backup[123698]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 02 10:35:16.439658 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[f48abe77-7af1-4add-b254-87ef0fb3a076]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:35:16.440436 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:35:16.488024 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.047s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:35:16.489137 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[905b435d-20d5-4cf8-9652-4980d671b308]: (4, ('/dev/sda1\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:35:16.490556 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:35:16.608283 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.117s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:35:16.609255 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[ca25729e-69ae-454d-98bf-3c7291c33cba]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:35:16.612649 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:35:16.648173 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.036s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:35:16.648490 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[f573da85-2f81-439a-8828-47dc0d77cd2e]: (4, ('9ace392b-cf5e-9c45-a31c-76b2b08d55ae\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:35:16.650125 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] Running cmd (subprocess): nvme version {{(pid=123698) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:35:16.709882 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] CMD "nvme version" returned: 0 in 0.060s {{(pid=123698) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:35:16.716792 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=123698) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 02 10:35:16.717386 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=123698) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 02 10:35:16.717826 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:7c8447db-24f2-4326-a844-150f6bd61ca4 dsc: {{(pid=123698) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 02 10:35:16.719171 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.utils [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] <== get_connector_properties: return (6675ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.16', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:9d66b0da6823', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '9ace392b-cf5e-9c45-a31c-76b2b08d55ae', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:7c8447db-24f2-4326-a844-150f6bd61ca4', 'found_dsc': ''} {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 10:35:18.612815 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connector [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] Factory for rbd on None {{(pid=123698) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 02 10:35:18.616692 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-c1451bbe-299e-4929-a8c1-a7b89b9e9f18', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'c1451bbe-299e-4929-a8c1-a7b89b9e9f18', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 10:35:18.617914 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.linuxrbd [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] opening connection to ceph cluster (timeout=-1). {{(pid=123698) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 02 10:35:18.826577 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] <== connect_volume: return (208ms) {'path': } {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 10:35:19.005169 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] Starting backup of volume='c1451bbe-299e-4929-a8c1-a7b89b9e9f18'. {{(pid=123698) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} May 02 10:35:19.006485 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] Volume file is RBD: attempting optimized backup {{(pid=123698) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}} May 02 10:35:19.119724 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] Creating base image 'volume-c1451bbe-299e-4929-a8c1-a7b89b9e9f18.backup.d14c165d-aab9-4c37-ab08-1b28760dea9b' {{(pid=123698) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}} May 02 10:35:19.123169 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=123698) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} May 02 10:35:22.481755 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] Using --from-snap 'None' for incremental backup of volume c1451bbe-299e-4929-a8c1-a7b89b9e9f18. {{(pid=123698) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}} May 02 10:35:22.482449 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] Creating backup snapshot='backup.d14c165d-aab9-4c37-ab08-1b28760dea9b.snap.1651487722.4817803' {{(pid=123698) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}} May 02 10:35:24.313717 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] Performing differential transfer from 'volume-c1451bbe-299e-4929-a8c1-a7b89b9e9f18' to 'volume-c1451bbe-299e-4929-a8c1-a7b89b9e9f18.backup.d14c165d-aab9-4c37-ab08-1b28760dea9b' {{(pid=123698) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} May 02 10:35:24.314312 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_g83ijlhs --pool volumes volumes/volume-c1451bbe-299e-4929-a8c1-a7b89b9e9f18@backup.d14c165d-aab9-4c37-ab08-1b28760dea9b.snap.1651487722.4817803 -' into... {{(pid=123698) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} May 02 10:35:24.315239 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-c1451bbe-299e-4929-a8c1-a7b89b9e9f18.backup.d14c165d-aab9-4c37-ab08-1b28760dea9b' {{(pid=123698) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} May 02 10:35:27.804907 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] Differential backup transfer completed in 3.4904s {{(pid=123698) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}} May 02 10:35:27.805622 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] Getting metadata type 'volume-base-metadata' {{(pid=123698) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 02 10:35:27.819549 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=123698) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 02 10:35:27.819953 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] Getting metadata type 'volume-metadata' {{(pid=123698) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 02 10:35:27.832132 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] No metadata type 'volume-metadata' available {{(pid=123698) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 02 10:35:27.832514 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] Getting metadata type 'volume-glance-metadata' {{(pid=123698) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 02 10:35:27.847692 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] No metadata type 'volume-glance-metadata' available {{(pid=123698) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} May 02 10:35:27.848426 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] Backing up metadata for volume c1451bbe-299e-4929-a8c1-a7b89b9e9f18. {{(pid=123698) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} May 02 10:35:28.019628 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] Backup 'd14c165d-aab9-4c37-ab08-1b28760dea9b' of volume c1451bbe-299e-4929-a8c1-a7b89b9e9f18 finished. {{(pid=123698) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} May 02 10:35:28.021422 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-c1451bbe-299e-4929-a8c1-a7b89b9e9f18', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'c1451bbe-299e-4929-a8c1-a7b89b9e9f18', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 10:35:28.258973 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] <== disconnect_volume: return (236ms) None {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 10:35:28.602104 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-49504678-dedd-4f20-9b98-0e90813a8232 tempest-VolumesBackupsTest-1849290619 None] Create backup finished. backup: d14c165d-aab9-4c37-ab08-1b28760dea9b. May 02 10:35:29.825424 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-8ad2f1bd-2f74-4469-9ce0-b54d0463a282 tempest-VolumesBackupsTest-1849290619 None] Delete backup started, backup: d14c165d-aab9-4c37-ab08-1b28760dea9b. May 02 10:35:29.827430 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-8ad2f1bd-2f74-4469-9ce0-b54d0463a282 tempest-VolumesBackupsTest-1849290619 None] Delete started for backup=d14c165d-aab9-4c37-ab08-1b28760dea9b {{(pid=123698) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} May 02 10:35:29.827883 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-8ad2f1bd-2f74-4469-9ce0-b54d0463a282 tempest-VolumesBackupsTest-1849290619 None] Trying diff format basename='volume-c1451bbe-299e-4929-a8c1-a7b89b9e9f18.backup.d14c165d-aab9-4c37-ab08-1b28760dea9b' for backup base image of volume c1451bbe-299e-4929-a8c1-a7b89b9e9f18. {{(pid=123698) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} May 02 10:35:30.182401 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-8ad2f1bd-2f74-4469-9ce0-b54d0463a282 tempest-VolumesBackupsTest-1849290619 None] Looking for snapshot of backup base 'volume-c1451bbe-299e-4929-a8c1-a7b89b9e9f18.backup.d14c165d-aab9-4c37-ab08-1b28760dea9b' {{(pid=123698) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 02 10:35:30.187089 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-8ad2f1bd-2f74-4469-9ce0-b54d0463a282 tempest-VolumesBackupsTest-1849290619 None] Found snapshot 'backup.d14c165d-aab9-4c37-ab08-1b28760dea9b.snap.1651487722.4817803' {{(pid=123698) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} May 02 10:35:30.187955 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-8ad2f1bd-2f74-4469-9ce0-b54d0463a282 tempest-VolumesBackupsTest-1849290619 None] Deleting backup snapshot='backup.d14c165d-aab9-4c37-ab08-1b28760dea9b.snap.1651487722.4817803' {{(pid=123698) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}} May 02 10:35:30.301621 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_db.sqlalchemy.engines [None req-e2e69ca3-23ac-4614-a31d-4f46fe2b66d3 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123698) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 02 10:35:32.179464 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:35:32.180050 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:35:33.450282 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.drivers.ceph [None req-8ad2f1bd-2f74-4469-9ce0-b54d0463a282 tempest-VolumesBackupsTest-1849290619 None] Deleting backup base image='volume-c1451bbe-299e-4929-a8c1-a7b89b9e9f18.backup.d14c165d-aab9-4c37-ab08-1b28760dea9b' of volume c1451bbe-299e-4929-a8c1-a7b89b9e9f18. May 02 10:35:40.403507 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-8ad2f1bd-2f74-4469-9ce0-b54d0463a282 tempest-VolumesBackupsTest-1849290619 None] Base backup image='volume-c1451bbe-299e-4929-a8c1-a7b89b9e9f18.backup.d14c165d-aab9-4c37-ab08-1b28760dea9b' of volume c1451bbe-299e-4929-a8c1-a7b89b9e9f18 deleted. {{(pid=123698) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} May 02 10:35:40.677516 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-8ad2f1bd-2f74-4469-9ce0-b54d0463a282 tempest-VolumesBackupsTest-1849290619 None] Delete of backup 'd14c165d-aab9-4c37-ab08-1b28760dea9b' for volume 'c1451bbe-299e-4929-a8c1-a7b89b9e9f18' finished. {{(pid=123698) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} May 02 10:35:40.706886 ncdv-master-1 cinder-backup[123698]: WARNING cinder.quota [None req-8ad2f1bd-2f74-4469-9ce0-b54d0463a282 tempest-VolumesBackupsTest-1849290619 None] Deprecated: Default quota for resource: backups is set by the default quota flag: quota_backups, it is now deprecated. Please use the default quota class for default quota. May 02 10:35:40.707776 ncdv-master-1 cinder-backup[123698]: WARNING cinder.quota [None req-8ad2f1bd-2f74-4469-9ce0-b54d0463a282 tempest-VolumesBackupsTest-1849290619 None] Deprecated: Default quota for resource: backup_gigabytes is set by the default quota flag: quota_backup_gigabytes, it is now deprecated. Please use the default quota class for default quota. May 02 10:35:40.843202 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.quota [None req-8ad2f1bd-2f74-4469-9ce0-b54d0463a282 tempest-VolumesBackupsTest-1849290619 None] Created reservations ['eed21e39-e838-49a7-bbbe-c516d97a5259', 'fc559b16-5c1c-4b0e-8635-64018c3c5f98'] {{(pid=123698) reserve /opt/stack/cinder/cinder/quota.py:755}} May 02 10:35:41.040327 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-8ad2f1bd-2f74-4469-9ce0-b54d0463a282 tempest-VolumesBackupsTest-1849290619 None] Delete backup finished, backup d14c165d-aab9-4c37-ab08-1b28760dea9b deleted. May 02 10:36:09.916327 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] Create backup started, backup: 85845315-21b0-4d0b-a72d-d7aad1728a0a volume: 6a607ad4-7820-465b-a861-f796c57fde8c. May 02 10:36:10.014736 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-02T10:36:09Z,data_timestamp=2022-05-02T10:36:09Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsTest-Backup-1368441604',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=85845315-21b0-4d0b-a72d-d7aad1728a0a,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='56a059bb653d412f81c66dfd5c50806c',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='f70906b656804455a60b4be9f825869d',volume_id=6a607ad4-7820-465b-a861-f796c57fde8c) May 02 10:36:10.518265 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.utils [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.16', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 10:36:10.520523 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:36:10.554506 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.034s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:36:10.555997 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[f5818170-1228-4d11-acc8-f2cdc1ea01d7]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:9d66b0da6823\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:36:10.562022 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:36:10.578066 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 02 10:36:10.592320 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[e313ea6b-e90d-4d10-affc-1f3b4bf66daf]: [Errno 2] No such file or directory: 'systool' {{(pid=154446) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 02 10:36:10.592320 ncdv-master-1 cinder-backup[123698]: Traceback (most recent call last): May 02 10:36:10.592320 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 02 10:36:10.592320 ncdv-master-1 cinder-backup[123698]: ret = func(*f_args, **f_kwargs) May 02 10:36:10.592320 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 02 10:36:10.592320 ncdv-master-1 cinder-backup[123698]: return func(*args, **kwargs) May 02 10:36:10.592320 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 02 10:36:10.592320 ncdv-master-1 cinder-backup[123698]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 02 10:36:10.592320 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 02 10:36:10.592320 ncdv-master-1 cinder-backup[123698]: return putils.execute(on_execute=on_execute, May 02 10:36:10.592320 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 02 10:36:10.592320 ncdv-master-1 cinder-backup[123698]: obj = subprocess.Popen(cmd, May 02 10:36:10.592320 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 02 10:36:10.592320 ncdv-master-1 cinder-backup[123698]: self._execute_child(args, executable, preexec_fn, close_fds, May 02 10:36:10.592320 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 02 10:36:10.592320 ncdv-master-1 cinder-backup[123698]: raise child_exception_type(errno_num, err_msg, err_filename) May 02 10:36:10.592320 ncdv-master-1 cinder-backup[123698]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 02 10:36:10.597301 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[e313ea6b-e90d-4d10-affc-1f3b4bf66daf]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:36:10.598425 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:36:10.625046 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 02 10:36:10.625046 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[ab016868-9602-4a06-9cb3-f43ac2547bd2]: [Errno 2] No such file or directory: 'systool' {{(pid=154446) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 02 10:36:10.625046 ncdv-master-1 cinder-backup[123698]: Traceback (most recent call last): May 02 10:36:10.625046 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 02 10:36:10.625046 ncdv-master-1 cinder-backup[123698]: ret = func(*f_args, **f_kwargs) May 02 10:36:10.625046 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 02 10:36:10.625046 ncdv-master-1 cinder-backup[123698]: return func(*args, **kwargs) May 02 10:36:10.625046 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 02 10:36:10.625046 ncdv-master-1 cinder-backup[123698]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 02 10:36:10.625046 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 02 10:36:10.625046 ncdv-master-1 cinder-backup[123698]: return putils.execute(on_execute=on_execute, May 02 10:36:10.625046 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 02 10:36:10.625046 ncdv-master-1 cinder-backup[123698]: obj = subprocess.Popen(cmd, May 02 10:36:10.625046 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 02 10:36:10.625046 ncdv-master-1 cinder-backup[123698]: self._execute_child(args, executable, preexec_fn, close_fds, May 02 10:36:10.625046 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 02 10:36:10.625046 ncdv-master-1 cinder-backup[123698]: raise child_exception_type(errno_num, err_msg, err_filename) May 02 10:36:10.625046 ncdv-master-1 cinder-backup[123698]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 02 10:36:10.625046 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[ab016868-9602-4a06-9cb3-f43ac2547bd2]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:36:10.627677 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:36:10.636759 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.012s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:36:10.637099 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[07d07c7c-ec95-4475-a022-aeb2867fe53e]: (4, ('/dev/sda1\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:36:10.638795 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:36:10.652990 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.013s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:36:10.653794 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[ac346000-d998-430e-93fa-8dd923826d53]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:36:10.655956 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:36:10.682651 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.025s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:36:10.683160 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[d02cdda1-b179-4a38-95a7-576e480f3206]: (4, ('9ace392b-cf5e-9c45-a31c-76b2b08d55ae\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:36:10.684062 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] Running cmd (subprocess): nvme version {{(pid=123698) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:36:10.724412 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] CMD "nvme version" returned: 0 in 0.040s {{(pid=123698) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:36:10.726615 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=123698) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 02 10:36:10.727094 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=123698) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 02 10:36:10.727697 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:7c8447db-24f2-4326-a844-150f6bd61ca4 dsc: {{(pid=123698) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 02 10:36:10.728450 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.utils [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] <== get_connector_properties: return (209ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.16', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:9d66b0da6823', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '9ace392b-cf5e-9c45-a31c-76b2b08d55ae', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:7c8447db-24f2-4326-a844-150f6bd61ca4', 'found_dsc': ''} {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 10:36:11.736505 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connector [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] Factory for rbd on None {{(pid=123698) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 02 10:36:11.738053 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-6a607ad4-7820-465b-a861-f796c57fde8c', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '6a607ad4-7820-465b-a861-f796c57fde8c', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 10:36:11.739138 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.linuxrbd [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] opening connection to ceph cluster (timeout=-1). {{(pid=123698) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 02 10:36:12.179352 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] <== connect_volume: return (440ms) {'path': } {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 10:36:12.187701 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_db.sqlalchemy.engines [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123698) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 02 10:36:12.280989 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] Starting backup of volume='6a607ad4-7820-465b-a861-f796c57fde8c'. {{(pid=123698) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} May 02 10:36:12.280989 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] Volume file is RBD: attempting optimized backup {{(pid=123698) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}} May 02 10:36:12.306930 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] Creating base image 'volume-6a607ad4-7820-465b-a861-f796c57fde8c.backup.85845315-21b0-4d0b-a72d-d7aad1728a0a' {{(pid=123698) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}} May 02 10:36:12.307444 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=123698) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} May 02 10:36:13.358131 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] Using --from-snap 'None' for incremental backup of volume 6a607ad4-7820-465b-a861-f796c57fde8c. {{(pid=123698) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}} May 02 10:36:13.359426 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] Creating backup snapshot='backup.85845315-21b0-4d0b-a72d-d7aad1728a0a.snap.1651487773.3589294' {{(pid=123698) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}} May 02 10:36:15.464923 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] Performing differential transfer from 'volume-6a607ad4-7820-465b-a861-f796c57fde8c' to 'volume-6a607ad4-7820-465b-a861-f796c57fde8c.backup.85845315-21b0-4d0b-a72d-d7aad1728a0a' {{(pid=123698) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} May 02 10:36:15.464923 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_oyeetvx9 --pool volumes volumes/volume-6a607ad4-7820-465b-a861-f796c57fde8c@backup.85845315-21b0-4d0b-a72d-d7aad1728a0a.snap.1651487773.3589294 -' into... {{(pid=123698) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} May 02 10:36:15.464923 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-6a607ad4-7820-465b-a861-f796c57fde8c.backup.85845315-21b0-4d0b-a72d-d7aad1728a0a' {{(pid=123698) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} May 02 10:36:32.185004 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:36:32.185644 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:37:12.694020 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] Differential backup transfer completed in 57.2360s {{(pid=123698) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}} May 02 10:37:12.694879 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] Getting metadata type 'volume-base-metadata' {{(pid=123698) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 02 10:37:12.707009 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=123698) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 02 10:37:12.707411 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] Getting metadata type 'volume-metadata' {{(pid=123698) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 02 10:37:12.715920 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] No metadata type 'volume-metadata' available {{(pid=123698) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 02 10:37:12.716308 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] Getting metadata type 'volume-glance-metadata' {{(pid=123698) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 02 10:37:12.724829 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=123698) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}} May 02 10:37:12.725649 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] Backing up metadata for volume 6a607ad4-7820-465b-a861-f796c57fde8c. {{(pid=123698) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} May 02 10:37:13.147262 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] Backup '85845315-21b0-4d0b-a72d-d7aad1728a0a' of volume 6a607ad4-7820-465b-a861-f796c57fde8c finished. {{(pid=123698) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} May 02 10:37:13.149087 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-6a607ad4-7820-465b-a861-f796c57fde8c', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '6a607ad4-7820-465b-a861-f796c57fde8c', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 10:37:14.155129 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] <== disconnect_volume: return (1004ms) None {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 10:37:14.371641 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-afb4635a-1eaf-4b63-88bf-e4e80a087043 tempest-VolumesBackupsTest-1849290619 None] Create backup finished. backup: 85845315-21b0-4d0b-a72d-d7aad1728a0a. May 02 10:37:17.778202 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] Restore backup started, backup: 85845315-21b0-4d0b-a72d-d7aad1728a0a volume: ed1e3fc5-f3c6-40bd-8c28-d19c62ac883a. May 02 10:37:17.793540 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.utils [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.16', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 10:37:17.795400 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:37:17.804362 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.009s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:37:17.804845 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[9fdc6960-79a9-4e36-a1de-103b3db7f1ca]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:9d66b0da6823\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:37:17.806201 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:37:17.813613 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 02 10:37:17.814420 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[00b30e33-402c-4966-a4f5-e6efaef2e9b7]: [Errno 2] No such file or directory: 'systool' {{(pid=154446) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 02 10:37:17.814420 ncdv-master-1 cinder-backup[123698]: Traceback (most recent call last): May 02 10:37:17.814420 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 02 10:37:17.814420 ncdv-master-1 cinder-backup[123698]: ret = func(*f_args, **f_kwargs) May 02 10:37:17.814420 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 02 10:37:17.814420 ncdv-master-1 cinder-backup[123698]: return func(*args, **kwargs) May 02 10:37:17.814420 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 02 10:37:17.814420 ncdv-master-1 cinder-backup[123698]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 02 10:37:17.814420 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 02 10:37:17.814420 ncdv-master-1 cinder-backup[123698]: return putils.execute(on_execute=on_execute, May 02 10:37:17.814420 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 02 10:37:17.814420 ncdv-master-1 cinder-backup[123698]: obj = subprocess.Popen(cmd, May 02 10:37:17.814420 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 02 10:37:17.814420 ncdv-master-1 cinder-backup[123698]: self._execute_child(args, executable, preexec_fn, close_fds, May 02 10:37:17.814420 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 02 10:37:17.814420 ncdv-master-1 cinder-backup[123698]: raise child_exception_type(errno_num, err_msg, err_filename) May 02 10:37:17.814420 ncdv-master-1 cinder-backup[123698]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 02 10:37:17.815829 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[00b30e33-402c-4966-a4f5-e6efaef2e9b7]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:37:17.816085 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:37:17.823072 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 02 10:37:17.823670 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[77171fcd-0c70-4c11-b7fc-1c3827a47372]: [Errno 2] No such file or directory: 'systool' {{(pid=154446) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 02 10:37:17.823670 ncdv-master-1 cinder-backup[123698]: Traceback (most recent call last): May 02 10:37:17.823670 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 02 10:37:17.823670 ncdv-master-1 cinder-backup[123698]: ret = func(*f_args, **f_kwargs) May 02 10:37:17.823670 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 02 10:37:17.823670 ncdv-master-1 cinder-backup[123698]: return func(*args, **kwargs) May 02 10:37:17.823670 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 02 10:37:17.823670 ncdv-master-1 cinder-backup[123698]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 02 10:37:17.823670 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 02 10:37:17.823670 ncdv-master-1 cinder-backup[123698]: return putils.execute(on_execute=on_execute, May 02 10:37:17.823670 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 02 10:37:17.823670 ncdv-master-1 cinder-backup[123698]: obj = subprocess.Popen(cmd, May 02 10:37:17.823670 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 02 10:37:17.823670 ncdv-master-1 cinder-backup[123698]: self._execute_child(args, executable, preexec_fn, close_fds, May 02 10:37:17.823670 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 02 10:37:17.823670 ncdv-master-1 cinder-backup[123698]: raise child_exception_type(errno_num, err_msg, err_filename) May 02 10:37:17.823670 ncdv-master-1 cinder-backup[123698]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 02 10:37:17.825077 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[77171fcd-0c70-4c11-b7fc-1c3827a47372]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:37:17.825800 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:37:17.834750 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.009s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:37:17.835353 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[4d9e9dc2-399c-4f08-bef1-3c56fca1d300]: (4, ('/dev/sda1\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:37:17.836768 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:37:17.847594 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.011s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:37:17.847918 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[7cdece96-895f-44ce-862b-656dafdadca2]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:37:17.854403 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:37:17.864356 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.015s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:37:17.864736 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[d18f338d-a708-451e-bf04-6b1271b53569]: (4, ('9ace392b-cf5e-9c45-a31c-76b2b08d55ae\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:37:17.865605 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] Running cmd (subprocess): nvme version {{(pid=123698) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:37:17.886057 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] CMD "nvme version" returned: 0 in 0.020s {{(pid=123698) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:37:17.888581 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=123698) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 02 10:37:17.889068 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=123698) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 02 10:37:17.889571 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:7c8447db-24f2-4326-a844-150f6bd61ca4 dsc: {{(pid=123698) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 02 10:37:17.891068 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.utils [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] <== get_connector_properties: return (96ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.16', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:9d66b0da6823', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '9ace392b-cf5e-9c45-a31c-76b2b08d55ae', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:7c8447db-24f2-4326-a844-150f6bd61ca4', 'found_dsc': ''} {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 10:37:18.766281 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connector [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] Factory for rbd on None {{(pid=123698) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 02 10:37:18.767999 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-ed1e3fc5-f3c6-40bd-8c28-d19c62ac883a', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'ed1e3fc5-f3c6-40bd-8c28-d19c62ac883a', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 10:37:18.769197 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.linuxrbd [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] opening connection to ceph cluster (timeout=-1). {{(pid=123698) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 02 10:37:19.274436 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] <== connect_volume: return (505ms) {'path': } {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 10:37:19.285563 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_db.sqlalchemy.engines [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123698) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 02 10:37:19.297814 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] Starting restore from Ceph backup=85845315-21b0-4d0b-a72d-d7aad1728a0a to volume=volume-ed1e3fc5-f3c6-40bd-8c28-d19c62ac883a {{(pid=123698) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1343}} May 02 10:37:19.425892 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] Looking for snapshot of backup base 'volume-6a607ad4-7820-465b-a861-f796c57fde8c.backup.85845315-21b0-4d0b-a72d-d7aad1728a0a' {{(pid=123698) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 02 10:37:19.426355 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] Found snapshot 'backup.85845315-21b0-4d0b-a72d-d7aad1728a0a.snap.1651487773.3589294' {{(pid=123698) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} May 02 10:37:19.443871 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] Volume file is RBD. {{(pid=123698) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1245}} May 02 10:37:19.513377 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] Attempting differential restore. {{(pid=123698) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1298}} May 02 10:37:19.513905 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] Attempting incremental restore from base='volume-6a607ad4-7820-465b-a861-f796c57fde8c.backup.85845315-21b0-4d0b-a72d-d7aad1728a0a' snap='backup.85845315-21b0-4d0b-a72d-d7aad1728a0a.snap.1651487773.3589294' {{(pid=123698) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1145}} May 02 10:37:19.514291 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] Performing differential transfer from 'volume-6a607ad4-7820-465b-a861-f796c57fde8c.backup.85845315-21b0-4d0b-a72d-d7aad1728a0a' to 'volume-ed1e3fc5-f3c6-40bd-8c28-d19c62ac883a' {{(pid=123698) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} May 02 10:37:19.515209 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-6a607ad4-7820-465b-a861-f796c57fde8c.backup.85845315-21b0-4d0b-a72d-d7aad1728a0a@backup.85845315-21b0-4d0b-a72d-d7aad1728a0a.snap.1651487773.3589294 -' into... {{(pid=123698) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} May 02 10:37:19.515568 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_e5a5f1kf --pool volumes - volumes/volume-ed1e3fc5-f3c6-40bd-8c28-d19c62ac883a' {{(pid=123698) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} May 02 10:37:32.182143 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:37:32.182891 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:38:12.474443 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] Restore transfer completed in 52.9599s {{(pid=123698) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1167}} May 02 10:38:12.480034 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=123698) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1354}} May 02 10:38:12.530079 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] Restoring volume base metadata {{(pid=123698) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} May 02 10:38:12.663728 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] No metadata of type 'volume-metadata' to restore {{(pid=123698) put /opt/stack/cinder/cinder/backup/driver.py:345}} May 02 10:38:12.666020 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] Restoring volume glance metadata {{(pid=123698) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}} May 02 10:38:13.962761 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] Restore to volume ed1e3fc5-f3c6-40bd-8c28-d19c62ac883a finished successfully. {{(pid=123698) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1361}} May 02 10:38:13.966903 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-ed1e3fc5-f3c6-40bd-8c28-d19c62ac883a', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'ed1e3fc5-f3c6-40bd-8c28-d19c62ac883a', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 10:38:14.188960 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] <== disconnect_volume: return (215ms) None {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 10:38:14.266761 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.manager [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] Encryption key ID for volume ed1e3fc5-f3c6-40bd-8c28-d19c62ac883a already matches encryption key ID in backup 85845315-21b0-4d0b-a72d-d7aad1728a0a. {{(pid=123698) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} May 02 10:38:14.492572 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-fb820719-832c-458c-bea5-bae1e9d92fed tempest-VolumesBackupsTest-1849290619 None] Finished restoring backup 85845315-21b0-4d0b-a72d-d7aad1728a0a to volume ed1e3fc5-f3c6-40bd-8c28-d19c62ac883a. May 02 10:38:32.178674 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:38:32.179519 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:38:33.181744 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-9d87bd7c-207e-465f-a19b-e4503ec14323 tempest-VolumesBackupsTest-1849290619 None] Delete backup started, backup: 85845315-21b0-4d0b-a72d-d7aad1728a0a. May 02 10:38:33.184037 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-9d87bd7c-207e-465f-a19b-e4503ec14323 tempest-VolumesBackupsTest-1849290619 None] Delete started for backup=85845315-21b0-4d0b-a72d-d7aad1728a0a {{(pid=123698) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} May 02 10:38:33.184462 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-9d87bd7c-207e-465f-a19b-e4503ec14323 tempest-VolumesBackupsTest-1849290619 None] Trying diff format basename='volume-6a607ad4-7820-465b-a861-f796c57fde8c.backup.85845315-21b0-4d0b-a72d-d7aad1728a0a' for backup base image of volume 6a607ad4-7820-465b-a861-f796c57fde8c. {{(pid=123698) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} May 02 10:38:33.612963 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-9d87bd7c-207e-465f-a19b-e4503ec14323 tempest-VolumesBackupsTest-1849290619 None] Looking for snapshot of backup base 'volume-6a607ad4-7820-465b-a861-f796c57fde8c.backup.85845315-21b0-4d0b-a72d-d7aad1728a0a' {{(pid=123698) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 02 10:38:33.613454 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-9d87bd7c-207e-465f-a19b-e4503ec14323 tempest-VolumesBackupsTest-1849290619 None] Found snapshot 'backup.85845315-21b0-4d0b-a72d-d7aad1728a0a.snap.1651487773.3589294' {{(pid=123698) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} May 02 10:38:33.613973 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-9d87bd7c-207e-465f-a19b-e4503ec14323 tempest-VolumesBackupsTest-1849290619 None] Deleting backup snapshot='backup.85845315-21b0-4d0b-a72d-d7aad1728a0a.snap.1651487773.3589294' {{(pid=123698) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}} May 02 10:38:37.673893 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.drivers.ceph [None req-9d87bd7c-207e-465f-a19b-e4503ec14323 tempest-VolumesBackupsTest-1849290619 None] Deleting backup base image='volume-6a607ad4-7820-465b-a861-f796c57fde8c.backup.85845315-21b0-4d0b-a72d-d7aad1728a0a' of volume 6a607ad4-7820-465b-a861-f796c57fde8c. May 02 10:38:40.319433 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_db.sqlalchemy.engines [None req-e2e69ca3-23ac-4614-a31d-4f46fe2b66d3 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123698) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 02 10:38:50.302704 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-9d87bd7c-207e-465f-a19b-e4503ec14323 tempest-VolumesBackupsTest-1849290619 None] Base backup image='volume-6a607ad4-7820-465b-a861-f796c57fde8c.backup.85845315-21b0-4d0b-a72d-d7aad1728a0a' of volume 6a607ad4-7820-465b-a861-f796c57fde8c deleted. {{(pid=123698) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} May 02 10:38:50.547051 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-9d87bd7c-207e-465f-a19b-e4503ec14323 tempest-VolumesBackupsTest-1849290619 None] Delete of backup '85845315-21b0-4d0b-a72d-d7aad1728a0a' for volume '6a607ad4-7820-465b-a861-f796c57fde8c' finished. {{(pid=123698) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} May 02 10:38:50.662169 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.quota [None req-9d87bd7c-207e-465f-a19b-e4503ec14323 tempest-VolumesBackupsTest-1849290619 None] Created reservations ['f66aef33-926a-4d95-82d0-1ed7d955d2f9', 'be286e3e-188a-4693-b71f-1a6b048420e4'] {{(pid=123698) reserve /opt/stack/cinder/cinder/quota.py:755}} May 02 10:38:50.913601 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-9d87bd7c-207e-465f-a19b-e4503ec14323 tempest-VolumesBackupsTest-1849290619 None] Delete backup finished, backup 85845315-21b0-4d0b-a72d-d7aad1728a0a deleted. May 02 10:39:22.055262 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] Create backup started, backup: 0c8a6cc7-d457-4a3d-8d37-477336ffc040 volume: f8a426a0-2921-4da7-a459-d3ceeaacd638. May 02 10:39:22.154853 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-02T10:39:21Z,data_timestamp=2022-05-02T10:39:21Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsV39Test-Backup-1114473205',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=0c8a6cc7-d457-4a3d-8d37-477336ffc040,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='6fb3f9dce951489aafe56b0d30841164',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='a6cd9c820ef74b2f9ef7db7846159495',volume_id=f8a426a0-2921-4da7-a459-d3ceeaacd638) May 02 10:39:22.228804 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.utils [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.16', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 10:39:22.231132 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:39:22.239910 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.009s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:39:22.246873 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[21c30d64-d0a3-44ed-a79d-b18eb5ea3ff1]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:9d66b0da6823\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:39:22.246873 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:39:22.249843 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 02 10:39:22.250611 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[eb9722a1-1a6b-4ae0-a891-ad39d4e38688]: [Errno 2] No such file or directory: 'systool' {{(pid=154446) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 02 10:39:22.250611 ncdv-master-1 cinder-backup[123698]: Traceback (most recent call last): May 02 10:39:22.250611 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 02 10:39:22.250611 ncdv-master-1 cinder-backup[123698]: ret = func(*f_args, **f_kwargs) May 02 10:39:22.250611 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 02 10:39:22.250611 ncdv-master-1 cinder-backup[123698]: return func(*args, **kwargs) May 02 10:39:22.250611 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 02 10:39:22.250611 ncdv-master-1 cinder-backup[123698]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 02 10:39:22.250611 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 02 10:39:22.250611 ncdv-master-1 cinder-backup[123698]: return putils.execute(on_execute=on_execute, May 02 10:39:22.250611 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 02 10:39:22.250611 ncdv-master-1 cinder-backup[123698]: obj = subprocess.Popen(cmd, May 02 10:39:22.250611 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 02 10:39:22.250611 ncdv-master-1 cinder-backup[123698]: self._execute_child(args, executable, preexec_fn, close_fds, May 02 10:39:22.250611 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 02 10:39:22.250611 ncdv-master-1 cinder-backup[123698]: raise child_exception_type(errno_num, err_msg, err_filename) May 02 10:39:22.250611 ncdv-master-1 cinder-backup[123698]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 02 10:39:22.251871 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[eb9722a1-1a6b-4ae0-a891-ad39d4e38688]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:39:22.254178 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:39:22.258800 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 02 10:39:22.260534 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[3ed7dea9-65b9-4d87-b2e3-a6853622ae4d]: [Errno 2] No such file or directory: 'systool' {{(pid=154446) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 02 10:39:22.260534 ncdv-master-1 cinder-backup[123698]: Traceback (most recent call last): May 02 10:39:22.260534 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 02 10:39:22.260534 ncdv-master-1 cinder-backup[123698]: ret = func(*f_args, **f_kwargs) May 02 10:39:22.260534 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 02 10:39:22.260534 ncdv-master-1 cinder-backup[123698]: return func(*args, **kwargs) May 02 10:39:22.260534 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 02 10:39:22.260534 ncdv-master-1 cinder-backup[123698]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 02 10:39:22.260534 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 02 10:39:22.260534 ncdv-master-1 cinder-backup[123698]: return putils.execute(on_execute=on_execute, May 02 10:39:22.260534 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 02 10:39:22.260534 ncdv-master-1 cinder-backup[123698]: obj = subprocess.Popen(cmd, May 02 10:39:22.260534 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 02 10:39:22.260534 ncdv-master-1 cinder-backup[123698]: self._execute_child(args, executable, preexec_fn, close_fds, May 02 10:39:22.260534 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 02 10:39:22.260534 ncdv-master-1 cinder-backup[123698]: raise child_exception_type(errno_num, err_msg, err_filename) May 02 10:39:22.260534 ncdv-master-1 cinder-backup[123698]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 02 10:39:22.261714 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[3ed7dea9-65b9-4d87-b2e3-a6853622ae4d]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:39:22.264188 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:39:22.275668 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.013s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:39:22.276018 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[87ed3801-372c-484b-9c10-22d0de3da38c]: (4, ('/dev/sda1\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:39:22.277340 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:39:22.287093 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.009s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:39:22.287421 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[e576ee1d-c67b-41f1-ae63-8b49296762f7]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:39:22.288726 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:39:22.317450 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.028s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:39:22.319251 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[3831b2ef-a2c6-4837-8906-572d6fc57c13]: (4, ('9ace392b-cf5e-9c45-a31c-76b2b08d55ae\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:39:22.319997 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] Running cmd (subprocess): nvme version {{(pid=123698) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:39:22.351992 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] CMD "nvme version" returned: 0 in 0.032s {{(pid=123698) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:39:22.354135 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=123698) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 02 10:39:22.354507 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=123698) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 02 10:39:22.354896 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:7c8447db-24f2-4326-a844-150f6bd61ca4 dsc: {{(pid=123698) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 02 10:39:22.355544 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.utils [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] <== get_connector_properties: return (126ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.16', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:9d66b0da6823', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '9ace392b-cf5e-9c45-a31c-76b2b08d55ae', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:7c8447db-24f2-4326-a844-150f6bd61ca4', 'found_dsc': ''} {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 10:39:23.277991 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connector [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] Factory for rbd on None {{(pid=123698) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 02 10:39:23.281094 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-f8a426a0-2921-4da7-a459-d3ceeaacd638', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'f8a426a0-2921-4da7-a459-d3ceeaacd638', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 10:39:23.282383 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.linuxrbd [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] opening connection to ceph cluster (timeout=-1). {{(pid=123698) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 02 10:39:23.615638 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] <== connect_volume: return (333ms) {'path': } {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 10:39:23.623828 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_db.sqlalchemy.engines [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123698) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 02 10:39:23.710834 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] Starting backup of volume='f8a426a0-2921-4da7-a459-d3ceeaacd638'. {{(pid=123698) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} May 02 10:39:23.713074 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] Volume file is RBD: attempting optimized backup {{(pid=123698) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}} May 02 10:39:23.759604 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] Creating base image 'volume-f8a426a0-2921-4da7-a459-d3ceeaacd638.backup.0c8a6cc7-d457-4a3d-8d37-477336ffc040' {{(pid=123698) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}} May 02 10:39:23.760023 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=123698) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} May 02 10:39:25.059154 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] Using --from-snap 'None' for incremental backup of volume f8a426a0-2921-4da7-a459-d3ceeaacd638. {{(pid=123698) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}} May 02 10:39:25.059754 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] Creating backup snapshot='backup.0c8a6cc7-d457-4a3d-8d37-477336ffc040.snap.1651487965.0591807' {{(pid=123698) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}} May 02 10:39:26.856846 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] Performing differential transfer from 'volume-f8a426a0-2921-4da7-a459-d3ceeaacd638' to 'volume-f8a426a0-2921-4da7-a459-d3ceeaacd638.backup.0c8a6cc7-d457-4a3d-8d37-477336ffc040' {{(pid=123698) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} May 02 10:39:26.857500 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_vjc5tkd_ --pool volumes volumes/volume-f8a426a0-2921-4da7-a459-d3ceeaacd638@backup.0c8a6cc7-d457-4a3d-8d37-477336ffc040.snap.1651487965.0591807 -' into... {{(pid=123698) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} May 02 10:39:26.857650 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-f8a426a0-2921-4da7-a459-d3ceeaacd638.backup.0c8a6cc7-d457-4a3d-8d37-477336ffc040' {{(pid=123698) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} May 02 10:39:30.715497 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] Differential backup transfer completed in 3.8584s {{(pid=123698) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}} May 02 10:39:30.716148 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] Getting metadata type 'volume-base-metadata' {{(pid=123698) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 02 10:39:30.727689 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=123698) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 02 10:39:30.728758 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] Getting metadata type 'volume-metadata' {{(pid=123698) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 02 10:39:30.735053 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] No metadata type 'volume-metadata' available {{(pid=123698) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 02 10:39:30.735431 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] Getting metadata type 'volume-glance-metadata' {{(pid=123698) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 02 10:39:30.741681 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] No metadata type 'volume-glance-metadata' available {{(pid=123698) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} May 02 10:39:30.742217 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] Backing up metadata for volume f8a426a0-2921-4da7-a459-d3ceeaacd638. {{(pid=123698) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} May 02 10:39:31.223687 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] Backup '0c8a6cc7-d457-4a3d-8d37-477336ffc040' of volume f8a426a0-2921-4da7-a459-d3ceeaacd638 finished. {{(pid=123698) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} May 02 10:39:31.225572 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-f8a426a0-2921-4da7-a459-d3ceeaacd638', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'f8a426a0-2921-4da7-a459-d3ceeaacd638', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 10:39:31.987371 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] <== disconnect_volume: return (760ms) None {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 10:39:32.174209 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:39:32.174566 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:39:32.289258 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-33143db9-c27b-47f8-acd5-26cf026540be tempest-VolumesBackupsV39Test-169541720 None] Create backup finished. backup: 0c8a6cc7-d457-4a3d-8d37-477336ffc040. May 02 10:39:33.602077 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-76989f0d-8f39-4c57-9f60-46a1bd7d7031 tempest-VolumesBackupsV39Test-169541720 None] Delete backup started, backup: 0c8a6cc7-d457-4a3d-8d37-477336ffc040. May 02 10:39:33.602077 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-76989f0d-8f39-4c57-9f60-46a1bd7d7031 tempest-VolumesBackupsV39Test-169541720 None] Delete started for backup=0c8a6cc7-d457-4a3d-8d37-477336ffc040 {{(pid=123698) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} May 02 10:39:33.602077 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-76989f0d-8f39-4c57-9f60-46a1bd7d7031 tempest-VolumesBackupsV39Test-169541720 None] Trying diff format basename='volume-f8a426a0-2921-4da7-a459-d3ceeaacd638.backup.0c8a6cc7-d457-4a3d-8d37-477336ffc040' for backup base image of volume f8a426a0-2921-4da7-a459-d3ceeaacd638. {{(pid=123698) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} May 02 10:39:33.925284 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-76989f0d-8f39-4c57-9f60-46a1bd7d7031 tempest-VolumesBackupsV39Test-169541720 None] Looking for snapshot of backup base 'volume-f8a426a0-2921-4da7-a459-d3ceeaacd638.backup.0c8a6cc7-d457-4a3d-8d37-477336ffc040' {{(pid=123698) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 02 10:39:33.925997 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-76989f0d-8f39-4c57-9f60-46a1bd7d7031 tempest-VolumesBackupsV39Test-169541720 None] Found snapshot 'backup.0c8a6cc7-d457-4a3d-8d37-477336ffc040.snap.1651487965.0591807' {{(pid=123698) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} May 02 10:39:33.926444 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-76989f0d-8f39-4c57-9f60-46a1bd7d7031 tempest-VolumesBackupsV39Test-169541720 None] Deleting backup snapshot='backup.0c8a6cc7-d457-4a3d-8d37-477336ffc040.snap.1651487965.0591807' {{(pid=123698) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}} May 02 10:39:37.514347 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.drivers.ceph [None req-76989f0d-8f39-4c57-9f60-46a1bd7d7031 tempest-VolumesBackupsV39Test-169541720 None] Deleting backup base image='volume-f8a426a0-2921-4da7-a459-d3ceeaacd638.backup.0c8a6cc7-d457-4a3d-8d37-477336ffc040' of volume f8a426a0-2921-4da7-a459-d3ceeaacd638. May 02 10:39:40.321633 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_db.sqlalchemy.engines [None req-e2e69ca3-23ac-4614-a31d-4f46fe2b66d3 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123698) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 02 10:39:46.609615 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-76989f0d-8f39-4c57-9f60-46a1bd7d7031 tempest-VolumesBackupsV39Test-169541720 None] Base backup image='volume-f8a426a0-2921-4da7-a459-d3ceeaacd638.backup.0c8a6cc7-d457-4a3d-8d37-477336ffc040' of volume f8a426a0-2921-4da7-a459-d3ceeaacd638 deleted. {{(pid=123698) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} May 02 10:39:46.901549 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-76989f0d-8f39-4c57-9f60-46a1bd7d7031 tempest-VolumesBackupsV39Test-169541720 None] Delete of backup '0c8a6cc7-d457-4a3d-8d37-477336ffc040' for volume 'f8a426a0-2921-4da7-a459-d3ceeaacd638' finished. {{(pid=123698) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} May 02 10:39:47.056360 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.quota [None req-76989f0d-8f39-4c57-9f60-46a1bd7d7031 tempest-VolumesBackupsV39Test-169541720 None] Created reservations ['deeaed17-20ff-42a7-990a-bdc8f25fee13', 'c693013b-1509-4d9d-9e3c-399748883ca3'] {{(pid=123698) reserve /opt/stack/cinder/cinder/quota.py:755}} May 02 10:39:47.267046 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-76989f0d-8f39-4c57-9f60-46a1bd7d7031 tempest-VolumesBackupsV39Test-169541720 None] Delete backup finished, backup 0c8a6cc7-d457-4a3d-8d37-477336ffc040 deleted. May 02 10:40:32.172267 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:40:32.173954 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:41:32.172198 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:41:32.174735 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:42:32.171575 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:42:32.172266 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:43:32.167321 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:43:32.167903 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:44:32.175153 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:44:32.179916 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:45:32.176076 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:45:32.176715 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:45:55.830910 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] Create backup started, backup: 1fda047b-3fa4-47f3-9d96-75a2cc7a3402 volume: f570807a-66b8-416f-b0c4-f488a0f13513 snapshot: e80dbe72-5018-4eb1-abbb-192fc2c0d6df. May 02 10:45:55.929232 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-02T10:45:55Z,data_timestamp=2022-05-02T10:45:52Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Backup-67453090',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=1fda047b-3fa4-47f3-9d96-75a2cc7a3402,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='23dc31c03ada4eba97ae47aab53487d9',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id='e80dbe72-5018-4eb1-abbb-192fc2c0d6df',status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='f58c631280fb4cdb917abc34660e3325',volume_id=f570807a-66b8-416f-b0c4-f488a0f13513) May 02 10:45:57.880165 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.utils [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.16', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 10:45:57.883838 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:45:57.911328 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.028s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:45:57.912038 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[b10f0945-6973-45c3-ac63-f42371f898f8]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:9d66b0da6823\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:45:57.916892 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:45:57.928251 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 02 10:45:57.928823 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[6aac31a4-c2ef-4c70-9230-39d3f64c77db]: [Errno 2] No such file or directory: 'systool' {{(pid=154446) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 02 10:45:57.928823 ncdv-master-1 cinder-backup[123698]: Traceback (most recent call last): May 02 10:45:57.928823 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 02 10:45:57.928823 ncdv-master-1 cinder-backup[123698]: ret = func(*f_args, **f_kwargs) May 02 10:45:57.928823 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 02 10:45:57.928823 ncdv-master-1 cinder-backup[123698]: return func(*args, **kwargs) May 02 10:45:57.928823 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 02 10:45:57.928823 ncdv-master-1 cinder-backup[123698]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 02 10:45:57.928823 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 02 10:45:57.928823 ncdv-master-1 cinder-backup[123698]: return putils.execute(on_execute=on_execute, May 02 10:45:57.928823 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 02 10:45:57.928823 ncdv-master-1 cinder-backup[123698]: obj = subprocess.Popen(cmd, May 02 10:45:57.928823 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 02 10:45:57.928823 ncdv-master-1 cinder-backup[123698]: self._execute_child(args, executable, preexec_fn, close_fds, May 02 10:45:57.928823 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 02 10:45:57.928823 ncdv-master-1 cinder-backup[123698]: raise child_exception_type(errno_num, err_msg, err_filename) May 02 10:45:57.928823 ncdv-master-1 cinder-backup[123698]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 02 10:45:57.930405 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[6aac31a4-c2ef-4c70-9230-39d3f64c77db]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:45:57.934376 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:45:57.942928 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 02 10:45:57.942928 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[1577b0c4-4eca-4f6a-b4ac-62d91971da87]: [Errno 2] No such file or directory: 'systool' {{(pid=154446) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 02 10:45:57.942928 ncdv-master-1 cinder-backup[123698]: Traceback (most recent call last): May 02 10:45:57.942928 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 02 10:45:57.942928 ncdv-master-1 cinder-backup[123698]: ret = func(*f_args, **f_kwargs) May 02 10:45:57.942928 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 02 10:45:57.942928 ncdv-master-1 cinder-backup[123698]: return func(*args, **kwargs) May 02 10:45:57.942928 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 02 10:45:57.942928 ncdv-master-1 cinder-backup[123698]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 02 10:45:57.942928 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 02 10:45:57.942928 ncdv-master-1 cinder-backup[123698]: return putils.execute(on_execute=on_execute, May 02 10:45:57.942928 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 02 10:45:57.942928 ncdv-master-1 cinder-backup[123698]: obj = subprocess.Popen(cmd, May 02 10:45:57.942928 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 02 10:45:57.942928 ncdv-master-1 cinder-backup[123698]: self._execute_child(args, executable, preexec_fn, close_fds, May 02 10:45:57.942928 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 02 10:45:57.942928 ncdv-master-1 cinder-backup[123698]: raise child_exception_type(errno_num, err_msg, err_filename) May 02 10:45:57.942928 ncdv-master-1 cinder-backup[123698]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 02 10:45:57.942928 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[1577b0c4-4eca-4f6a-b4ac-62d91971da87]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:45:57.948525 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:45:57.975844 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.029s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:45:57.976082 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[28606fc2-d1e2-4cd5-9be6-6bf35c599339]: (4, ('/dev/sda1\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:45:57.979423 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:45:58.043080 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.064s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:45:58.043732 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[60544f07-ce60-46a4-98fa-5d75d8f665b9]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:45:58.049367 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:45:58.072665 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.027s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:45:58.073317 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[59b137f3-37df-4d93-b445-e56b33bcc3ad]: (4, ('9ace392b-cf5e-9c45-a31c-76b2b08d55ae\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:45:58.074324 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] Running cmd (subprocess): nvme version {{(pid=123698) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:45:58.132733 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] CMD "nvme version" returned: 0 in 0.058s {{(pid=123698) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:45:58.134900 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=123698) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 02 10:45:58.135269 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=123698) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 02 10:45:58.135936 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:7c8447db-24f2-4326-a844-150f6bd61ca4 dsc: {{(pid=123698) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 02 10:45:58.136579 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.utils [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] <== get_connector_properties: return (255ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.16', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:9d66b0da6823', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '9ace392b-cf5e-9c45-a31c-76b2b08d55ae', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:7c8447db-24f2-4326-a844-150f6bd61ca4', 'found_dsc': ''} {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 10:45:59.319033 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connector [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] Factory for rbd on None {{(pid=123698) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 02 10:45:59.320840 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-b2af681b-29a3-4460-a5c7-613eda5347ef', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'b2af681b-29a3-4460-a5c7-613eda5347ef', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 10:45:59.322077 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.linuxrbd [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] opening connection to ceph cluster (timeout=-1). {{(pid=123698) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 02 10:45:59.514983 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] <== connect_volume: return (192ms) {'path': } {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 10:45:59.524337 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_db.sqlalchemy.engines [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123698) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 02 10:45:59.655588 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] Starting backup of volume='f570807a-66b8-416f-b0c4-f488a0f13513'. {{(pid=123698) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} May 02 10:45:59.690222 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=123698) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} May 02 10:45:59.690625 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] Creating backup base image='volume-f570807a-66b8-416f-b0c4-f488a0f13513.backup.base' for volume f570807a-66b8-416f-b0c4-f488a0f13513. {{(pid=123698) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:875}} May 02 10:46:00.357737 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] Copying data from volume f570807a-66b8-416f-b0c4-f488a0f13513. {{(pid=123698) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:887}} May 02 10:46:00.585563 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] Transferring data between 'volume-f570807a-66b8-416f-b0c4-f488a0f13513' and 'volume-f570807a-66b8-416f-b0c4-f488a0f13513.backup.base' {{(pid=123698) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:417}} May 02 10:46:00.585877 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] 8 chunks of 134217728 bytes to be transferred {{(pid=123698) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:421}} May 02 10:46:11.333424 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] Transferred chunk 1 of 8 (12196K/s) {{(pid=123698) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} May 02 10:46:21.836331 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] Transferred chunk 2 of 8 (12481K/s) {{(pid=123698) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} May 02 10:46:32.175728 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:46:32.176444 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:46:32.653333 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] Transferred chunk 3 of 8 (12119K/s) {{(pid=123698) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} May 02 10:46:43.377153 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] Transferred chunk 4 of 8 (12223K/s) {{(pid=123698) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} May 02 10:46:54.711148 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] Transferred chunk 5 of 8 (11565K/s) {{(pid=123698) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} May 02 10:47:06.446479 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] Transferred chunk 6 of 8 (11171K/s) {{(pid=123698) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} May 02 10:47:16.846120 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] Transferred chunk 7 of 8 (12604K/s) {{(pid=123698) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} May 02 10:47:28.440164 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] Transferred chunk 8 of 8 (11304K/s) {{(pid=123698) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} May 02 10:47:28.880170 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] Getting metadata type 'volume-base-metadata' {{(pid=123698) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 02 10:47:28.904888 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=123698) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 02 10:47:28.906309 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] Getting metadata type 'volume-metadata' {{(pid=123698) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 02 10:47:28.916465 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] No metadata type 'volume-metadata' available {{(pid=123698) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 02 10:47:28.917737 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] Getting metadata type 'volume-glance-metadata' {{(pid=123698) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 02 10:47:28.925455 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] No metadata type 'volume-glance-metadata' available {{(pid=123698) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} May 02 10:47:28.931904 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] Backing up metadata for volume f570807a-66b8-416f-b0c4-f488a0f13513. {{(pid=123698) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} May 02 10:47:29.116705 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] Backup '1fda047b-3fa4-47f3-9d96-75a2cc7a3402' of volume f570807a-66b8-416f-b0c4-f488a0f13513 finished. {{(pid=123698) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} May 02 10:47:29.118269 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-b2af681b-29a3-4460-a5c7-613eda5347ef', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'b2af681b-29a3-4460-a5c7-613eda5347ef', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 10:47:29.241549 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] <== disconnect_volume: return (122ms) None {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 10:47:29.618031 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-dd1e626a-c41e-4e32-a321-7c42923675e3 tempest-VolumesSnapshotTestJSON-616137826 None] Create backup finished. backup: 1fda047b-3fa4-47f3-9d96-75a2cc7a3402. May 02 10:47:30.668999 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-66a0fed4-c39b-4218-b1a2-517c9fb14459 tempest-VolumesSnapshotTestJSON-616137826 None] Delete backup started, backup: 1fda047b-3fa4-47f3-9d96-75a2cc7a3402. May 02 10:47:30.671473 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-66a0fed4-c39b-4218-b1a2-517c9fb14459 tempest-VolumesSnapshotTestJSON-616137826 None] Delete started for backup=1fda047b-3fa4-47f3-9d96-75a2cc7a3402 {{(pid=123698) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} May 02 10:47:30.672216 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-66a0fed4-c39b-4218-b1a2-517c9fb14459 tempest-VolumesSnapshotTestJSON-616137826 None] Trying diff format basename='volume-f570807a-66b8-416f-b0c4-f488a0f13513.backup.1fda047b-3fa4-47f3-9d96-75a2cc7a3402' for backup base image of volume f570807a-66b8-416f-b0c4-f488a0f13513. {{(pid=123698) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} May 02 10:47:30.712288 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-66a0fed4-c39b-4218-b1a2-517c9fb14459 tempest-VolumesSnapshotTestJSON-616137826 None] Image 'volume-f570807a-66b8-416f-b0c4-f488a0f13513.backup.1fda047b-3fa4-47f3-9d96-75a2cc7a3402' not found - trying diff format name {{(pid=123698) _rbd_image_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:693}} May 02 10:47:30.982670 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-66a0fed4-c39b-4218-b1a2-517c9fb14459 tempest-VolumesSnapshotTestJSON-616137826 None] Looking for snapshot of backup base 'volume-f570807a-66b8-416f-b0c4-f488a0f13513.backup.base' {{(pid=123698) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 02 10:47:30.983221 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-66a0fed4-c39b-4218-b1a2-517c9fb14459 tempest-VolumesSnapshotTestJSON-616137826 None] Backup base 'volume-f570807a-66b8-416f-b0c4-f488a0f13513.backup.base' has no snapshots {{(pid=123698) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:955}} May 02 10:47:30.983700 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-66a0fed4-c39b-4218-b1a2-517c9fb14459 tempest-VolumesSnapshotTestJSON-616137826 None] No backup snapshot to delete {{(pid=123698) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:500}} May 02 10:47:31.263998 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.drivers.ceph [None req-66a0fed4-c39b-4218-b1a2-517c9fb14459 tempest-VolumesSnapshotTestJSON-616137826 None] Deleting backup base image='volume-f570807a-66b8-416f-b0c4-f488a0f13513.backup.base' of volume f570807a-66b8-416f-b0c4-f488a0f13513. May 02 10:47:32.180768 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:47:32.186742 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:47:40.330486 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_db.sqlalchemy.engines [None req-e2e69ca3-23ac-4614-a31d-4f46fe2b66d3 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123698) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 02 10:47:45.371191 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-66a0fed4-c39b-4218-b1a2-517c9fb14459 tempest-VolumesSnapshotTestJSON-616137826 None] Base backup image='volume-f570807a-66b8-416f-b0c4-f488a0f13513.backup.base' of volume f570807a-66b8-416f-b0c4-f488a0f13513 deleted. {{(pid=123698) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} May 02 10:47:45.571274 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-66a0fed4-c39b-4218-b1a2-517c9fb14459 tempest-VolumesSnapshotTestJSON-616137826 None] Delete of backup '1fda047b-3fa4-47f3-9d96-75a2cc7a3402' for volume 'f570807a-66b8-416f-b0c4-f488a0f13513' finished. {{(pid=123698) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} May 02 10:47:45.721180 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.quota [None req-66a0fed4-c39b-4218-b1a2-517c9fb14459 tempest-VolumesSnapshotTestJSON-616137826 None] Created reservations ['3c044485-cb28-44bd-984b-8c5a9255fd9c', '0f054590-978e-438c-8843-8d1edff9e16f'] {{(pid=123698) reserve /opt/stack/cinder/cinder/quota.py:755}} May 02 10:47:45.874495 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-66a0fed4-c39b-4218-b1a2-517c9fb14459 tempest-VolumesSnapshotTestJSON-616137826 None] Delete backup finished, backup 1fda047b-3fa4-47f3-9d96-75a2cc7a3402 deleted. May 02 10:48:32.184509 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:48:32.186274 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:49:32.184302 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:49:32.188415 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:50:32.189571 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:50:32.190312 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:51:29.390055 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] Create backup started, backup: 5d54b583-3ca9-4e90-aa1f-0b0ffb591de2 volume: d12d006a-e3e5-424d-b4d4-5836d1700b51. May 02 10:51:29.452235 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-02T10:51:28Z,data_timestamp=2022-05-02T10:51:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsAdminTest-Backup-1081081643',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=5d54b583-3ca9-4e90-aa1f-0b0ffb591de2,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='ed2eac51a96c4377a9837675807d0bbd',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='456241eca666492ca5b25dfd2fb53769',volume_id=d12d006a-e3e5-424d-b4d4-5836d1700b51) May 02 10:51:29.495246 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.utils [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.16', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 10:51:29.497216 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:51:29.506539 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.009s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:51:29.506964 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[d1c75d80-175e-4011-b90b-37f25e9a3f18]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:9d66b0da6823\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:51:29.509342 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:51:29.520412 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 02 10:51:29.520789 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[067f19c3-73e6-4336-a443-71ac3a82f72b]: [Errno 2] No such file or directory: 'systool' {{(pid=154446) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 02 10:51:29.520789 ncdv-master-1 cinder-backup[123698]: Traceback (most recent call last): May 02 10:51:29.520789 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 02 10:51:29.520789 ncdv-master-1 cinder-backup[123698]: ret = func(*f_args, **f_kwargs) May 02 10:51:29.520789 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 02 10:51:29.520789 ncdv-master-1 cinder-backup[123698]: return func(*args, **kwargs) May 02 10:51:29.520789 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 02 10:51:29.520789 ncdv-master-1 cinder-backup[123698]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 02 10:51:29.520789 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 02 10:51:29.520789 ncdv-master-1 cinder-backup[123698]: return putils.execute(on_execute=on_execute, May 02 10:51:29.520789 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 02 10:51:29.520789 ncdv-master-1 cinder-backup[123698]: obj = subprocess.Popen(cmd, May 02 10:51:29.520789 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 02 10:51:29.520789 ncdv-master-1 cinder-backup[123698]: self._execute_child(args, executable, preexec_fn, close_fds, May 02 10:51:29.520789 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 02 10:51:29.520789 ncdv-master-1 cinder-backup[123698]: raise child_exception_type(errno_num, err_msg, err_filename) May 02 10:51:29.520789 ncdv-master-1 cinder-backup[123698]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 02 10:51:29.521798 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[067f19c3-73e6-4336-a443-71ac3a82f72b]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:51:29.525222 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:51:29.531215 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 02 10:51:29.531953 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[eccb3c88-6f04-479f-95d5-59eb49b41f8d]: [Errno 2] No such file or directory: 'systool' {{(pid=154446) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 02 10:51:29.531953 ncdv-master-1 cinder-backup[123698]: Traceback (most recent call last): May 02 10:51:29.531953 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 02 10:51:29.531953 ncdv-master-1 cinder-backup[123698]: ret = func(*f_args, **f_kwargs) May 02 10:51:29.531953 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 02 10:51:29.531953 ncdv-master-1 cinder-backup[123698]: return func(*args, **kwargs) May 02 10:51:29.531953 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 02 10:51:29.531953 ncdv-master-1 cinder-backup[123698]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 02 10:51:29.531953 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 02 10:51:29.531953 ncdv-master-1 cinder-backup[123698]: return putils.execute(on_execute=on_execute, May 02 10:51:29.531953 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 02 10:51:29.531953 ncdv-master-1 cinder-backup[123698]: obj = subprocess.Popen(cmd, May 02 10:51:29.531953 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 02 10:51:29.531953 ncdv-master-1 cinder-backup[123698]: self._execute_child(args, executable, preexec_fn, close_fds, May 02 10:51:29.531953 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 02 10:51:29.531953 ncdv-master-1 cinder-backup[123698]: raise child_exception_type(errno_num, err_msg, err_filename) May 02 10:51:29.531953 ncdv-master-1 cinder-backup[123698]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 02 10:51:29.536887 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[eccb3c88-6f04-479f-95d5-59eb49b41f8d]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:51:29.536887 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:51:29.542824 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.009s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:51:29.543108 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[524f3f29-5f40-46d8-97b7-91055407f07a]: (4, ('/dev/sda1\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:51:29.544242 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:51:29.553939 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.009s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:51:29.554270 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[9d579a89-28ca-4def-8f72-de3dd39e219d]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:51:29.555412 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:51:29.565182 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.009s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:51:29.565506 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[8a0cc943-2af3-48a3-9e52-69266d89e14e]: (4, ('9ace392b-cf5e-9c45-a31c-76b2b08d55ae\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:51:29.566268 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] Running cmd (subprocess): nvme version {{(pid=123698) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:51:29.587888 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] CMD "nvme version" returned: 0 in 0.021s {{(pid=123698) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:51:29.590159 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=123698) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 02 10:51:29.590529 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=123698) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 02 10:51:29.590900 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:7c8447db-24f2-4326-a844-150f6bd61ca4 dsc: {{(pid=123698) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 02 10:51:29.591529 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.utils [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] <== get_connector_properties: return (95ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.16', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:9d66b0da6823', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '9ace392b-cf5e-9c45-a31c-76b2b08d55ae', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:7c8447db-24f2-4326-a844-150f6bd61ca4', 'found_dsc': ''} {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 10:51:30.327580 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connector [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] Factory for rbd on None {{(pid=123698) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 02 10:51:30.336034 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-d12d006a-e3e5-424d-b4d4-5836d1700b51', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'd12d006a-e3e5-424d-b4d4-5836d1700b51', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 10:51:30.336034 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.linuxrbd [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] opening connection to ceph cluster (timeout=-1). {{(pid=123698) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 02 10:51:30.503642 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] <== connect_volume: return (172ms) {'path': } {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 10:51:30.524132 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_db.sqlalchemy.engines [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123698) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 02 10:51:30.596484 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] Starting backup of volume='d12d006a-e3e5-424d-b4d4-5836d1700b51'. {{(pid=123698) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} May 02 10:51:30.597215 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] Volume file is RBD: attempting optimized backup {{(pid=123698) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}} May 02 10:51:30.620352 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] Creating base image 'volume-d12d006a-e3e5-424d-b4d4-5836d1700b51.backup.5d54b583-3ca9-4e90-aa1f-0b0ffb591de2' {{(pid=123698) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}} May 02 10:51:30.625210 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=123698) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} May 02 10:51:31.046106 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] Using --from-snap 'None' for incremental backup of volume d12d006a-e3e5-424d-b4d4-5836d1700b51. {{(pid=123698) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}} May 02 10:51:31.051196 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] Creating backup snapshot='backup.5d54b583-3ca9-4e90-aa1f-0b0ffb591de2.snap.1651488691.0461326' {{(pid=123698) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}} May 02 10:51:32.194380 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:51:32.195041 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:51:33.261457 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] Performing differential transfer from 'volume-d12d006a-e3e5-424d-b4d4-5836d1700b51' to 'volume-d12d006a-e3e5-424d-b4d4-5836d1700b51.backup.5d54b583-3ca9-4e90-aa1f-0b0ffb591de2' {{(pid=123698) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} May 02 10:51:33.262519 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_vpop7t4w --pool volumes volumes/volume-d12d006a-e3e5-424d-b4d4-5836d1700b51@backup.5d54b583-3ca9-4e90-aa1f-0b0ffb591de2.snap.1651488691.0461326 -' into... {{(pid=123698) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} May 02 10:51:33.262885 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-d12d006a-e3e5-424d-b4d4-5836d1700b51.backup.5d54b583-3ca9-4e90-aa1f-0b0ffb591de2' {{(pid=123698) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} May 02 10:51:35.230483 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] Differential backup transfer completed in 1.9691s {{(pid=123698) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}} May 02 10:51:35.231144 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] Getting metadata type 'volume-base-metadata' {{(pid=123698) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 02 10:51:35.244910 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=123698) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 02 10:51:35.245507 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] Getting metadata type 'volume-metadata' {{(pid=123698) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 02 10:51:35.252734 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] No metadata type 'volume-metadata' available {{(pid=123698) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 02 10:51:35.253311 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] Getting metadata type 'volume-glance-metadata' {{(pid=123698) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 02 10:51:35.261678 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] No metadata type 'volume-glance-metadata' available {{(pid=123698) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} May 02 10:51:35.262522 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] Backing up metadata for volume d12d006a-e3e5-424d-b4d4-5836d1700b51. {{(pid=123698) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} May 02 10:51:35.473161 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] Backup '5d54b583-3ca9-4e90-aa1f-0b0ffb591de2' of volume d12d006a-e3e5-424d-b4d4-5836d1700b51 finished. {{(pid=123698) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} May 02 10:51:35.475346 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-d12d006a-e3e5-424d-b4d4-5836d1700b51', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'd12d006a-e3e5-424d-b4d4-5836d1700b51', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 10:51:35.806623 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] <== disconnect_volume: return (329ms) None {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 10:51:35.984234 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-b6b3eb7d-d5cc-4117-a577-611766c00562 tempest-VolumesBackupsAdminTest-292168352 None] Create backup finished. backup: 5d54b583-3ca9-4e90-aa1f-0b0ffb591de2. May 02 10:51:36.789465 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-7127fd68-6588-4e9f-96e3-86d80cdfe51b tempest-VolumesBackupsAdminTest-1865443922 None] Export record started, backup: 5d54b583-3ca9-4e90-aa1f-0b0ffb591de2. May 02 10:51:36.793129 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-7127fd68-6588-4e9f-96e3-86d80cdfe51b tempest-VolumesBackupsAdminTest-1865443922 None] Export record finished, backup 5d54b583-3ca9-4e90-aa1f-0b0ffb591de2 exported. May 02 10:51:37.195327 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-ebbfc648-2a66-4f2c-8eee-0cebcbed1416 tempest-VolumesBackupsAdminTest-1865443922 None] Import record started, backup_url: eyJkcml2ZXJfaW5mbyI6IHt9LCAiaWQiOiAiNTY0ZTViOGItYjNlYS00YWU5LWEwNmItNWI4MGUwMDc5YzhjIiwgInVzZXJfaWQiOiAiNDU2MjQxZWNhNjY2NDkyY2E1YjI1ZGZkMmZiNTM3NjkiLCAicHJvamVjdF9pZCI6ICJlZDJlYWM1MWE5NmM0Mzc3YTk4Mzc2NzU4MDdkMGJiZCIsICJ2b2x1bWVfaWQiOiAiZDEyZDAwNmEtZTNlNS00MjRkLWI0ZDQtNTgzNmQxNzAwYjUxIiwgImhvc3QiOiAibmNkdi1tYXN0ZXItMSIsICJhdmFpbGFiaWxpdHlfem9uZSI6IG51bGwsICJjb250YWluZXIiOiAiYmFja3VwcyIsICJwYXJlbnRfaWQiOiBudWxsLCAic3RhdHVzIjogImF2YWlsYWJsZSIsICJmYWlsX3JlYXNvbiI6IG51bGwsICJzaXplIjogMSwgImRpc3BsYXlfbmFtZSI6ICJ0ZW1wZXN0LVZvbHVtZXNCYWNrdXBzQWRtaW5UZXN0LUJhY2t1cC0xMDgxMDgxNjQzIiwgImRpc3BsYXlfZGVzY3JpcHRpb24iOiBudWxsLCAic2VydmljZV9tZXRhZGF0YSI6ICJ7XCJiYXNlXCI6IFwidm9sdW1lLWQxMmQwMDZhLWUzZTUtNDI0ZC1iNGQ0LTU4MzZkMTcwMGI1MS5iYWNrdXAuNWQ1NGI1ODMtM2NhOS00ZTkwLWFhMWYtMGIwZmZiNTkxZGUyXCJ9IiwgInNlcnZpY2UiOiAiY2luZGVyLmJhY2t1cC5kcml2ZXJzLmNlcGguQ2VwaEJhY2t1cERyaXZlciIsICJvYmplY3RfY291bnQiOiAwLCAidGVtcF92b2x1bWVfaWQiOiBudWxsLCAidGVtcF9zbmFwc2hvdF9pZCI6IG51bGwsICJudW1fZGVwZW5kZW50X2JhY2t1cHMiOiAwLCAic25hcHNob3RfaWQiOiBudWxsLCAiZGF0YV90aW1lc3RhbXAiOiAiMjAyMi0wNS0wMlQxMDo1MToyOVoiLCAicmVzdG9yZV92b2x1bWVfaWQiOiBudWxsLCAibWV0YWRhdGEiOiB7fSwgImVuY3J5cHRpb25fa2V5X2lkIjogbnVsbCwgImNyZWF0ZWRfYXQiOiAiMjAyMi0wNS0wMlQxMDo1MToyOVoiLCAidXBkYXRlZF9hdCI6ICIyMDIyLTA1LTAyVDEwOjUxOjM2WiIsICJkZWxldGVkX2F0IjogbnVsbCwgImRlbGV0ZWQiOiBmYWxzZX0=. May 02 10:51:37.210799 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_db.sqlalchemy.engines [None req-ebbfc648-2a66-4f2c-8eee-0cebcbed1416 tempest-VolumesBackupsAdminTest-1865443922 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123698) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 02 10:51:37.367746 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-ebbfc648-2a66-4f2c-8eee-0cebcbed1416 tempest-VolumesBackupsAdminTest-1865443922 None] Import record id 564e5b8b-b3ea-4ae9-a06b-5b80e0079c8c metadata from driver finished. May 02 10:51:41.146881 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] Restore backup started, backup: 5d54b583-3ca9-4e90-aa1f-0b0ffb591de2 volume: ee79e3e6-c565-4dfd-8225-fa966d6fde80. May 02 10:51:41.165001 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.utils [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.16', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 10:51:41.166428 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:51:41.177590 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.011s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:51:41.177984 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[27d94309-9a15-4b0b-94a6-f247549ff19e]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:9d66b0da6823\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:51:41.182038 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:51:41.196751 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 02 10:51:41.199943 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[8582fa28-962a-4b35-8ef7-8cad3f8e490c]: [Errno 2] No such file or directory: 'systool' {{(pid=154446) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 02 10:51:41.199943 ncdv-master-1 cinder-backup[123698]: Traceback (most recent call last): May 02 10:51:41.199943 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 02 10:51:41.199943 ncdv-master-1 cinder-backup[123698]: ret = func(*f_args, **f_kwargs) May 02 10:51:41.199943 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 02 10:51:41.199943 ncdv-master-1 cinder-backup[123698]: return func(*args, **kwargs) May 02 10:51:41.199943 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 02 10:51:41.199943 ncdv-master-1 cinder-backup[123698]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 02 10:51:41.199943 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 02 10:51:41.199943 ncdv-master-1 cinder-backup[123698]: return putils.execute(on_execute=on_execute, May 02 10:51:41.199943 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 02 10:51:41.199943 ncdv-master-1 cinder-backup[123698]: obj = subprocess.Popen(cmd, May 02 10:51:41.199943 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 02 10:51:41.199943 ncdv-master-1 cinder-backup[123698]: self._execute_child(args, executable, preexec_fn, close_fds, May 02 10:51:41.199943 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 02 10:51:41.199943 ncdv-master-1 cinder-backup[123698]: raise child_exception_type(errno_num, err_msg, err_filename) May 02 10:51:41.199943 ncdv-master-1 cinder-backup[123698]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 02 10:51:41.199943 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[8582fa28-962a-4b35-8ef7-8cad3f8e490c]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:51:41.206579 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:51:41.220567 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 02 10:51:41.221222 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[a57a6637-0b31-49dc-81a5-d691105a5500]: [Errno 2] No such file or directory: 'systool' {{(pid=154446) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 02 10:51:41.221222 ncdv-master-1 cinder-backup[123698]: Traceback (most recent call last): May 02 10:51:41.221222 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 02 10:51:41.221222 ncdv-master-1 cinder-backup[123698]: ret = func(*f_args, **f_kwargs) May 02 10:51:41.221222 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 02 10:51:41.221222 ncdv-master-1 cinder-backup[123698]: return func(*args, **kwargs) May 02 10:51:41.221222 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 02 10:51:41.221222 ncdv-master-1 cinder-backup[123698]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 02 10:51:41.221222 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 02 10:51:41.221222 ncdv-master-1 cinder-backup[123698]: return putils.execute(on_execute=on_execute, May 02 10:51:41.221222 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 02 10:51:41.221222 ncdv-master-1 cinder-backup[123698]: obj = subprocess.Popen(cmd, May 02 10:51:41.221222 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 02 10:51:41.221222 ncdv-master-1 cinder-backup[123698]: self._execute_child(args, executable, preexec_fn, close_fds, May 02 10:51:41.221222 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 02 10:51:41.221222 ncdv-master-1 cinder-backup[123698]: raise child_exception_type(errno_num, err_msg, err_filename) May 02 10:51:41.221222 ncdv-master-1 cinder-backup[123698]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 02 10:51:41.222788 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[a57a6637-0b31-49dc-81a5-d691105a5500]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:51:41.224625 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:51:41.237278 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.012s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:51:41.238295 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[09687203-c719-49e5-b846-aeb507efa5c9]: (4, ('/dev/sda1\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:51:41.241443 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:51:41.257705 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.017s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:51:41.258116 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[3a461469-b615-4d34-a37d-75ae83c6d99a]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:51:41.262369 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:51:41.271403 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.012s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:51:41.271755 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[b8ddbe21-3c52-49dd-84c5-fd12579d9780]: (4, ('9ace392b-cf5e-9c45-a31c-76b2b08d55ae\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:51:41.272591 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] Running cmd (subprocess): nvme version {{(pid=123698) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:51:41.314403 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] CMD "nvme version" returned: 0 in 0.036s {{(pid=123698) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:51:41.314403 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=123698) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 02 10:51:41.314403 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=123698) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 02 10:51:41.314403 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:7c8447db-24f2-4326-a844-150f6bd61ca4 dsc: {{(pid=123698) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 02 10:51:41.314817 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.utils [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] <== get_connector_properties: return (147ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.16', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:9d66b0da6823', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '9ace392b-cf5e-9c45-a31c-76b2b08d55ae', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:7c8447db-24f2-4326-a844-150f6bd61ca4', 'found_dsc': ''} {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 10:51:42.063092 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connector [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] Factory for rbd on None {{(pid=123698) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 02 10:51:42.064759 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-ee79e3e6-c565-4dfd-8225-fa966d6fde80', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'ee79e3e6-c565-4dfd-8225-fa966d6fde80', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 10:51:42.065950 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.linuxrbd [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] opening connection to ceph cluster (timeout=-1). {{(pid=123698) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 02 10:51:42.327261 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] <== connect_volume: return (261ms) {'path': } {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 10:51:42.335811 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_db.sqlalchemy.engines [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123698) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 02 10:51:42.346713 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] Starting restore from Ceph backup=5d54b583-3ca9-4e90-aa1f-0b0ffb591de2 to volume=volume-ee79e3e6-c565-4dfd-8225-fa966d6fde80 {{(pid=123698) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1343}} May 02 10:51:42.521905 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] Looking for snapshot of backup base 'volume-d12d006a-e3e5-424d-b4d4-5836d1700b51.backup.5d54b583-3ca9-4e90-aa1f-0b0ffb591de2' {{(pid=123698) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 02 10:51:42.522336 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] Found snapshot 'backup.5d54b583-3ca9-4e90-aa1f-0b0ffb591de2.snap.1651488691.0461326' {{(pid=123698) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} May 02 10:51:42.527319 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] Volume file is RBD. {{(pid=123698) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1245}} May 02 10:51:42.560254 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] Attempting differential restore. {{(pid=123698) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1298}} May 02 10:51:42.560776 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] Attempting incremental restore from base='volume-d12d006a-e3e5-424d-b4d4-5836d1700b51.backup.5d54b583-3ca9-4e90-aa1f-0b0ffb591de2' snap='backup.5d54b583-3ca9-4e90-aa1f-0b0ffb591de2.snap.1651488691.0461326' {{(pid=123698) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1145}} May 02 10:51:42.561174 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] Performing differential transfer from 'volume-d12d006a-e3e5-424d-b4d4-5836d1700b51.backup.5d54b583-3ca9-4e90-aa1f-0b0ffb591de2' to 'volume-ee79e3e6-c565-4dfd-8225-fa966d6fde80' {{(pid=123698) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} May 02 10:51:42.561580 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-d12d006a-e3e5-424d-b4d4-5836d1700b51.backup.5d54b583-3ca9-4e90-aa1f-0b0ffb591de2@backup.5d54b583-3ca9-4e90-aa1f-0b0ffb591de2.snap.1651488691.0461326 -' into... {{(pid=123698) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} May 02 10:51:42.561946 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_59hz70aj --pool volumes - volumes/volume-ee79e3e6-c565-4dfd-8225-fa966d6fde80' {{(pid=123698) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} May 02 10:51:45.940289 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] Restore transfer completed in 3.3785s {{(pid=123698) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1167}} May 02 10:51:45.943819 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=123698) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1354}} May 02 10:51:46.009837 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] Restoring volume base metadata {{(pid=123698) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} May 02 10:51:46.115852 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] No metadata of type 'volume-metadata' to restore {{(pid=123698) put /opt/stack/cinder/cinder/backup/driver.py:345}} May 02 10:51:46.117963 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] No metadata of type 'volume-glance-metadata' to restore {{(pid=123698) put /opt/stack/cinder/cinder/backup/driver.py:345}} May 02 10:51:46.120492 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] Restore to volume ee79e3e6-c565-4dfd-8225-fa966d6fde80 finished successfully. {{(pid=123698) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1361}} May 02 10:51:46.122992 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-ee79e3e6-c565-4dfd-8225-fa966d6fde80', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'ee79e3e6-c565-4dfd-8225-fa966d6fde80', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 10:51:46.291934 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] <== disconnect_volume: return (167ms) None {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 10:51:46.346915 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.manager [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] Encryption key ID for volume ee79e3e6-c565-4dfd-8225-fa966d6fde80 already matches encryption key ID in backup 5d54b583-3ca9-4e90-aa1f-0b0ffb591de2. {{(pid=123698) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} May 02 10:51:46.698377 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-a5489022-c3f6-466b-8354-879ec52caf25 tempest-VolumesBackupsAdminTest-292168352 None] Finished restoring backup 5d54b583-3ca9-4e90-aa1f-0b0ffb591de2 to volume ee79e3e6-c565-4dfd-8225-fa966d6fde80. May 02 10:51:47.775891 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-78528137-3bdf-46af-a22b-eb3bc4b5c99f tempest-VolumesBackupsAdminTest-1865443922 None] Delete backup started, backup: 564e5b8b-b3ea-4ae9-a06b-5b80e0079c8c. May 02 10:51:47.778689 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-78528137-3bdf-46af-a22b-eb3bc4b5c99f tempest-VolumesBackupsAdminTest-1865443922 None] Delete started for backup=564e5b8b-b3ea-4ae9-a06b-5b80e0079c8c {{(pid=123698) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} May 02 10:51:47.779108 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-78528137-3bdf-46af-a22b-eb3bc4b5c99f tempest-VolumesBackupsAdminTest-1865443922 None] Trying diff format basename='volume-d12d006a-e3e5-424d-b4d4-5836d1700b51.backup.5d54b583-3ca9-4e90-aa1f-0b0ffb591de2' for backup base image of volume d12d006a-e3e5-424d-b4d4-5836d1700b51. {{(pid=123698) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} May 02 10:51:48.008511 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-78528137-3bdf-46af-a22b-eb3bc4b5c99f tempest-VolumesBackupsAdminTest-1865443922 None] Looking for snapshot of backup base 'volume-d12d006a-e3e5-424d-b4d4-5836d1700b51.backup.5d54b583-3ca9-4e90-aa1f-0b0ffb591de2' {{(pid=123698) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 02 10:51:48.009251 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-78528137-3bdf-46af-a22b-eb3bc4b5c99f tempest-VolumesBackupsAdminTest-1865443922 None] Backup '564e5b8b-b3ea-4ae9-a06b-5b80e0079c8c' has no snapshot {{(pid=123698) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:962}} May 02 10:51:48.010228 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-78528137-3bdf-46af-a22b-eb3bc4b5c99f tempest-VolumesBackupsAdminTest-1865443922 None] No backup snapshot to delete {{(pid=123698) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:500}} May 02 10:51:48.224817 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.drivers.ceph [None req-78528137-3bdf-46af-a22b-eb3bc4b5c99f tempest-VolumesBackupsAdminTest-1865443922 None] Backup base image of volume d12d006a-e3e5-424d-b4d4-5836d1700b51 still has 1 snapshots so skipping base image delete. May 02 10:51:48.945337 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-78528137-3bdf-46af-a22b-eb3bc4b5c99f tempest-VolumesBackupsAdminTest-1865443922 None] Metadata backup object 'backup.564e5b8b-b3ea-4ae9-a06b-5b80e0079c8c.meta' not found - ignoring {{(pid=123698) remove_if_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:162}} May 02 10:51:48.948999 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-78528137-3bdf-46af-a22b-eb3bc4b5c99f tempest-VolumesBackupsAdminTest-1865443922 None] Delete of backup '564e5b8b-b3ea-4ae9-a06b-5b80e0079c8c' for volume 'd12d006a-e3e5-424d-b4d4-5836d1700b51' finished. {{(pid=123698) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} May 02 10:51:48.955219 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_db.sqlalchemy.engines [None req-78528137-3bdf-46af-a22b-eb3bc4b5c99f tempest-VolumesBackupsAdminTest-1865443922 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123698) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 02 10:51:49.053094 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.quota [None req-78528137-3bdf-46af-a22b-eb3bc4b5c99f tempest-VolumesBackupsAdminTest-1865443922 None] Created reservations ['5471191c-cb8c-489f-a165-165195f6d803', 'eeed4382-22d2-48ff-b253-a22d7e8f02a5'] {{(pid=123698) reserve /opt/stack/cinder/cinder/quota.py:755}} May 02 10:51:49.218665 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-78528137-3bdf-46af-a22b-eb3bc4b5c99f tempest-VolumesBackupsAdminTest-1865443922 None] Delete backup finished, backup 564e5b8b-b3ea-4ae9-a06b-5b80e0079c8c deleted. May 02 10:51:50.003226 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-fab97163-791b-4f3d-b772-41d85f88ffd3 tempest-VolumesBackupsAdminTest-292168352 None] Delete backup started, backup: 5d54b583-3ca9-4e90-aa1f-0b0ffb591de2. May 02 10:51:50.004998 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-fab97163-791b-4f3d-b772-41d85f88ffd3 tempest-VolumesBackupsAdminTest-292168352 None] Delete started for backup=5d54b583-3ca9-4e90-aa1f-0b0ffb591de2 {{(pid=123698) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} May 02 10:51:50.005503 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-fab97163-791b-4f3d-b772-41d85f88ffd3 tempest-VolumesBackupsAdminTest-292168352 None] Trying diff format basename='volume-d12d006a-e3e5-424d-b4d4-5836d1700b51.backup.5d54b583-3ca9-4e90-aa1f-0b0ffb591de2' for backup base image of volume d12d006a-e3e5-424d-b4d4-5836d1700b51. {{(pid=123698) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} May 02 10:51:50.431575 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-fab97163-791b-4f3d-b772-41d85f88ffd3 tempest-VolumesBackupsAdminTest-292168352 None] Looking for snapshot of backup base 'volume-d12d006a-e3e5-424d-b4d4-5836d1700b51.backup.5d54b583-3ca9-4e90-aa1f-0b0ffb591de2' {{(pid=123698) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 02 10:51:50.432238 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-fab97163-791b-4f3d-b772-41d85f88ffd3 tempest-VolumesBackupsAdminTest-292168352 None] Found snapshot 'backup.5d54b583-3ca9-4e90-aa1f-0b0ffb591de2.snap.1651488691.0461326' {{(pid=123698) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} May 02 10:51:50.432642 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-fab97163-791b-4f3d-b772-41d85f88ffd3 tempest-VolumesBackupsAdminTest-292168352 None] Deleting backup snapshot='backup.5d54b583-3ca9-4e90-aa1f-0b0ffb591de2.snap.1651488691.0461326' {{(pid=123698) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}} May 02 10:51:50.529657 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_db.sqlalchemy.engines [None req-e2e69ca3-23ac-4614-a31d-4f46fe2b66d3 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123698) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 02 10:51:53.651585 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.drivers.ceph [None req-fab97163-791b-4f3d-b772-41d85f88ffd3 tempest-VolumesBackupsAdminTest-292168352 None] Deleting backup base image='volume-d12d006a-e3e5-424d-b4d4-5836d1700b51.backup.5d54b583-3ca9-4e90-aa1f-0b0ffb591de2' of volume d12d006a-e3e5-424d-b4d4-5836d1700b51. May 02 10:51:59.230752 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-fab97163-791b-4f3d-b772-41d85f88ffd3 tempest-VolumesBackupsAdminTest-292168352 None] Base backup image='volume-d12d006a-e3e5-424d-b4d4-5836d1700b51.backup.5d54b583-3ca9-4e90-aa1f-0b0ffb591de2' of volume d12d006a-e3e5-424d-b4d4-5836d1700b51 deleted. {{(pid=123698) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} May 02 10:51:59.398485 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-fab97163-791b-4f3d-b772-41d85f88ffd3 tempest-VolumesBackupsAdminTest-292168352 None] Delete of backup '5d54b583-3ca9-4e90-aa1f-0b0ffb591de2' for volume 'd12d006a-e3e5-424d-b4d4-5836d1700b51' finished. {{(pid=123698) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} May 02 10:51:59.475261 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.quota [None req-fab97163-791b-4f3d-b772-41d85f88ffd3 tempest-VolumesBackupsAdminTest-292168352 None] Created reservations ['b3367732-8fd7-4a87-a190-b3de1fe32e7b', 'dc52057f-8fd7-4e6f-bf41-4c4d8638a77e'] {{(pid=123698) reserve /opt/stack/cinder/cinder/quota.py:755}} May 02 10:51:59.629861 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-fab97163-791b-4f3d-b772-41d85f88ffd3 tempest-VolumesBackupsAdminTest-292168352 None] Delete backup finished, backup 5d54b583-3ca9-4e90-aa1f-0b0ffb591de2 deleted. May 02 10:52:03.424130 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] Create backup started, backup: 810bb01c-a183-4044-a1ec-52099d05b1c1 volume: 28f5a82a-c70e-428b-b9af-7197fc3d359e. May 02 10:52:03.492835 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-02T10:52:03Z,data_timestamp=2022-05-02T10:52:03Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsAdminTest-Backup-1031484434',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=810bb01c-a183-4044-a1ec-52099d05b1c1,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='ed2eac51a96c4377a9837675807d0bbd',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='456241eca666492ca5b25dfd2fb53769',volume_id=28f5a82a-c70e-428b-b9af-7197fc3d359e) May 02 10:52:03.532315 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.utils [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.16', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 10:52:03.533860 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:52:03.541815 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.008s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:52:03.542138 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[2fc78da3-67fd-4ef6-a479-46a2c515a281]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:9d66b0da6823\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:52:03.543937 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:52:03.550085 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 02 10:52:03.559040 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[d41045bb-2b43-4204-aa96-83b1ea60fe1e]: [Errno 2] No such file or directory: 'systool' {{(pid=154446) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 02 10:52:03.559040 ncdv-master-1 cinder-backup[123698]: Traceback (most recent call last): May 02 10:52:03.559040 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 02 10:52:03.559040 ncdv-master-1 cinder-backup[123698]: ret = func(*f_args, **f_kwargs) May 02 10:52:03.559040 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 02 10:52:03.559040 ncdv-master-1 cinder-backup[123698]: return func(*args, **kwargs) May 02 10:52:03.559040 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 02 10:52:03.559040 ncdv-master-1 cinder-backup[123698]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 02 10:52:03.559040 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 02 10:52:03.559040 ncdv-master-1 cinder-backup[123698]: return putils.execute(on_execute=on_execute, May 02 10:52:03.559040 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 02 10:52:03.559040 ncdv-master-1 cinder-backup[123698]: obj = subprocess.Popen(cmd, May 02 10:52:03.559040 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 02 10:52:03.559040 ncdv-master-1 cinder-backup[123698]: self._execute_child(args, executable, preexec_fn, close_fds, May 02 10:52:03.559040 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 02 10:52:03.559040 ncdv-master-1 cinder-backup[123698]: raise child_exception_type(errno_num, err_msg, err_filename) May 02 10:52:03.559040 ncdv-master-1 cinder-backup[123698]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 02 10:52:03.559040 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[d41045bb-2b43-4204-aa96-83b1ea60fe1e]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:52:03.559040 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:52:03.560293 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 02 10:52:03.560293 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[ebc8ec97-2666-4b18-be7e-adc1797b9b64]: [Errno 2] No such file or directory: 'systool' {{(pid=154446) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 02 10:52:03.560293 ncdv-master-1 cinder-backup[123698]: Traceback (most recent call last): May 02 10:52:03.560293 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 02 10:52:03.560293 ncdv-master-1 cinder-backup[123698]: ret = func(*f_args, **f_kwargs) May 02 10:52:03.560293 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 02 10:52:03.560293 ncdv-master-1 cinder-backup[123698]: return func(*args, **kwargs) May 02 10:52:03.560293 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 02 10:52:03.560293 ncdv-master-1 cinder-backup[123698]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 02 10:52:03.560293 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 02 10:52:03.560293 ncdv-master-1 cinder-backup[123698]: return putils.execute(on_execute=on_execute, May 02 10:52:03.560293 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 02 10:52:03.560293 ncdv-master-1 cinder-backup[123698]: obj = subprocess.Popen(cmd, May 02 10:52:03.560293 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 02 10:52:03.560293 ncdv-master-1 cinder-backup[123698]: self._execute_child(args, executable, preexec_fn, close_fds, May 02 10:52:03.560293 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 02 10:52:03.560293 ncdv-master-1 cinder-backup[123698]: raise child_exception_type(errno_num, err_msg, err_filename) May 02 10:52:03.560293 ncdv-master-1 cinder-backup[123698]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 02 10:52:03.560293 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[ebc8ec97-2666-4b18-be7e-adc1797b9b64]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:52:03.561270 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:52:03.569128 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.008s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:52:03.569444 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[cba8b302-23ff-4c44-9655-32f194e5f6e9]: (4, ('/dev/sda1\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:52:03.570713 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:52:03.579751 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.009s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:52:03.580073 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[790eaeb4-cba1-4848-bbae-cbbfd73886c2]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:52:03.581278 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:52:03.588448 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.007s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:52:03.588742 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[23a595ea-9282-4ac6-8b1f-3971d0a1d4e3]: (4, ('9ace392b-cf5e-9c45-a31c-76b2b08d55ae\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 10:52:03.589506 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] Running cmd (subprocess): nvme version {{(pid=123698) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 10:52:03.611072 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] CMD "nvme version" returned: 0 in 0.021s {{(pid=123698) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 10:52:03.612840 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=123698) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 02 10:52:03.613213 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=123698) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 02 10:52:03.613584 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:7c8447db-24f2-4326-a844-150f6bd61ca4 dsc: {{(pid=123698) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 02 10:52:03.614210 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.utils [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] <== get_connector_properties: return (81ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.16', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:9d66b0da6823', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '9ace392b-cf5e-9c45-a31c-76b2b08d55ae', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:7c8447db-24f2-4326-a844-150f6bd61ca4', 'found_dsc': ''} {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 10:52:04.264889 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connector [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] Factory for rbd on None {{(pid=123698) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 02 10:52:04.266512 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-28f5a82a-c70e-428b-b9af-7197fc3d359e', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '28f5a82a-c70e-428b-b9af-7197fc3d359e', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 10:52:04.267570 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.linuxrbd [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] opening connection to ceph cluster (timeout=-1). {{(pid=123698) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 02 10:52:04.543976 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] <== connect_volume: return (276ms) {'path': } {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 10:52:04.552608 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_db.sqlalchemy.engines [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123698) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 02 10:52:04.612858 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] Starting backup of volume='28f5a82a-c70e-428b-b9af-7197fc3d359e'. {{(pid=123698) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} May 02 10:52:04.613925 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] Volume file is RBD: attempting optimized backup {{(pid=123698) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}} May 02 10:52:04.633506 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] Creating base image 'volume-28f5a82a-c70e-428b-b9af-7197fc3d359e.backup.810bb01c-a183-4044-a1ec-52099d05b1c1' {{(pid=123698) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}} May 02 10:52:04.633865 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=123698) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} May 02 10:52:05.472839 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] Using --from-snap 'None' for incremental backup of volume 28f5a82a-c70e-428b-b9af-7197fc3d359e. {{(pid=123698) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}} May 02 10:52:05.473231 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] Creating backup snapshot='backup.810bb01c-a183-4044-a1ec-52099d05b1c1.snap.1651488725.4728518' {{(pid=123698) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}} May 02 10:52:07.551689 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] Performing differential transfer from 'volume-28f5a82a-c70e-428b-b9af-7197fc3d359e' to 'volume-28f5a82a-c70e-428b-b9af-7197fc3d359e.backup.810bb01c-a183-4044-a1ec-52099d05b1c1' {{(pid=123698) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} May 02 10:52:07.552318 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_d3wnb10b --pool volumes volumes/volume-28f5a82a-c70e-428b-b9af-7197fc3d359e@backup.810bb01c-a183-4044-a1ec-52099d05b1c1.snap.1651488725.4728518 -' into... {{(pid=123698) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} May 02 10:52:07.552454 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-28f5a82a-c70e-428b-b9af-7197fc3d359e.backup.810bb01c-a183-4044-a1ec-52099d05b1c1' {{(pid=123698) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} May 02 10:52:10.723778 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] Differential backup transfer completed in 3.1715s {{(pid=123698) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}} May 02 10:52:10.724311 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] Getting metadata type 'volume-base-metadata' {{(pid=123698) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 02 10:52:10.736464 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=123698) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 02 10:52:10.736916 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] Getting metadata type 'volume-metadata' {{(pid=123698) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 02 10:52:10.743156 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] No metadata type 'volume-metadata' available {{(pid=123698) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 02 10:52:10.743537 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] Getting metadata type 'volume-glance-metadata' {{(pid=123698) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 02 10:52:10.749626 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] No metadata type 'volume-glance-metadata' available {{(pid=123698) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} May 02 10:52:10.750167 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] Backing up metadata for volume 28f5a82a-c70e-428b-b9af-7197fc3d359e. {{(pid=123698) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} May 02 10:52:10.912729 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] Backup '810bb01c-a183-4044-a1ec-52099d05b1c1' of volume 28f5a82a-c70e-428b-b9af-7197fc3d359e finished. {{(pid=123698) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} May 02 10:52:10.914482 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-28f5a82a-c70e-428b-b9af-7197fc3d359e', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '28f5a82a-c70e-428b-b9af-7197fc3d359e', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 10:52:11.599707 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] <== disconnect_volume: return (684ms) None {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 10:52:11.795909 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-e9381b61-0c84-44b3-b9ad-6b31494a5797 tempest-VolumesBackupsAdminTest-292168352 None] Create backup finished. backup: 810bb01c-a183-4044-a1ec-52099d05b1c1. May 02 10:52:12.761691 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-26f09eba-f869-4375-be76-e6d263c1ac96 tempest-VolumesBackupsAdminTest-1865443922 None] Reset backup status started, backup_id: 810bb01c-a183-4044-a1ec-52099d05b1c1, status: error. May 02 10:52:12.762064 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-26f09eba-f869-4375-be76-e6d263c1ac96 tempest-VolumesBackupsAdminTest-1865443922 None] Backup service: cinder.backup.drivers.ceph.CephBackupDriver. May 02 10:52:13.949707 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-f054c2f5-9851-40a1-9fad-56b2415d2b1c tempest-VolumesBackupsAdminTest-292168352 None] Delete backup started, backup: 810bb01c-a183-4044-a1ec-52099d05b1c1. May 02 10:52:13.951284 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-f054c2f5-9851-40a1-9fad-56b2415d2b1c tempest-VolumesBackupsAdminTest-292168352 None] Delete started for backup=810bb01c-a183-4044-a1ec-52099d05b1c1 {{(pid=123698) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} May 02 10:52:13.951707 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-f054c2f5-9851-40a1-9fad-56b2415d2b1c tempest-VolumesBackupsAdminTest-292168352 None] Trying diff format basename='volume-28f5a82a-c70e-428b-b9af-7197fc3d359e.backup.810bb01c-a183-4044-a1ec-52099d05b1c1' for backup base image of volume 28f5a82a-c70e-428b-b9af-7197fc3d359e. {{(pid=123698) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} May 02 10:52:14.168037 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-f054c2f5-9851-40a1-9fad-56b2415d2b1c tempest-VolumesBackupsAdminTest-292168352 None] Looking for snapshot of backup base 'volume-28f5a82a-c70e-428b-b9af-7197fc3d359e.backup.810bb01c-a183-4044-a1ec-52099d05b1c1' {{(pid=123698) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 02 10:52:14.168528 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-f054c2f5-9851-40a1-9fad-56b2415d2b1c tempest-VolumesBackupsAdminTest-292168352 None] Found snapshot 'backup.810bb01c-a183-4044-a1ec-52099d05b1c1.snap.1651488725.4728518' {{(pid=123698) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} May 02 10:52:14.169242 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-f054c2f5-9851-40a1-9fad-56b2415d2b1c tempest-VolumesBackupsAdminTest-292168352 None] Deleting backup snapshot='backup.810bb01c-a183-4044-a1ec-52099d05b1c1.snap.1651488725.4728518' {{(pid=123698) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}} May 02 10:52:16.710717 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.drivers.ceph [None req-f054c2f5-9851-40a1-9fad-56b2415d2b1c tempest-VolumesBackupsAdminTest-292168352 None] Deleting backup base image='volume-28f5a82a-c70e-428b-b9af-7197fc3d359e.backup.810bb01c-a183-4044-a1ec-52099d05b1c1' of volume 28f5a82a-c70e-428b-b9af-7197fc3d359e. May 02 10:52:20.526959 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_db.sqlalchemy.engines [None req-e2e69ca3-23ac-4614-a31d-4f46fe2b66d3 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123698) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 02 10:52:21.733307 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-f054c2f5-9851-40a1-9fad-56b2415d2b1c tempest-VolumesBackupsAdminTest-292168352 None] Base backup image='volume-28f5a82a-c70e-428b-b9af-7197fc3d359e.backup.810bb01c-a183-4044-a1ec-52099d05b1c1' of volume 28f5a82a-c70e-428b-b9af-7197fc3d359e deleted. {{(pid=123698) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} May 02 10:52:21.942342 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-f054c2f5-9851-40a1-9fad-56b2415d2b1c tempest-VolumesBackupsAdminTest-292168352 None] Delete of backup '810bb01c-a183-4044-a1ec-52099d05b1c1' for volume '28f5a82a-c70e-428b-b9af-7197fc3d359e' finished. {{(pid=123698) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} May 02 10:52:22.029558 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.quota [None req-f054c2f5-9851-40a1-9fad-56b2415d2b1c tempest-VolumesBackupsAdminTest-292168352 None] Created reservations ['b3d4b9ce-352f-42b0-8731-96576750c4d4', 'a95fbef7-c26b-42fb-93c1-902092bdfd58'] {{(pid=123698) reserve /opt/stack/cinder/cinder/quota.py:755}} May 02 10:52:22.223843 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-f054c2f5-9851-40a1-9fad-56b2415d2b1c tempest-VolumesBackupsAdminTest-292168352 None] Delete backup finished, backup 810bb01c-a183-4044-a1ec-52099d05b1c1 deleted. May 02 10:52:32.199699 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:52:32.200375 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:53:32.205351 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:53:32.206027 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:54:32.205114 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:54:32.212198 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:55:32.209325 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:55:32.210000 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:56:32.206581 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:56:32.207200 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:57:32.211707 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:57:32.212337 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:58:32.216770 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:58:32.217459 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 10:59:32.221707 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 10:59:32.222775 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:00:32.218543 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:00:32.219130 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:01:32.223229 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:01:32.223828 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:02:32.228274 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:02:32.228936 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:03:32.233604 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:03:32.234316 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:05:32.239741 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:05:32.240403 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:05:37.642503 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] Create backup started, backup: 26c2f375-6404-426f-bea1-0de87bbfde44 volume: ac7e4c82-340c-4a2e-be64-71a3e48b66c8. May 02 11:05:37.740150 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-02T11:05:37Z,data_timestamp=2022-05-02T11:05:37Z,deleted=False,deleted_at=None,display_description=None,display_name=None,encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=26c2f375-6404-426f-bea1-0de87bbfde44,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='dccab55a052648fbb6bbc3b93eaf849b',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='ea624fdbcf21473e8572542184a717df',volume_id=ac7e4c82-340c-4a2e-be64-71a3e48b66c8) May 02 11:05:37.778308 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.utils [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.16', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 11:05:37.779834 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 11:05:37.788212 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.008s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 11:05:37.790242 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[851a286f-1064-4b32-8751-e8be4ce2eca2]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:9d66b0da6823\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 11:05:37.790432 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 11:05:37.796533 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 02 11:05:37.797707 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[36c1bfeb-cb20-4247-befd-7a0cb03136b2]: [Errno 2] No such file or directory: 'systool' {{(pid=154446) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 02 11:05:37.797707 ncdv-master-1 cinder-backup[123698]: Traceback (most recent call last): May 02 11:05:37.797707 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 02 11:05:37.797707 ncdv-master-1 cinder-backup[123698]: ret = func(*f_args, **f_kwargs) May 02 11:05:37.797707 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 02 11:05:37.797707 ncdv-master-1 cinder-backup[123698]: return func(*args, **kwargs) May 02 11:05:37.797707 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 02 11:05:37.797707 ncdv-master-1 cinder-backup[123698]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 02 11:05:37.797707 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 02 11:05:37.797707 ncdv-master-1 cinder-backup[123698]: return putils.execute(on_execute=on_execute, May 02 11:05:37.797707 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 02 11:05:37.797707 ncdv-master-1 cinder-backup[123698]: obj = subprocess.Popen(cmd, May 02 11:05:37.797707 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 02 11:05:37.797707 ncdv-master-1 cinder-backup[123698]: self._execute_child(args, executable, preexec_fn, close_fds, May 02 11:05:37.797707 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 02 11:05:37.797707 ncdv-master-1 cinder-backup[123698]: raise child_exception_type(errno_num, err_msg, err_filename) May 02 11:05:37.797707 ncdv-master-1 cinder-backup[123698]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 02 11:05:37.799054 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[36c1bfeb-cb20-4247-befd-7a0cb03136b2]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 11:05:37.799858 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 11:05:37.805805 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 02 11:05:37.806613 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[94570bd6-e0cc-4be4-837d-b4cab606042a]: [Errno 2] No such file or directory: 'systool' {{(pid=154446) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 02 11:05:37.806613 ncdv-master-1 cinder-backup[123698]: Traceback (most recent call last): May 02 11:05:37.806613 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 02 11:05:37.806613 ncdv-master-1 cinder-backup[123698]: ret = func(*f_args, **f_kwargs) May 02 11:05:37.806613 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 02 11:05:37.806613 ncdv-master-1 cinder-backup[123698]: return func(*args, **kwargs) May 02 11:05:37.806613 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 02 11:05:37.806613 ncdv-master-1 cinder-backup[123698]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 02 11:05:37.806613 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 02 11:05:37.806613 ncdv-master-1 cinder-backup[123698]: return putils.execute(on_execute=on_execute, May 02 11:05:37.806613 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 02 11:05:37.806613 ncdv-master-1 cinder-backup[123698]: obj = subprocess.Popen(cmd, May 02 11:05:37.806613 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 02 11:05:37.806613 ncdv-master-1 cinder-backup[123698]: self._execute_child(args, executable, preexec_fn, close_fds, May 02 11:05:37.806613 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 02 11:05:37.806613 ncdv-master-1 cinder-backup[123698]: raise child_exception_type(errno_num, err_msg, err_filename) May 02 11:05:37.806613 ncdv-master-1 cinder-backup[123698]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 02 11:05:37.807807 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[94570bd6-e0cc-4be4-837d-b4cab606042a]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 11:05:37.808224 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 11:05:37.817415 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.009s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 11:05:37.817748 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[07151351-fd83-4463-a00b-7e04d7d45510]: (4, ('/dev/sda1\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 11:05:37.818959 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 11:05:37.828421 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.009s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 11:05:37.828789 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[9c01dc5f-3a21-474d-b40b-4e6e003bef14]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 11:05:37.829948 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 11:05:37.836923 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.007s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 11:05:37.837248 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[883c491a-3374-4274-be2d-6028804f640f]: (4, ('9ace392b-cf5e-9c45-a31c-76b2b08d55ae\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 11:05:37.838025 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] Running cmd (subprocess): nvme version {{(pid=123698) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 11:05:37.859783 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] CMD "nvme version" returned: 0 in 0.021s {{(pid=123698) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 11:05:37.861690 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=123698) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 02 11:05:37.862061 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=123698) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 02 11:05:37.862430 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:7c8447db-24f2-4326-a844-150f6bd61ca4 dsc: {{(pid=123698) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 02 11:05:37.863058 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.utils [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] <== get_connector_properties: return (83ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.16', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:9d66b0da6823', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '9ace392b-cf5e-9c45-a31c-76b2b08d55ae', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:7c8447db-24f2-4326-a844-150f6bd61ca4', 'found_dsc': ''} {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 11:05:38.607576 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connector [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] Factory for rbd on None {{(pid=123698) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 02 11:05:38.609200 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-ac7e4c82-340c-4a2e-be64-71a3e48b66c8', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'ac7e4c82-340c-4a2e-be64-71a3e48b66c8', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 11:05:38.610258 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.linuxrbd [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] opening connection to ceph cluster (timeout=-1). {{(pid=123698) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 02 11:05:38.745283 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] <== connect_volume: return (135ms) {'path': } {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 11:05:38.752539 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_db.sqlalchemy.engines [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123698) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 02 11:05:38.819545 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] Starting backup of volume='ac7e4c82-340c-4a2e-be64-71a3e48b66c8'. {{(pid=123698) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} May 02 11:05:38.820475 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] Volume file is RBD: attempting optimized backup {{(pid=123698) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}} May 02 11:05:38.841811 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] Creating base image 'volume-ac7e4c82-340c-4a2e-be64-71a3e48b66c8.backup.26c2f375-6404-426f-bea1-0de87bbfde44' {{(pid=123698) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}} May 02 11:05:38.842185 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=123698) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} May 02 11:05:39.345248 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] Using --from-snap 'None' for incremental backup of volume ac7e4c82-340c-4a2e-be64-71a3e48b66c8. {{(pid=123698) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}} May 02 11:05:39.345656 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] Creating backup snapshot='backup.26c2f375-6404-426f-bea1-0de87bbfde44.snap.1651489539.3452759' {{(pid=123698) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}} May 02 11:05:41.061741 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] Performing differential transfer from 'volume-ac7e4c82-340c-4a2e-be64-71a3e48b66c8' to 'volume-ac7e4c82-340c-4a2e-be64-71a3e48b66c8.backup.26c2f375-6404-426f-bea1-0de87bbfde44' {{(pid=123698) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} May 02 11:05:41.062919 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_qbpvnusu --pool volumes volumes/volume-ac7e4c82-340c-4a2e-be64-71a3e48b66c8@backup.26c2f375-6404-426f-bea1-0de87bbfde44.snap.1651489539.3452759 -' into... {{(pid=123698) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} May 02 11:05:41.063311 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-ac7e4c82-340c-4a2e-be64-71a3e48b66c8.backup.26c2f375-6404-426f-bea1-0de87bbfde44' {{(pid=123698) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} May 02 11:06:20.859948 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] Differential backup transfer completed in 39.7980s {{(pid=123698) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}} May 02 11:06:20.860666 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] Getting metadata type 'volume-base-metadata' {{(pid=123698) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 02 11:06:20.872068 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=123698) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 02 11:06:20.872501 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] Getting metadata type 'volume-metadata' {{(pid=123698) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 02 11:06:20.878630 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] No metadata type 'volume-metadata' available {{(pid=123698) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 02 11:06:20.879011 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] Getting metadata type 'volume-glance-metadata' {{(pid=123698) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 02 11:06:20.886578 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=123698) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}} May 02 11:06:20.887064 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] Backing up metadata for volume ac7e4c82-340c-4a2e-be64-71a3e48b66c8. {{(pid=123698) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} May 02 11:06:21.044562 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] Backup '26c2f375-6404-426f-bea1-0de87bbfde44' of volume ac7e4c82-340c-4a2e-be64-71a3e48b66c8 finished. {{(pid=123698) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} May 02 11:06:21.046339 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-ac7e4c82-340c-4a2e-be64-71a3e48b66c8', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'ac7e4c82-340c-4a2e-be64-71a3e48b66c8', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 11:06:21.367232 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] <== disconnect_volume: return (319ms) None {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 11:06:21.596014 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-76318d36-fd5a-4112-9641-52dab4fed878 tempest-TestVolumeBackupRestore-2081214727 None] Create backup finished. backup: 26c2f375-6404-426f-bea1-0de87bbfde44. May 02 11:06:23.421267 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] Restore backup started, backup: 26c2f375-6404-426f-bea1-0de87bbfde44 volume: 6580c21e-356d-4737-8f55-bec015b2d872. May 02 11:06:23.437357 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.utils [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.16', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 11:06:23.437357 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 11:06:23.444319 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.008s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 11:06:23.444693 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[cff53539-28d0-4e6f-9fd6-f0171eb224c5]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:9d66b0da6823\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 11:06:23.446525 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 11:06:23.453604 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 02 11:06:23.454179 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[c6c027fc-bcb1-4f71-916e-06798d9c7db8]: [Errno 2] No such file or directory: 'systool' {{(pid=154446) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 02 11:06:23.454179 ncdv-master-1 cinder-backup[123698]: Traceback (most recent call last): May 02 11:06:23.454179 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 02 11:06:23.454179 ncdv-master-1 cinder-backup[123698]: ret = func(*f_args, **f_kwargs) May 02 11:06:23.454179 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 02 11:06:23.454179 ncdv-master-1 cinder-backup[123698]: return func(*args, **kwargs) May 02 11:06:23.454179 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 02 11:06:23.454179 ncdv-master-1 cinder-backup[123698]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 02 11:06:23.454179 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 02 11:06:23.454179 ncdv-master-1 cinder-backup[123698]: return putils.execute(on_execute=on_execute, May 02 11:06:23.454179 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 02 11:06:23.454179 ncdv-master-1 cinder-backup[123698]: obj = subprocess.Popen(cmd, May 02 11:06:23.454179 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 02 11:06:23.454179 ncdv-master-1 cinder-backup[123698]: self._execute_child(args, executable, preexec_fn, close_fds, May 02 11:06:23.454179 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 02 11:06:23.454179 ncdv-master-1 cinder-backup[123698]: raise child_exception_type(errno_num, err_msg, err_filename) May 02 11:06:23.454179 ncdv-master-1 cinder-backup[123698]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 02 11:06:23.455565 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[c6c027fc-bcb1-4f71-916e-06798d9c7db8]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 11:06:23.458010 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 11:06:23.462112 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 02 11:06:23.462590 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[91dd5522-7417-4d8f-a83b-522ae60105eb]: [Errno 2] No such file or directory: 'systool' {{(pid=154446) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 02 11:06:23.462590 ncdv-master-1 cinder-backup[123698]: Traceback (most recent call last): May 02 11:06:23.462590 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 02 11:06:23.462590 ncdv-master-1 cinder-backup[123698]: ret = func(*f_args, **f_kwargs) May 02 11:06:23.462590 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 02 11:06:23.462590 ncdv-master-1 cinder-backup[123698]: return func(*args, **kwargs) May 02 11:06:23.462590 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 02 11:06:23.462590 ncdv-master-1 cinder-backup[123698]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 02 11:06:23.462590 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 02 11:06:23.462590 ncdv-master-1 cinder-backup[123698]: return putils.execute(on_execute=on_execute, May 02 11:06:23.462590 ncdv-master-1 cinder-backup[123698]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 02 11:06:23.462590 ncdv-master-1 cinder-backup[123698]: obj = subprocess.Popen(cmd, May 02 11:06:23.462590 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 02 11:06:23.462590 ncdv-master-1 cinder-backup[123698]: self._execute_child(args, executable, preexec_fn, close_fds, May 02 11:06:23.462590 ncdv-master-1 cinder-backup[123698]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 02 11:06:23.462590 ncdv-master-1 cinder-backup[123698]: raise child_exception_type(errno_num, err_msg, err_filename) May 02 11:06:23.462590 ncdv-master-1 cinder-backup[123698]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 02 11:06:23.464024 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[91dd5522-7417-4d8f-a83b-522ae60105eb]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 11:06:23.464648 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 11:06:23.472779 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.008s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 11:06:23.473106 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[3ed48f70-cf29-4984-8e21-f259493bf83e]: (4, ('/dev/sda1\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 11:06:23.474283 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 11:06:23.482487 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.008s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 11:06:23.482821 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[ecc6ae81-0ad6-4ac8-a39d-ad1a58ae9f04]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 11:06:23.484010 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 11:06:23.490994 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.006s {{(pid=154446) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 11:06:23.491281 ncdv-master-1 cinder-backup[123698]: DEBUG oslo.privsep.daemon [-] privsep: reply[c75dd6c2-e39f-4ba5-8a3a-133a9aec7f6a]: (4, ('9ace392b-cf5e-9c45-a31c-76b2b08d55ae\n', '')) {{(pid=154446) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 02 11:06:23.492064 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] Running cmd (subprocess): nvme version {{(pid=123698) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 02 11:06:23.512560 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_concurrency.processutils [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] CMD "nvme version" returned: 0 in 0.020s {{(pid=123698) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 02 11:06:23.514401 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=123698) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 02 11:06:23.514771 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=123698) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 02 11:06:23.515232 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.lightos [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:7c8447db-24f2-4326-a844-150f6bd61ca4 dsc: {{(pid=123698) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 02 11:06:23.515905 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.utils [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] <== get_connector_properties: return (80ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.16', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:9d66b0da6823', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '9ace392b-cf5e-9c45-a31c-76b2b08d55ae', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:7c8447db-24f2-4326-a844-150f6bd61ca4', 'found_dsc': ''} {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 11:06:24.265033 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connector [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] Factory for rbd on None {{(pid=123698) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 02 11:06:24.266721 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-6580c21e-356d-4737-8f55-bec015b2d872', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '6580c21e-356d-4737-8f55-bec015b2d872', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 11:06:24.267760 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.linuxrbd [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] opening connection to ceph cluster (timeout=-1). {{(pid=123698) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 02 11:06:24.430085 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] <== connect_volume: return (162ms) {'path': } {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 11:06:24.437945 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_db.sqlalchemy.engines [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123698) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 02 11:06:24.447966 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] Starting restore from Ceph backup=26c2f375-6404-426f-bea1-0de87bbfde44 to volume=volume-6580c21e-356d-4737-8f55-bec015b2d872 {{(pid=123698) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1343}} May 02 11:06:24.510055 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] Looking for snapshot of backup base 'volume-ac7e4c82-340c-4a2e-be64-71a3e48b66c8.backup.26c2f375-6404-426f-bea1-0de87bbfde44' {{(pid=123698) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 02 11:06:24.510572 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] Found snapshot 'backup.26c2f375-6404-426f-bea1-0de87bbfde44.snap.1651489539.3452759' {{(pid=123698) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} May 02 11:06:24.514966 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] Volume file is RBD. {{(pid=123698) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1245}} May 02 11:06:24.544397 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] Attempting differential restore. {{(pid=123698) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1298}} May 02 11:06:24.544908 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] Attempting incremental restore from base='volume-ac7e4c82-340c-4a2e-be64-71a3e48b66c8.backup.26c2f375-6404-426f-bea1-0de87bbfde44' snap='backup.26c2f375-6404-426f-bea1-0de87bbfde44.snap.1651489539.3452759' {{(pid=123698) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1145}} May 02 11:06:24.545307 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] Performing differential transfer from 'volume-ac7e4c82-340c-4a2e-be64-71a3e48b66c8.backup.26c2f375-6404-426f-bea1-0de87bbfde44' to 'volume-6580c21e-356d-4737-8f55-bec015b2d872' {{(pid=123698) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} May 02 11:06:24.545715 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-ac7e4c82-340c-4a2e-be64-71a3e48b66c8.backup.26c2f375-6404-426f-bea1-0de87bbfde44@backup.26c2f375-6404-426f-bea1-0de87bbfde44.snap.1651489539.3452759 -' into... {{(pid=123698) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} May 02 11:06:24.546089 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_z4g1l1zu --pool volumes - volumes/volume-6580c21e-356d-4737-8f55-bec015b2d872' {{(pid=123698) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} May 02 11:06:32.245017 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:06:32.245684 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:07:04.838167 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] Restore transfer completed in 40.2925s {{(pid=123698) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1167}} May 02 11:07:04.839065 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=123698) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1354}} May 02 11:07:04.887479 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] Restoring volume base metadata {{(pid=123698) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} May 02 11:07:04.986399 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] No metadata of type 'volume-metadata' to restore {{(pid=123698) put /opt/stack/cinder/cinder/backup/driver.py:345}} May 02 11:07:04.986800 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.driver [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] Restoring volume glance metadata {{(pid=123698) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}} May 02 11:07:05.918687 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] Restore to volume 6580c21e-356d-4737-8f55-bec015b2d872 finished successfully. {{(pid=123698) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1361}} May 02 11:07:05.920438 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-6580c21e-356d-4737-8f55-bec015b2d872', 'hosts': ['10.222.0.16'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '6580c21e-356d-4737-8f55-bec015b2d872', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 02 11:07:06.051476 ncdv-master-1 cinder-backup[123698]: DEBUG os_brick.initiator.connectors.rbd [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] <== disconnect_volume: return (130ms) None {{(pid=123698) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 02 11:07:06.097087 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.manager [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] Encryption key ID for volume 6580c21e-356d-4737-8f55-bec015b2d872 already matches encryption key ID in backup 26c2f375-6404-426f-bea1-0de87bbfde44. {{(pid=123698) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} May 02 11:07:06.323444 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-0fbb69da-cac2-45ff-a1ed-0369c783f171 tempest-TestVolumeBackupRestore-2081214727 None] Finished restoring backup 26c2f375-6404-426f-bea1-0de87bbfde44 to volume 6580c21e-356d-4737-8f55-bec015b2d872. May 02 11:07:32.240485 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:07:32.241161 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:08:32.245112 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:08:32.245776 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:09:32.250507 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:09:32.251161 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:10:32.255946 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:10:32.256642 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:11:32.260648 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:11:32.261370 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:12:27.119705 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-8ff1b4c2-2aa1-4258-a7d1-1d06d3e7fe72 tempest-TestVolumeBackupRestore-2081214727 None] Delete backup started, backup: 26c2f375-6404-426f-bea1-0de87bbfde44. May 02 11:12:27.119705 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-8ff1b4c2-2aa1-4258-a7d1-1d06d3e7fe72 tempest-TestVolumeBackupRestore-2081214727 None] Delete started for backup=26c2f375-6404-426f-bea1-0de87bbfde44 {{(pid=123698) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} May 02 11:12:27.119705 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-8ff1b4c2-2aa1-4258-a7d1-1d06d3e7fe72 tempest-TestVolumeBackupRestore-2081214727 None] Trying diff format basename='volume-ac7e4c82-340c-4a2e-be64-71a3e48b66c8.backup.26c2f375-6404-426f-bea1-0de87bbfde44' for backup base image of volume ac7e4c82-340c-4a2e-be64-71a3e48b66c8. {{(pid=123698) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} May 02 11:12:27.250355 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-8ff1b4c2-2aa1-4258-a7d1-1d06d3e7fe72 tempest-TestVolumeBackupRestore-2081214727 None] Looking for snapshot of backup base 'volume-ac7e4c82-340c-4a2e-be64-71a3e48b66c8.backup.26c2f375-6404-426f-bea1-0de87bbfde44' {{(pid=123698) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 02 11:12:27.250681 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-8ff1b4c2-2aa1-4258-a7d1-1d06d3e7fe72 tempest-TestVolumeBackupRestore-2081214727 None] Found snapshot 'backup.26c2f375-6404-426f-bea1-0de87bbfde44.snap.1651489539.3452759' {{(pid=123698) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} May 02 11:12:27.251059 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-8ff1b4c2-2aa1-4258-a7d1-1d06d3e7fe72 tempest-TestVolumeBackupRestore-2081214727 None] Deleting backup snapshot='backup.26c2f375-6404-426f-bea1-0de87bbfde44.snap.1651489539.3452759' {{(pid=123698) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}} May 02 11:12:29.752440 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.drivers.ceph [None req-8ff1b4c2-2aa1-4258-a7d1-1d06d3e7fe72 tempest-TestVolumeBackupRestore-2081214727 None] Deleting backup base image='volume-ac7e4c82-340c-4a2e-be64-71a3e48b66c8.backup.26c2f375-6404-426f-bea1-0de87bbfde44' of volume ac7e4c82-340c-4a2e-be64-71a3e48b66c8. May 02 11:12:30.642615 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_db.sqlalchemy.engines [None req-e2e69ca3-23ac-4614-a31d-4f46fe2b66d3 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123698) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 02 11:12:32.255875 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:12:32.256923 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:12:39.336067 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-8ff1b4c2-2aa1-4258-a7d1-1d06d3e7fe72 tempest-TestVolumeBackupRestore-2081214727 None] Base backup image='volume-ac7e4c82-340c-4a2e-be64-71a3e48b66c8.backup.26c2f375-6404-426f-bea1-0de87bbfde44' of volume ac7e4c82-340c-4a2e-be64-71a3e48b66c8 deleted. {{(pid=123698) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} May 02 11:12:39.514790 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.backup.drivers.ceph [None req-8ff1b4c2-2aa1-4258-a7d1-1d06d3e7fe72 tempest-TestVolumeBackupRestore-2081214727 None] Delete of backup '26c2f375-6404-426f-bea1-0de87bbfde44' for volume 'ac7e4c82-340c-4a2e-be64-71a3e48b66c8' finished. {{(pid=123698) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} May 02 11:12:39.602418 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.quota [None req-8ff1b4c2-2aa1-4258-a7d1-1d06d3e7fe72 tempest-TestVolumeBackupRestore-2081214727 None] Created reservations ['23e27f9a-a11f-4282-ae54-d5c981b66e17', 'e1cb5848-dbfa-494b-853d-31ce40065736'] {{(pid=123698) reserve /opt/stack/cinder/cinder/quota.py:755}} May 02 11:12:39.785699 ncdv-master-1 cinder-backup[123698]: INFO cinder.backup.manager [None req-8ff1b4c2-2aa1-4258-a7d1-1d06d3e7fe72 tempest-TestVolumeBackupRestore-2081214727 None] Delete backup finished, backup 26c2f375-6404-426f-bea1-0de87bbfde44 deleted. May 02 11:13:32.261630 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:13:32.262264 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:14:32.266760 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:14:32.267410 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:15:32.271729 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:15:32.272411 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:16:32.267909 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:16:32.268695 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:17:32.264733 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:17:32.265366 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:18:32.269779 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:18:32.270400 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:19:32.265645 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:19:32.266252 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:20:32.269963 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:20:32.270702 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:21:32.274696 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:21:32.275436 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:22:32.278050 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:22:32.278625 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:23:32.283968 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:23:32.284692 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:24:32.280584 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:24:32.281298 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:25:32.285774 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:25:32.286482 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:26:32.290839 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:26:32.291481 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:27:32.295516 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:27:32.296085 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:28:32.300900 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:28:32.301458 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:29:32.305690 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:29:32.306274 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:30:32.310285 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:30:32.310899 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:31:32.315053 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:31:32.315630 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:32:32.320184 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:32:32.320746 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:33:32.325219 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:33:32.325890 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:34:32.330952 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:34:32.331536 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:35:32.336123 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:35:32.336790 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:36:32.340591 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:36:32.341254 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:37:32.337273 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:37:32.337893 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:38:32.342624 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:38:32.343271 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:39:32.347147 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:39:32.347788 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:40:32.351769 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:40:32.352352 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:41:32.357203 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:41:32.357813 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:42:32.353023 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:42:32.353609 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:43:32.357677 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:43:32.358306 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:44:32.353753 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:44:32.354356 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:45:32.359111 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:45:32.359779 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:46:32.355645 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:46:32.356277 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:47:32.360171 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:47:32.360822 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:48:32.365131 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:48:32.365800 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:49:32.370819 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:49:32.371454 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:50:32.375986 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:50:32.376665 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:51:32.380664 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:51:32.381298 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:52:32.376739 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:52:32.377394 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 02 11:53:32.381279 ncdv-master-1 cinder-backup[123698]: DEBUG oslo_service.periodic_task [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123698) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 02 11:53:32.381932 ncdv-master-1 cinder-backup[123698]: DEBUG cinder.manager [None req-908945ff-deb1-466c-8811-f7d31c82b972 None None] Notifying Schedulers of capabilities ... {{(pid=123698) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}