-- Logs begin at Sat 2022-05-28 20:01:13 UTC, end at Sat 2022-05-28 22:29:23 UTC. -- May 28 20:52:53.798238 ncdv-master-1 systemd[1]: Started Devstack devstack@c-bak.service. May 28 20:52:56.988750 ncdv-master-1 cinder-backup[135384]: INFO cinder.cmd.backup [-] Backup running with 1 processes. May 28 20:52:56.990553 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_concurrency.lockutils [-] Acquired lock "singleton_lock" {{(pid=135384) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:294}} May 28 20:52:56.991104 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_concurrency.lockutils [-] Releasing lock "singleton_lock" {{(pid=135384) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:312}} May 28 20:52:57.228190 ncdv-master-1 cinder-backup[135384]: INFO dbcounter [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] Registered counter for database cinder May 28 20:52:57.263784 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_db.sqlalchemy.engines [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=135384) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 28 20:52:57.263784 ncdv-master-1 cinder-backup[135384]: DEBUG dbcounter [-] [135384] Writer thread running {{(pid=135384) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} May 28 20:52:57.524271 ncdv-master-1 cinder-backup[135384]: INFO cinder.rpc [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] Automatically selected cinder-backup objects version 1.39 as minimum service version. May 28 20:52:57.532907 ncdv-master-1 cinder-backup[135384]: INFO cinder.rpc [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] Automatically selected cinder-backup RPC version 2.3 as minimum service version. May 28 20:52:57.538660 ncdv-master-1 cinder-backup[135384]: INFO cinder.rpc [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] Automatically selected cinder-volume objects version 1.39 as minimum service version. May 28 20:52:57.543445 ncdv-master-1 cinder-backup[135384]: INFO cinder.rpc [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] Automatically selected cinder-volume RPC version 3.18 as minimum service version. May 28 20:52:57.549498 ncdv-master-1 cinder-backup[135384]: INFO cinder.rpc [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] Automatically selected cinder-scheduler objects version 1.39 as minimum service version. May 28 20:52:57.556718 ncdv-master-1 cinder-backup[135384]: INFO cinder.rpc [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] Automatically selected cinder-scheduler RPC version 3.12 as minimum service version. May 28 20:52:57.616625 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_db.sqlalchemy.engines [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=135384) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 28 20:52:57.728838 ncdv-master-1 cinder-backup[135384]: INFO oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] Starting 1 workers May 28 20:52:57.734196 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] Started child 135833 {{(pid=135384) _start_child /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:575}} May 28 20:52:57.735564 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] Full set of CONF: {{(pid=135384) wait /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:649}} May 28 20:52:57.737759 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] ******************************************************************************** {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} May 28 20:52:57.737759 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] Configuration options gathered from: {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} May 28 20:52:57.737759 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] command line args: ['--config-file', '/etc/cinder/cinder.conf'] {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} May 28 20:52:57.738284 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] config files: ['/etc/cinder/cinder.conf'] {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} May 28 20:52:57.738571 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] ================================================================================ {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} May 28 20:52:57.739467 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] allow_availability_zone_fallback = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.739933 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] allow_compression_on_image_upload = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.740502 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] allowed_direct_url_schemes = [] {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.740966 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] api_rate_limit = True {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.741624 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] auth_strategy = keystone {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.742106 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_availability_zone = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.742228 ncdv-master-1 cinder-backup[135833]: INFO cinder.service [-] Starting cinder-backup node (version 20.1.0) May 28 20:52:57.742735 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backup_api_class = cinder.backup.api.API {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.743085 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backup_ceph_chunk_size = 134217728 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.743564 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backup_ceph_conf = /etc/ceph/ceph.conf {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.743979 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backup_ceph_image_journals = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.744450 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backup_ceph_pool = backups {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.745552 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backup_ceph_stripe_count = 0 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.746769 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backup_ceph_stripe_unit = 0 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.747240 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backup_ceph_user = cinder-bak {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.747725 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backup_driver = cinder.backup.drivers.ceph.CephBackupDriver {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.748183 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backup_driver_init_check_interval = 60 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.748620 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backup_driver_stats_polling_interval = 60 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.749066 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backup_manager = cinder.backup.manager.BackupManager {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.749668 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backup_max_operations = 15 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.750162 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backup_metadata_version = 2 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.750643 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backup_name_template = backup-%s {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.751056 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backup_native_threads_pool_size = 60 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.751516 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backup_object_number_per_notification = 10 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.752233 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backup_service_inithost_offload = True {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.752663 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backup_timer_interval = 120 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.753118 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backup_use_same_host = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.753565 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backup_use_temp_snapshot = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.754035 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backup_workers = 1 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.754507 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] chap_password = **** {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.757306 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] chap_username = {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.757913 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.758429 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cinder_internal_tenant_project_id = 3186f9d3c2b341afa8b098ebcaaf974b {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.759111 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cinder_internal_tenant_user_id = afc08d15f24e46e29c34d618d955bd28 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.770425 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] compression_format = gzip {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.770425 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] config_dir = [] {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.770425 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] config_file = ['/etc/cinder/cinder.conf'] {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.770425 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] config_source = [] {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.770425 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] consistencygroup_api_class = cinder.consistencygroup.api.API {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.770425 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] control_exchange = openstack {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.771119 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] debug = True {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.771119 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] default_availability_zone = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.771119 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] default_group_type = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.771119 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'swiftclient=WARN', 'botocore=WARN'] {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.771119 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] default_volume_type = ceph {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.771692 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] driver_client_cert = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.771692 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] driver_client_cert_key = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.771692 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] driver_data_namespace = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.771692 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] driver_ssl_cert_path = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.772647 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] driver_ssl_cert_verify = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.773068 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] driver_use_ssl = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.773443 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] enable_new_services = True {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.773887 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] enable_unsupported_driver = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.774308 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] enabled_backends = ['ceph'] {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.774688 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] enforce_multipath_for_image_xfer = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.775218 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] filter_function = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.775594 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] glance_api_insecure = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.775970 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] glance_api_servers = ['http://10.222.0.33/image'] {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.777165 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] glance_api_ssl_compression = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.777534 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] glance_ca_certificates_file = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.778009 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] glance_catalog_info = image:glance:publicURL {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.778359 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] glance_certfile = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.778868 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] glance_core_properties = ['checksum', 'container_format', 'disk_format', 'image_name', 'image_id', 'min_disk', 'min_ram', 'name', 'size'] {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.779237 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] glance_keyfile = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.779611 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] glance_num_retries = 3 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.780091 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] glance_request_timeout = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.785870 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] goodness_function = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.785870 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] graceful_shutdown_timeout = 5 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.785870 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] group_api_class = cinder.group.api.API {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.785870 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] host = ncdv-master-1 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.785870 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] iet_conf = /etc/iet/ietd.conf {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.785870 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] image_compress_on_upload = True {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.786384 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] image_conversion_address_space_limit = 1 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.786384 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] image_conversion_cpu_limit = 60 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.786384 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] image_conversion_dir = /opt/stack/data/cinder/conversion {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.786384 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] image_upload_use_cinder_backend = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.786384 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] image_upload_use_internal_tenant = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.786384 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] image_volume_cache_enabled = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.786774 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] image_volume_cache_max_count = 0 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.786774 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] image_volume_cache_max_size_gb = 0 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.787120 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] instance_format = [instance: %(uuid)s] {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.787630 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] instance_uuid_format = [instance: %(uuid)s] {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.788129 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] iscsi_iotype = fileio {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.794021 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] iscsi_secondary_ip_addresses = [] {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.794021 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] iscsi_target_flags = {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.794021 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] iscsi_write_cache = on {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.794021 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] iser_helper = tgtadm {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.794021 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] iser_ip_address = 10.222.0.33 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.794021 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] iser_port = 3260 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.794021 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.794839 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] log_config_append = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.794839 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.794839 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] log_dir = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.794839 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] log_file = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.794839 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] log_options = True {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.794839 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] log_rotate_interval = 1 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.796426 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] log_rotate_interval_type = days {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.796426 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] log_rotation_type = none {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.796426 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.796728 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.797387 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.798079 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.798622 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.798918 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] max_age = 0 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.800699 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] max_logfile_count = 30 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.800699 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] max_logfile_size_mb = 200 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.801007 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] max_over_subscription_ratio = 20.0 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.801584 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] message_reap_interval = 86400 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.802163 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] message_ttl = 2592000 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.802549 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] monkey_patch = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.803119 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] monkey_patch_modules = [] {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.806659 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] my_ip = 10.222.0.33 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.806659 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] no_snapshot_gb_quota = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.806659 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] num_iser_scan_tries = 3 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.806659 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] num_shell_tries = 3 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.806659 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] num_volume_device_scan_tries = 3 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.806659 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] nvmet_ns_id = 10 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.806659 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] nvmet_port_id = 1 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.807286 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] osapi_max_limit = 1000 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.807286 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] osapi_volume_ext_list = [] {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.808080 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] osapi_volume_extension = ['cinder.api.contrib.standard_extensions'] {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.808567 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] osapi_volume_listen = 0.0.0.0 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.809028 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] osapi_volume_listen_port = 8776 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.809490 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] osapi_volume_use_ssl = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.810091 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] osapi_volume_workers = 2 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.912854 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] per_volume_size_limit = -1 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.912854 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] periodic_fuzzy_delay = 60 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.912854 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] periodic_interval = 60 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.912854 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] project_id_regex = [0-9a-f\-]+ {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.912854 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] public_endpoint = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.912854 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] publish_errors = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.912854 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] quota_backup_gigabytes = 1000 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.913653 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] quota_backups = 10 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.913653 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] quota_consistencygroups = 10 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.913653 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] quota_driver = cinder.quota.DbQuotaDriver {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.913653 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] quota_gigabytes = 1000 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.913653 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] quota_groups = 10 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.913653 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] quota_snapshots = 10 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.913653 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] quota_volumes = 10 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.914167 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] rate_limit_burst = 0 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.914167 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] rate_limit_except_level = CRITICAL {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.914167 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] rate_limit_interval = 0 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.914167 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] replication_device = **** {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.914167 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] report_discard_supported = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.914167 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] report_interval = 10 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.914167 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] reservation_clean_interval = 86400 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.914667 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] reservation_expire = 86400 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.914667 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] reserved_percentage = 0 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.914667 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] resource_query_filters_file = /etc/cinder/resource_filters.json {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.914667 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] restore_discard_excess_bytes = True {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.914667 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] rootwrap_config = /etc/cinder/rootwrap.conf {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.914667 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] rpc_response_timeout = 120 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.915085 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] run_external_periodic_tasks = True {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.915085 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] scheduler_manager = cinder.scheduler.manager.SchedulerManager {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.915085 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] scst_target_driver = iscsi {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.915085 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] scst_target_iqn_name = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.915085 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] service_down_time = 60 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.915085 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] snapshot_name_template = snapshot-%s {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.915502 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] split_loggers = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.915502 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] state_path = /opt/stack/data/cinder {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.915502 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] storage_availability_zone = nova {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.915502 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] storage_protocol = iSCSI {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.915502 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] syslog_log_facility = LOG_USER {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.915502 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] target_helper = lioadm {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.915910 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] target_ip_address = 10.222.0.33 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.915910 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] target_port = 3260 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.915910 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] target_prefix = iqn.2010-10.org.openstack: {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.915910 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] target_protocol = iscsi {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.915910 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] trace_flags = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.915910 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] transfer_api_class = cinder.transfer.api.API {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.916300 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] transport_url = **** {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.916300 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] until_refresh = 0 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.916300 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] use_chap_auth = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.916300 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] use_default_quota_class = True {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.916300 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] use_eventlog = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.916300 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] use_journal = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.916300 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] use_json = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.916753 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] use_multipath_for_image_xfer = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.916753 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] use_stderr = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.916753 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] use_syslog = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.916753 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] verify_glance_signatures = enabled {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.916753 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] volume_api_class = cinder.volume.api.API {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.916753 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] volume_backend_name = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.916753 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] volume_clear = zero {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.917289 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] volume_clear_ionice = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.917289 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] volume_clear_size = 0 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.917289 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.917494 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.lockutils [None req-4e1c8c72-88fc-48af-a286-271ec58fec98 None None] Lock "cleanup_incomplete_backups_135384" acquired by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: waited 0.001s {{(pid=135833) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:386}} May 28 20:52:57.917580 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] volume_copy_bps_limit = 0 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.917652 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-4e1c8c72-88fc-48af-a286-271ec58fec98 None None] Cleaning up incomplete backup operations. May 28 20:52:57.917727 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] volume_dd_blocksize = 1M {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.917727 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] volume_manager = cinder.volume.manager.VolumeManager {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.917727 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] volume_name_template = volume-%s {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.917727 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] volume_usage_audit_period = month {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.917727 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] volumes_dir = /opt/stack/data/cinder/volumes {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.917727 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] watch_log_file = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 28 20:52:57.918144 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.918144 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.918144 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_concurrency.disable_process_locking = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.918144 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_concurrency.lock_path = /opt/stack/data/cinder {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.918144 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.918144 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.918588 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_metrics.metrics_process_name = {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.918588 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.918588 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.918588 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.admin_password = **** {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.918588 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.admin_tenant_name = admin {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.918588 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.admin_token = **** {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.918981 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.admin_user = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.918981 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.auth_admin_prefix = {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.918981 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.918981 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.auth_port = 35357 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.918981 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.auth_protocol = https {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.918981 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.auth_section = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.918981 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.auth_type = password {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.919481 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.auth_uri = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.919481 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.auth_version = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.919481 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.cache = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.919481 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.cafile = /opt/stack/data/ca-bundle.pem {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.919481 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.certfile = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.919481 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.delay_auth_decision = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.919855 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.enforce_token_bind = permissive {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.919855 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.http_connect_timeout = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.919855 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.http_request_max_retries = 3 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.919855 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.identity_uri = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.919855 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.include_service_catalog = True {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.919855 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.insecure = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.920232 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.interface = public {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.920232 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.keyfile = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.920232 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.920232 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.920232 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.920232 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.920609 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.920609 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.memcache_secret_key = **** {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.920609 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.memcache_security_strategy = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.920609 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.920609 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.memcached_servers = ['localhost:11211'] {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.920609 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.region_name = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.920990 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.service_token_roles = ['service'] {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.920990 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.service_token_roles_required = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.920990 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.service_type = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.920990 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.token_cache_time = 300 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.920990 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] keystone_authtoken.www_authenticate_uri = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.920990 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.backend = dogpile.cache.null {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.921376 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.backend_argument = **** {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.921376 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.config_prefix = cache.oslo {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.921376 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.dead_timeout = 60.0 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.921376 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.debug_cache_backend = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.921376 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.enable_retry_client = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.921376 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.enable_socket_keepalive = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.921376 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.enabled = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.921824 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.expiration_time = 600 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.921824 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.hashclient_retry_attempts = 2 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.921824 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.hashclient_retry_delay = 1.0 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.921824 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.memcache_dead_retry = 300 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.921824 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.memcache_pool_connection_get_timeout = 10 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.921824 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.memcache_pool_flush_on_reconnect = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.921824 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.memcache_pool_maxsize = 10 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.922312 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.memcache_pool_unused_timeout = 60 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.922312 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.memcache_servers = ['localhost:11211'] {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.922312 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.memcache_socket_timeout = 1.0 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.922312 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.proxies = [] {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.922312 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.retry_attempts = 2 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.922312 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.retry_delay = 0.0 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.922312 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.socket_keepalive_count = 1 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.922771 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.socket_keepalive_idle = 1 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.922771 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.socket_keepalive_interval = 1 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.922771 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.tls_allowed_ciphers = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.922771 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.tls_cafile = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.922771 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.tls_certfile = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.922771 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.tls_enabled = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.922771 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cache.tls_keyfile = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.923203 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] coordination.backend_url = etcd3+http://10.222.0.33:2379 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.923203 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] database.backend = sqlalchemy {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.923203 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] database.connection = **** {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.923203 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] database.connection_debug = 0 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.923203 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] database.connection_parameters = {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.923203 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] database.connection_recycle_time = 3600 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.923203 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] database.connection_trace = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.923717 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] database.db_inc_retry_interval = True {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.923717 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] database.db_max_retries = 20 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.923717 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] database.db_max_retry_interval = 10 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.923717 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] database.db_retry_interval = 1 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.923717 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] database.max_overflow = 50 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.923717 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] database.max_pool_size = 5 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.923717 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] database.max_retries = 10 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.924231 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] database.mysql_enable_ndb = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.924231 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] database.mysql_sql_mode = TRADITIONAL {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.924231 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] database.pool_timeout = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.924231 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] database.retry_interval = 10 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.924231 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] database.slave_connection = **** {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.924231 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] database.sqlite_synchronous = True {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.924231 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] database.use_db_reconnect = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.924669 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_policy.enforce_new_defaults = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.924669 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_policy.enforce_scope = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.924669 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_policy.policy_default_rule = default {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.924669 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_policy.policy_dirs = ['policy.d'] {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.924669 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_policy.policy_file = policy.yaml {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.924669 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.925132 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.925132 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.925132 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.925132 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.925132 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] profiler.connection_string = messaging:// {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.925132 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] profiler.enabled = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.925491 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] profiler.es_doc_type = notification {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.925491 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] profiler.es_scroll_size = 10000 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.925491 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] profiler.es_scroll_time = 2m {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.925491 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] profiler.filter_error_trace = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.925491 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] profiler.hmac_keys = SECRET_KEY {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.925491 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] profiler.sentinel_service_name = mymaster {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.925491 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] profiler.socket_timeout = 0.1 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.933179 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] profiler.trace_sqlalchemy = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.933179 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] privsep_osbrick.capabilities = [21] {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.933179 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] privsep_osbrick.group = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.933179 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] privsep_osbrick.helper_command = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.933179 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] privsep_osbrick.logger_name = oslo_privsep.daemon {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.933179 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] privsep_osbrick.thread_pool_size = 4 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.933179 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] privsep_osbrick.user = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.934110 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cinder_sys_admin.capabilities = [0, 1, 2, 3, 12, 21] {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.934110 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cinder_sys_admin.group = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.934110 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cinder_sys_admin.helper_command = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.934110 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cinder_sys_admin.logger_name = oslo_privsep.daemon {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.934110 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cinder_sys_admin.thread_pool_size = 4 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.934110 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] cinder_sys_admin.user = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.934110 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] service_user.auth_section = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.934574 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] service_user.auth_type = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.934574 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] service_user.cafile = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.934574 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] service_user.certfile = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.934574 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] service_user.collect_timing = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.934574 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] service_user.insecure = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.934574 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] service_user.keyfile = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.934574 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] service_user.send_service_user_token = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.935077 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] service_user.split_loggers = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.935077 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] service_user.timeout = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.935077 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.backend_availability_zone = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.935077 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.chap_password = **** {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.935077 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.chap_username = {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.935077 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.935477 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.deferred_deletion_delay = 0 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.935477 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.deferred_deletion_purge_interval = 60 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.935477 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.driver_client_cert = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.935477 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.driver_client_cert_key = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.935477 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.driver_data_namespace = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.935477 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.driver_ssl_cert_path = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.935880 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.driver_ssl_cert_verify = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.935880 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.driver_use_ssl = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.935880 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.enable_deferred_deletion = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.935880 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.enable_unsupported_driver = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.935880 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.enforce_multipath_for_image_xfer = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.935880 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.filter_function = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.936291 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.goodness_function = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.936291 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.iet_conf = /etc/iet/ietd.conf {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.936291 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.image_upload_use_cinder_backend = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.936291 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.image_upload_use_internal_tenant = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.936291 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.image_volume_cache_enabled = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.936291 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.image_volume_cache_max_count = 0 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.936643 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.image_volume_cache_max_size_gb = 0 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.936643 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.iscsi_iotype = fileio {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.936643 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.iscsi_secondary_ip_addresses = [] {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.936643 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.iscsi_target_flags = {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.936643 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.iscsi_write_cache = on {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.936643 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.iser_helper = tgtadm {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.937033 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.iser_ip_address = 10.222.0.33 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.937033 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.iser_port = 3260 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.937033 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.937033 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.max_over_subscription_ratio = 20.0 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.937033 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.num_iser_scan_tries = 3 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.937033 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.num_shell_tries = 3 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.937423 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.num_volume_device_scan_tries = 3 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.937423 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.nvmet_ns_id = 10 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.937423 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.nvmet_port_id = 1 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.937423 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.rados_connect_timeout = -1 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.937423 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.rados_connection_interval = 5 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.937423 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.rados_connection_retries = 3 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.937423 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.rbd_ceph_conf = {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.937929 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.rbd_cluster_name = ceph {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.937929 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.rbd_exclusive_cinder_pool = True {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.937929 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.rbd_flatten_volume_from_snapshot = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.937929 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.rbd_max_clone_depth = 5 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.937929 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.rbd_pool = rbd {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.937929 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.rbd_secret_uuid = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.938316 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.rbd_store_chunk_size = 4 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.938316 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.rbd_user = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.938316 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.replication_connect_timeout = 5 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.938316 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.replication_device = **** {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.938316 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.report_discard_supported = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.938316 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.report_dynamic_total_capacity = True {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.938726 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.reserved_percentage = 0 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.938726 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.scst_target_driver = iscsi {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.938726 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.scst_target_iqn_name = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.938726 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.storage_protocol = iSCSI {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.938726 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.target_helper = tgtadm {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.938726 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.target_ip_address = 10.222.0.33 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.939183 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.target_port = 3260 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.939183 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.target_prefix = iqn.2010-10.org.openstack: {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.939183 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.target_protocol = iscsi {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.939183 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.trace_flags = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.939183 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.unique_fqdn_network = True {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.945866 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.use_chap_auth = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.945866 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.use_multipath_for_image_xfer = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.945866 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.volume_backend_name = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.945866 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.volume_clear = zero {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.945866 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.volume_clear_ionice = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.945866 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.volume_clear_size = 0 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.946496 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.946496 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.volume_copy_bps_limit = 0 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.946496 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.volume_dd_blocksize = 1M {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.946496 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] backend_defaults.volumes_dir = /opt/stack/data/cinder/volumes {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.946496 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_reports.file_event_handler = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.946496 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_reports.file_event_handler_interval = 1 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.947194 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_reports.log_dir = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.947900 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.948375 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.948855 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.949312 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.949770 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.950469 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.951169 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.heartbeat_in_pthread = True {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.951644 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.heartbeat_rate = 2 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.952269 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.952725 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.953366 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.953902 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.954530 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.954993 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.956113 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.956608 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.957065 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.957524 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.958025 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.958477 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.958949 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.959432 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.959921 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.ssl = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.960628 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.ssl_ca_file = {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.961286 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.ssl_cert_file = {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.961739 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.ssl_enforce_fips_mode = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.962223 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.ssl_key_file = {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.962670 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_rabbit.ssl_version = {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.963152 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_notifications.driver = [] {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.963840 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_notifications.retry = -1 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.964339 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.964752 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] oslo_messaging_notifications.transport_url = **** {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.965257 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] key_manager.backend = cinder.keymgr.conf_key_mgr.ConfKeyManager {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.965742 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] barbican.auth_endpoint = http://localhost/identity/v3 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.966305 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] barbican.barbican_api_version = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.966726 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] barbican.barbican_endpoint = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.967174 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] barbican.barbican_endpoint_type = public {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.967592 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] barbican.barbican_region_name = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.968014 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] barbican.cafile = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.968451 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] barbican.certfile = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.968871 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] barbican.collect_timing = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.969278 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] barbican.insecure = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.969681 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] barbican.keyfile = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.970184 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] barbican.number_of_retries = 60 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.971761 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] barbican.retry_delay = 1 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.972174 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] barbican.send_service_user_token = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.972583 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] barbican.split_loggers = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.972995 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] barbican.timeout = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.973406 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] barbican.verify_ssl = True {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.973973 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] barbican.verify_ssl_path = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.974396 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] barbican_service_user.auth_section = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.974809 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] barbican_service_user.auth_type = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.975232 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] barbican_service_user.cafile = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.975636 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] barbican_service_user.certfile = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.976207 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] barbican_service_user.collect_timing = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.976643 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] barbican_service_user.insecure = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.977033 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] barbican_service_user.keyfile = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.977445 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] barbican_service_user.split_loggers = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.977891 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] barbican_service_user.timeout = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.978674 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] vault.approle_role_id = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.979082 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] vault.approle_secret_id = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.979504 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] vault.cafile = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.980054 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] vault.certfile = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.980736 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] vault.collect_timing = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.981147 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] vault.insecure = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.981569 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] vault.keyfile = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.982186 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] vault.kv_mountpoint = secret {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.982604 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] vault.kv_version = 2 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.983015 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] vault.namespace = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.983473 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] vault.root_token_id = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.984019 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] vault.split_loggers = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.984447 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] vault.ssl_ca_crt_file = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.984850 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] vault.timeout = None {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.985259 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] vault.use_ssl = False {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.985702 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] vault.vault_url = http://127.0.0.1:8200 {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 28 20:52:57.986454 ncdv-master-1 cinder-backup[135384]: DEBUG oslo_service.service [None req-a29bed21-012c-40ab-9ec6-b01eefe1a8af None None] ******************************************************************************** {{(pid=135384) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} May 28 20:52:58.054848 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_db.sqlalchemy.engines [None req-4e1c8c72-88fc-48af-a286-271ec58fec98 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=135833) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 28 20:52:58.059697 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writer thread running {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} May 28 20:52:58.086347 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.lockutils [None req-4e1c8c72-88fc-48af-a286-271ec58fec98 None None] Lock "cleanup_incomplete_backups_135384" "released" by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: held 0.250s {{(pid=135833) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:400}} May 28 20:52:58.093494 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_db.sqlalchemy.engines [None req-4e1c8c72-88fc-48af-a286-271ec58fec98 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=135833) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 28 20:52:58.107677 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-4e1c8c72-88fc-48af-a286-271ec58fec98 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 20:52:58.126603 ncdv-master-1 cinder-backup[135833]: INFO cinder.keymgr.migration [None req-7eac9c9a-1381-4afd-bc25-5d03d31acd9f None None] Not migrating encryption keys because the ConfKeyManager is still in use. May 28 20:52:58.147445 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.service [None req-4e1c8c72-88fc-48af-a286-271ec58fec98 None None] Creating RPC server for service cinder-backup {{(pid=135833) start /opt/stack/cinder/cinder/service.py:231}} May 28 20:52:58.161237 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.service [None req-4e1c8c72-88fc-48af-a286-271ec58fec98 None None] Pinning object versions for RPC server serializer to 1.39 {{(pid=135833) start /opt/stack/cinder/cinder/service.py:237}} May 28 20:53:07.634731 ncdv-master-1 cinder-backup[135384]: DEBUG dbcounter [-] [135384] Writing DB stats cinder:SELECT=16,cinder:INSERT=1 {{(pid=135384) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 20:53:08.148949 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=17,cinder:INSERT=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 20:53:08.151769 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 20:53:28.208886 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 20:53:38.211520 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 20:53:58.203128 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 20:54:18.213616 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 20:54:18.217519 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 20:54:35.201452 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 20:54:35.202002 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 20:54:58.217570 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 20:54:58.219156 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 20:55:35.197467 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 20:55:35.198209 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 20:55:38.217243 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 20:55:38.218763 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=4 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 20:56:18.219071 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 20:56:18.220576 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 20:56:35.202362 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 20:56:35.203005 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 20:56:58.221317 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 20:56:58.222020 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 20:57:08.349766 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 20:57:35.206926 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 20:57:35.207523 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 20:57:38.221172 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 20:57:48.218626 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 20:58:18.221556 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 20:58:18.222297 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 20:58:28.392786 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 20:58:35.211733 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 20:58:35.212314 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 20:58:58.223643 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 20:59:08.228081 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 20:59:35.216139 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 20:59:35.217374 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 20:59:38.225762 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 20:59:48.230784 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:00:18.226936 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:00:18.229670 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:00:35.221813 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:00:35.222503 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:00:58.224362 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:00:58.225858 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=9,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:01:08.339411 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:01:28.228285 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:01:35.227188 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:01:35.227885 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:01:38.235000 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:01:58.229516 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:02:08.231301 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=9,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:02:35.232378 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:02:35.233370 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:02:38.228034 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:02:48.235227 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:02:58.241330 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:03:28.236376 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:03:35.229447 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:03:35.230177 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:03:38.239266 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:03:58.241936 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:04:08.251873 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:04:18.245514 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:04:35.229756 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:04:35.229756 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:04:38.246400 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:04:48.246774 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:05:18.253019 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:05:35.230294 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:05:35.235263 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:05:48.249759 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:05:58.246453 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:06:18.261458 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:06:35.231532 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:06:35.232236 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:06:38.267185 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:07:08.261365 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:07:08.280171 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=9,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:07:28.284079 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:07:35.234242 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:07:35.234987 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:07:48.289819 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=4 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:07:58.294692 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:08:28.292876 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:08:28.293789 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:08:35.229625 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:08:35.230296 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:08:58.287654 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:09:08.305713 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:09:18.323400 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:09:35.234532 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:09:35.235108 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:09:38.300264 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:09:58.305180 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:10:08.316157 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:10:18.313800 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:10:28.309969 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:10:35.243922 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:10:35.243922 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:10:48.306806 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:11:08.310723 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:11:28.312354 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:11:35.236538 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:11:35.238849 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:11:48.303722 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:11:58.318167 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:12:28.306242 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=9,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:12:35.234889 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:12:35.235581 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:12:48.307992 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:12:58.315634 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:13:18.320179 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:13:18.320823 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:13:35.230840 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:13:35.231514 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:13:38.334346 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:13:58.315706 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=10,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:14:18.328720 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:14:35.226944 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:14:35.227906 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:14:38.325464 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:14:58.333948 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:15:18.339877 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:15:28.343267 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:15:35.232965 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:15:35.234354 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:15:38.357270 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:15:58.367871 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:16:18.364210 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:16:35.232021 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:16:35.232021 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:16:38.367190 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:16:58.369283 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:17:18.367183 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:17:35.230067 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:17:35.230794 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:17:38.374513 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:17:58.387310 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:18:18.389040 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:18:28.390287 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:18:35.232839 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:18:35.236039 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:18:48.389624 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:19:08.411702 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:19:28.414252 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:19:28.415086 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:19:35.230667 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:19:35.231279 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:19:58.422830 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:20:08.432418 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:20:28.426827 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:20:35.237168 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:20:35.237780 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:20:38.424819 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:21:08.423704 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:21:08.424385 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:21:35.237995 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:21:35.239042 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:21:48.439141 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=11,cinder:UPDATE=4 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:22:08.442231 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:22:28.434970 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:22:35.234582 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:22:35.238378 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:22:48.446643 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:23:08.441936 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:UPDATE=3,cinder:SELECT=4 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:23:28.457040 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:23:35.235012 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:23:35.235580 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:23:48.494370 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:23:48.494370 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:23:58.949406 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:24:28.449462 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:24:35.240480 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:24:35.241056 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:24:38.459589 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:24:48.471808 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:25:08.463550 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:25:08.465475 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:25:26.987237 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] Create backup started, backup: a5804b82-1246-4fb7-89ec-ca69ce549f7c volume: 3fe26f75-3c61-47e7-b119-e1b004aba4b9. May 28 21:25:27.050328 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-28T21:25:26Z,data_timestamp=2022-05-28T21:25:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsV39Test-Backup-1353229858',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=a5804b82-1246-4fb7-89ec-ca69ce549f7c,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='f791936ea1fc49cfb07986f26a47cabe',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='2267c55067d549e79963425bfc21f45c',volume_id=3fe26f75-3c61-47e7-b119-e1b004aba4b9) May 28 21:25:27.137257 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.utils [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.33', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:25:27.139323 ncdv-master-1 cinder-backup[135833]: INFO oslo.privsep.daemon [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] Running privsep helper: ['sudo', 'cinder-rootwrap', '/etc/cinder/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/cinder/cinder.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmp7muslwi1/privsep.sock'] May 28 21:25:27.210590 ncdv-master-1 sudo[167596]: ubuntu : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf privsep-helper --config-file /etc/cinder/cinder.conf --privsep_context os_brick.privileged.default --privsep_sock_path /tmp/tmp7muslwi1/privsep.sock May 28 21:25:27.212260 ncdv-master-1 sudo[167596]: pam_unix(sudo:session): session opened for user root by (uid=0) May 28 21:25:28.476373 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_db.sqlalchemy.engines [None req-1e29fabe-dc57-4ef4-ad80-ba98d322b6cd None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=135833) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 28 21:25:33.998284 ncdv-master-1 sudo[167596]: pam_unix(sudo:session): session closed for user root May 28 21:25:34.002222 ncdv-master-1 cinder-backup[135833]: INFO oslo.privsep.daemon [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] Spawned new privsep daemon via rootwrap May 28 21:25:34.003944 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] Accepted privsep connection to /tmp/tmp7muslwi1/privsep.sock {{(pid=135833) __init__ /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:362}} May 28 21:25:34.008791 ncdv-master-1 cinder-backup[135833]: INFO oslo.privsep.daemon [-] privsep daemon starting May 28 21:25:34.010950 ncdv-master-1 cinder-backup[135833]: INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 May 28 21:25:34.011855 ncdv-master-1 cinder-backup[135833]: INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none May 28 21:25:34.013192 ncdv-master-1 cinder-backup[135833]: INFO oslo.privsep.daemon [-] privsep daemon running as pid 167755 May 28 21:25:34.026775 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[962670f2-2700-4a88-88f0-63413d308ace]: (2,) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:25:34.468955 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:25:34.526588 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.058s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:25:34.527237 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[4e2311f7-8437-44d4-8fb9-f6022167080a]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:a90a241cea8\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:25:34.528873 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:25:34.550887 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 28 21:25:34.574649 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[f7bced60-bdfc-4938-81e2-bbdd216cbbf4]: [Errno 2] No such file or directory: 'systool' {{(pid=167755) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 28 21:25:34.574649 ncdv-master-1 cinder-backup[135833]: Traceback (most recent call last): May 28 21:25:34.574649 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 28 21:25:34.574649 ncdv-master-1 cinder-backup[135833]: ret = func(*f_args, **f_kwargs) May 28 21:25:34.574649 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 28 21:25:34.574649 ncdv-master-1 cinder-backup[135833]: return func(*args, **kwargs) May 28 21:25:34.574649 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 28 21:25:34.574649 ncdv-master-1 cinder-backup[135833]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 28 21:25:34.574649 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 28 21:25:34.574649 ncdv-master-1 cinder-backup[135833]: return putils.execute(on_execute=on_execute, May 28 21:25:34.574649 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 28 21:25:34.574649 ncdv-master-1 cinder-backup[135833]: obj = subprocess.Popen(cmd, May 28 21:25:34.574649 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 28 21:25:34.574649 ncdv-master-1 cinder-backup[135833]: self._execute_child(args, executable, preexec_fn, close_fds, May 28 21:25:34.574649 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 28 21:25:34.574649 ncdv-master-1 cinder-backup[135833]: raise child_exception_type(errno_num, err_msg, err_filename) May 28 21:25:34.574649 ncdv-master-1 cinder-backup[135833]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 28 21:25:34.576331 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[f7bced60-bdfc-4938-81e2-bbdd216cbbf4]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:25:34.578066 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:25:34.605319 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 28 21:25:34.606052 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[f2030d83-32bb-4050-85ed-75bd4cfbd679]: [Errno 2] No such file or directory: 'systool' {{(pid=167755) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 28 21:25:34.606052 ncdv-master-1 cinder-backup[135833]: Traceback (most recent call last): May 28 21:25:34.606052 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 28 21:25:34.606052 ncdv-master-1 cinder-backup[135833]: ret = func(*f_args, **f_kwargs) May 28 21:25:34.606052 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 28 21:25:34.606052 ncdv-master-1 cinder-backup[135833]: return func(*args, **kwargs) May 28 21:25:34.606052 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 28 21:25:34.606052 ncdv-master-1 cinder-backup[135833]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 28 21:25:34.606052 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 28 21:25:34.606052 ncdv-master-1 cinder-backup[135833]: return putils.execute(on_execute=on_execute, May 28 21:25:34.606052 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 28 21:25:34.606052 ncdv-master-1 cinder-backup[135833]: obj = subprocess.Popen(cmd, May 28 21:25:34.606052 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 28 21:25:34.606052 ncdv-master-1 cinder-backup[135833]: self._execute_child(args, executable, preexec_fn, close_fds, May 28 21:25:34.606052 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 28 21:25:34.606052 ncdv-master-1 cinder-backup[135833]: raise child_exception_type(errno_num, err_msg, err_filename) May 28 21:25:34.606052 ncdv-master-1 cinder-backup[135833]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 28 21:25:34.607691 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[f2030d83-32bb-4050-85ed-75bd4cfbd679]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:25:34.622720 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:25:34.674320 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.064s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:25:34.675978 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[1b711370-0845-4183-9b8f-5f335e851358]: (4, ('/dev/sda1\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:25:34.678302 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:25:34.746403 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.069s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:25:34.746723 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[eb881623-1793-4ff6-8019-bb2e28e9e459]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:25:34.748002 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:25:34.764448 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.016s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:25:34.764706 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[a5657334-a838-44da-902f-ce9931c17d0d]: (4, ('fa6a5dbb-8967-894e-8b8d-5384b73a89bf\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:25:34.765781 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] Running cmd (subprocess): nvme version {{(pid=135833) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:25:34.817261 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] CMD "nvme version" returned: 0 in 0.051s {{(pid=135833) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:25:34.846827 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=135833) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 28 21:25:34.847921 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=135833) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 28 21:25:34.848664 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:41d0083e-6728-42e9-88e9-cd92e21a8271 dsc: {{(pid=135833) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 28 21:25:34.849889 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.utils [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] <== get_connector_properties: return (7711ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.33', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:a90a241cea8', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'fa6a5dbb-8967-894e-8b8d-5384b73a89bf', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:41d0083e-6728-42e9-88e9-cd92e21a8271', 'found_dsc': ''} {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:25:35.242003 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:25:35.242003 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:25:36.617172 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connector [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] Factory for rbd on None {{(pid=135833) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 28 21:25:36.620017 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-3fe26f75-3c61-47e7-b119-e1b004aba4b9', 'hosts': ['10.222.0.33'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '3fe26f75-3c61-47e7-b119-e1b004aba4b9', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:25:36.621275 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.linuxrbd [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] opening connection to ceph cluster (timeout=-1). {{(pid=135833) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 28 21:25:36.831218 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] <== connect_volume: return (209ms) {'path': } {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:25:36.915713 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] Starting backup of volume='3fe26f75-3c61-47e7-b119-e1b004aba4b9'. {{(pid=135833) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1028}} May 28 21:25:36.916712 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] Volume file is RBD: attempting optimized backup {{(pid=135833) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1039}} May 28 21:25:36.979657 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] Creating base image 'volume-3fe26f75-3c61-47e7-b119-e1b004aba4b9.backup.a5804b82-1246-4fb7-89ec-ca69ce549f7c' {{(pid=135833) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:469}} May 28 21:25:36.980080 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=135833) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:277}} May 28 21:25:39.257152 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] Using --from-snap 'None' for incremental backup of volume 3fe26f75-3c61-47e7-b119-e1b004aba4b9. {{(pid=135833) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:811}} May 28 21:25:39.258239 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] Creating backup snapshot='backup.a5804b82-1246-4fb7-89ec-ca69ce549f7c.snap.1653773139.257785' {{(pid=135833) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:817}} May 28 21:25:40.337213 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] Performing differential transfer from 'volume-3fe26f75-3c61-47e7-b119-e1b004aba4b9' to 'volume-3fe26f75-3c61-47e7-b119-e1b004aba4b9.backup.a5804b82-1246-4fb7-89ec-ca69ce549f7c' {{(pid=135833) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:658}} May 28 21:25:40.337765 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_esti8tbo --pool volumes volumes/volume-3fe26f75-3c61-47e7-b119-e1b004aba4b9@backup.a5804b82-1246-4fb7-89ec-ca69ce549f7c.snap.1653773139.257785 -' into... {{(pid=135833) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:615}} May 28 21:25:40.338006 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-3fe26f75-3c61-47e7-b119-e1b004aba4b9.backup.a5804b82-1246-4fb7-89ec-ca69ce549f7c' {{(pid=135833) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:616}} May 28 21:25:41.696010 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] Differential backup transfer completed in 1.3578s {{(pid=135833) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:837}} May 28 21:25:41.696719 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] Getting metadata type 'volume-base-metadata' {{(pid=135833) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 28 21:25:41.701968 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=15,cinder:UPDATE=4 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:25:41.710904 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=135833) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 28 21:25:41.711306 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] Getting metadata type 'volume-metadata' {{(pid=135833) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 28 21:25:41.724038 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] No metadata type 'volume-metadata' available {{(pid=135833) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 28 21:25:41.724420 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] Getting metadata type 'volume-glance-metadata' {{(pid=135833) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 28 21:25:41.734396 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] No metadata type 'volume-glance-metadata' available {{(pid=135833) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} May 28 21:25:41.734946 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] Backing up metadata for volume 3fe26f75-3c61-47e7-b119-e1b004aba4b9. {{(pid=135833) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:1003}} May 28 21:25:41.841273 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] Backup 'a5804b82-1246-4fb7-89ec-ca69ce549f7c' of volume 3fe26f75-3c61-47e7-b119-e1b004aba4b9 finished. {{(pid=135833) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1069}} May 28 21:25:41.843227 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-3fe26f75-3c61-47e7-b119-e1b004aba4b9', 'hosts': ['10.222.0.33'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '3fe26f75-3c61-47e7-b119-e1b004aba4b9', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:25:42.315961 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] <== disconnect_volume: return (465ms) None {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:25:42.624643 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-ff84e19c-a885-4d90-97f5-0c9c7ea0513c tempest-VolumesBackupsV39Test-1074139284 None] Create backup finished. backup: a5804b82-1246-4fb7-89ec-ca69ce549f7c. May 28 21:25:44.010778 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-217336a4-ad83-4604-a79b-83e667f3ca49 tempest-VolumesBackupsV39Test-1074139284 None] Delete backup started, backup: a5804b82-1246-4fb7-89ec-ca69ce549f7c. May 28 21:25:44.014422 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-217336a4-ad83-4604-a79b-83e667f3ca49 tempest-VolumesBackupsV39Test-1074139284 None] Delete started for backup=a5804b82-1246-4fb7-89ec-ca69ce549f7c {{(pid=135833) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1376}} May 28 21:25:44.014960 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-217336a4-ad83-4604-a79b-83e667f3ca49 tempest-VolumesBackupsV39Test-1074139284 None] Trying diff format basename='volume-3fe26f75-3c61-47e7-b119-e1b004aba4b9.backup.a5804b82-1246-4fb7-89ec-ca69ce549f7c' for backup base image of volume 3fe26f75-3c61-47e7-b119-e1b004aba4b9. {{(pid=135833) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:543}} May 28 21:25:44.211183 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-217336a4-ad83-4604-a79b-83e667f3ca49 tempest-VolumesBackupsV39Test-1074139284 None] Looking for snapshot of backup base 'volume-3fe26f75-3c61-47e7-b119-e1b004aba4b9.backup.a5804b82-1246-4fb7-89ec-ca69ce549f7c' {{(pid=135833) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:957}} May 28 21:25:44.211653 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-217336a4-ad83-4604-a79b-83e667f3ca49 tempest-VolumesBackupsV39Test-1074139284 None] Found snapshot 'backup.a5804b82-1246-4fb7-89ec-ca69ce549f7c.snap.1653773139.257785' {{(pid=135833) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:976}} May 28 21:25:44.212187 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-217336a4-ad83-4604-a79b-83e667f3ca49 tempest-VolumesBackupsV39Test-1074139284 None] Deleting backup snapshot='backup.a5804b82-1246-4fb7-89ec-ca69ce549f7c.snap.1653773139.257785' {{(pid=135833) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:501}} May 28 21:25:46.671005 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.drivers.ceph [None req-217336a4-ad83-4604-a79b-83e667f3ca49 tempest-VolumesBackupsV39Test-1074139284 None] Deleting backup base image='volume-3fe26f75-3c61-47e7-b119-e1b004aba4b9.backup.a5804b82-1246-4fb7-89ec-ca69ce549f7c' of volume 3fe26f75-3c61-47e7-b119-e1b004aba4b9. May 28 21:25:48.468584 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_db.sqlalchemy.engines [None req-1e29fabe-dc57-4ef4-ad80-ba98d322b6cd None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=135833) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 28 21:25:49.083501 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-217336a4-ad83-4604-a79b-83e667f3ca49 tempest-VolumesBackupsV39Test-1074139284 None] Base backup image='volume-3fe26f75-3c61-47e7-b119-e1b004aba4b9.backup.a5804b82-1246-4fb7-89ec-ca69ce549f7c' of volume 3fe26f75-3c61-47e7-b119-e1b004aba4b9 deleted. {{(pid=135833) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:591}} May 28 21:25:49.219739 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-217336a4-ad83-4604-a79b-83e667f3ca49 tempest-VolumesBackupsV39Test-1074139284 None] Delete of backup 'a5804b82-1246-4fb7-89ec-ca69ce549f7c' for volume '3fe26f75-3c61-47e7-b119-e1b004aba4b9' finished. {{(pid=135833) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1404}} May 28 21:25:49.389215 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.quota [None req-217336a4-ad83-4604-a79b-83e667f3ca49 tempest-VolumesBackupsV39Test-1074139284 None] Created reservations ['952ddcf2-1d7c-4405-9a43-18bee05822e9', '788cd686-b011-4b1b-a007-53f1c16bb2e7'] {{(pid=135833) reserve /opt/stack/cinder/cinder/quota.py:755}} May 28 21:25:49.400317 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=21,cinder:UPDATE=5,cinder:INSERT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:25:49.604101 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-217336a4-ad83-4604-a79b-83e667f3ca49 tempest-VolumesBackupsV39Test-1074139284 None] Delete backup finished, backup a5804b82-1246-4fb7-89ec-ca69ce549f7c deleted. May 28 21:26:18.469263 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:26:18.470780 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=16,cinder:UPDATE=8 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:26:29.015359 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:26:35.240548 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:26:35.243034 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:26:48.468320 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:27:08.471029 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:27:18.473274 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:27:35.239230 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:27:35.239828 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:27:38.469704 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:27:48.473330 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:28:08.480253 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:28:18.486244 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:28:35.240045 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:28:35.240743 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:28:48.502033 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:28:58.513503 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=3,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:29:18.494810 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:29:19.055024 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] Create backup started, backup: 0048f84a-8eab-4495-a615-827ef86560e9 volume: f64c199d-aafe-4b81-8b40-af7ac5fa31b5. May 28 21:29:19.181681 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-28T21:29:18Z,data_timestamp=2022-05-28T21:29:18Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsTest-Backup-1185412182',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=0048f84a-8eab-4495-a615-827ef86560e9,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='071f3ee8ccba4f4a891cac48aaa54d0a',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='38d2c184e2ad446cb0481b1421439771',volume_id=f64c199d-aafe-4b81-8b40-af7ac5fa31b5) May 28 21:29:19.277566 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.utils [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.33', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:29:19.283394 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:29:19.324206 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.040s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:29:19.324786 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[e6534728-78e0-4aa6-9ac2-3222ce2b7a38]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:a90a241cea8\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:29:19.329344 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:29:19.339174 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 28 21:29:19.340382 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[783d7b7d-a95d-43b5-bc76-c8f22f8d4427]: [Errno 2] No such file or directory: 'systool' {{(pid=167755) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 28 21:29:19.340382 ncdv-master-1 cinder-backup[135833]: Traceback (most recent call last): May 28 21:29:19.340382 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 28 21:29:19.340382 ncdv-master-1 cinder-backup[135833]: ret = func(*f_args, **f_kwargs) May 28 21:29:19.340382 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 28 21:29:19.340382 ncdv-master-1 cinder-backup[135833]: return func(*args, **kwargs) May 28 21:29:19.340382 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 28 21:29:19.340382 ncdv-master-1 cinder-backup[135833]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 28 21:29:19.340382 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 28 21:29:19.340382 ncdv-master-1 cinder-backup[135833]: return putils.execute(on_execute=on_execute, May 28 21:29:19.340382 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 28 21:29:19.340382 ncdv-master-1 cinder-backup[135833]: obj = subprocess.Popen(cmd, May 28 21:29:19.340382 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 28 21:29:19.340382 ncdv-master-1 cinder-backup[135833]: self._execute_child(args, executable, preexec_fn, close_fds, May 28 21:29:19.340382 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 28 21:29:19.340382 ncdv-master-1 cinder-backup[135833]: raise child_exception_type(errno_num, err_msg, err_filename) May 28 21:29:19.340382 ncdv-master-1 cinder-backup[135833]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 28 21:29:19.342410 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[783d7b7d-a95d-43b5-bc76-c8f22f8d4427]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:29:19.349540 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:29:19.373745 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 28 21:29:19.374141 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[e5647b25-7a11-4a92-897d-8b18d70e60f9]: [Errno 2] No such file or directory: 'systool' {{(pid=167755) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 28 21:29:19.374141 ncdv-master-1 cinder-backup[135833]: Traceback (most recent call last): May 28 21:29:19.374141 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 28 21:29:19.374141 ncdv-master-1 cinder-backup[135833]: ret = func(*f_args, **f_kwargs) May 28 21:29:19.374141 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 28 21:29:19.374141 ncdv-master-1 cinder-backup[135833]: return func(*args, **kwargs) May 28 21:29:19.374141 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 28 21:29:19.374141 ncdv-master-1 cinder-backup[135833]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 28 21:29:19.374141 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 28 21:29:19.374141 ncdv-master-1 cinder-backup[135833]: return putils.execute(on_execute=on_execute, May 28 21:29:19.374141 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 28 21:29:19.374141 ncdv-master-1 cinder-backup[135833]: obj = subprocess.Popen(cmd, May 28 21:29:19.374141 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 28 21:29:19.374141 ncdv-master-1 cinder-backup[135833]: self._execute_child(args, executable, preexec_fn, close_fds, May 28 21:29:19.374141 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 28 21:29:19.374141 ncdv-master-1 cinder-backup[135833]: raise child_exception_type(errno_num, err_msg, err_filename) May 28 21:29:19.374141 ncdv-master-1 cinder-backup[135833]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 28 21:29:19.376432 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[e5647b25-7a11-4a92-897d-8b18d70e60f9]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:29:19.378605 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:29:19.399944 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.021s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:29:19.400388 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[d990b2d8-bf5c-4713-b278-aa6a0532e082]: (4, ('/dev/sda1\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:29:19.404541 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:29:19.419357 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.016s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:29:19.420041 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[ad03bef6-75f2-4656-a86e-0ac059dc1b9e]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:29:19.422923 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:29:19.435327 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.013s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:29:19.437550 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[8c97af16-bee2-4004-bb11-54523e6ef9c0]: (4, ('fa6a5dbb-8967-894e-8b8d-5384b73a89bf\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:29:19.438668 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] Running cmd (subprocess): nvme version {{(pid=135833) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:29:19.472843 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] CMD "nvme version" returned: 0 in 0.033s {{(pid=135833) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:29:19.475349 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=135833) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 28 21:29:19.476051 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=135833) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 28 21:29:19.476546 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:41d0083e-6728-42e9-88e9-cd92e21a8271 dsc: {{(pid=135833) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 28 21:29:19.477324 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.utils [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] <== get_connector_properties: return (199ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.33', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:a90a241cea8', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'fa6a5dbb-8967-894e-8b8d-5384b73a89bf', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:41d0083e-6728-42e9-88e9-cd92e21a8271', 'found_dsc': ''} {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:29:20.500995 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connector [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] Factory for rbd on None {{(pid=135833) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 28 21:29:20.505072 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-f64c199d-aafe-4b81-8b40-af7ac5fa31b5', 'hosts': ['10.222.0.33'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'f64c199d-aafe-4b81-8b40-af7ac5fa31b5', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:29:20.507046 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.linuxrbd [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] opening connection to ceph cluster (timeout=-1). {{(pid=135833) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 28 21:29:20.669865 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] <== connect_volume: return (163ms) {'path': } {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:29:20.679574 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_db.sqlalchemy.engines [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=135833) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 28 21:29:20.681657 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=17,cinder:UPDATE=4 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:29:20.772657 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] Starting backup of volume='f64c199d-aafe-4b81-8b40-af7ac5fa31b5'. {{(pid=135833) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1028}} May 28 21:29:20.773765 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] Volume file is RBD: attempting optimized backup {{(pid=135833) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1039}} May 28 21:29:20.795921 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] Creating base image 'volume-f64c199d-aafe-4b81-8b40-af7ac5fa31b5.backup.0048f84a-8eab-4495-a615-827ef86560e9' {{(pid=135833) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:469}} May 28 21:29:20.796334 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=135833) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:277}} May 28 21:29:21.172875 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] Using --from-snap 'None' for incremental backup of volume f64c199d-aafe-4b81-8b40-af7ac5fa31b5. {{(pid=135833) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:811}} May 28 21:29:21.173285 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] Creating backup snapshot='backup.0048f84a-8eab-4495-a615-827ef86560e9.snap.1653773361.172898' {{(pid=135833) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:817}} May 28 21:29:21.970993 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] Performing differential transfer from 'volume-f64c199d-aafe-4b81-8b40-af7ac5fa31b5' to 'volume-f64c199d-aafe-4b81-8b40-af7ac5fa31b5.backup.0048f84a-8eab-4495-a615-827ef86560e9' {{(pid=135833) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:658}} May 28 21:29:21.972057 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_vqbuscqe --pool volumes volumes/volume-f64c199d-aafe-4b81-8b40-af7ac5fa31b5@backup.0048f84a-8eab-4495-a615-827ef86560e9.snap.1653773361.172898 -' into... {{(pid=135833) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:615}} May 28 21:29:21.974979 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-f64c199d-aafe-4b81-8b40-af7ac5fa31b5.backup.0048f84a-8eab-4495-a615-827ef86560e9' {{(pid=135833) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:616}} May 28 21:29:23.859605 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] Differential backup transfer completed in 1.8867s {{(pid=135833) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:837}} May 28 21:29:23.861945 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] Getting metadata type 'volume-base-metadata' {{(pid=135833) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 28 21:29:23.889742 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=135833) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 28 21:29:23.890196 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] Getting metadata type 'volume-metadata' {{(pid=135833) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 28 21:29:23.901193 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] No metadata type 'volume-metadata' available {{(pid=135833) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 28 21:29:23.901583 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] Getting metadata type 'volume-glance-metadata' {{(pid=135833) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 28 21:29:23.911124 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] No metadata type 'volume-glance-metadata' available {{(pid=135833) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} May 28 21:29:23.913109 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] Backing up metadata for volume f64c199d-aafe-4b81-8b40-af7ac5fa31b5. {{(pid=135833) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:1003}} May 28 21:29:24.205440 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] Backup '0048f84a-8eab-4495-a615-827ef86560e9' of volume f64c199d-aafe-4b81-8b40-af7ac5fa31b5 finished. {{(pid=135833) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1069}} May 28 21:29:24.210437 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-f64c199d-aafe-4b81-8b40-af7ac5fa31b5', 'hosts': ['10.222.0.33'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'f64c199d-aafe-4b81-8b40-af7ac5fa31b5', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:29:24.324233 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] <== disconnect_volume: return (111ms) None {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:29:24.607125 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-9197d28f-6186-41e5-be56-bf095d874c7a tempest-VolumesBackupsTest-1716294911 None] Create backup finished. backup: 0048f84a-8eab-4495-a615-827ef86560e9. May 28 21:29:25.404331 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-47732fac-5914-4b59-99b1-4711bfa66883 tempest-VolumesBackupsTest-1716294911 None] Delete backup started, backup: 0048f84a-8eab-4495-a615-827ef86560e9. May 28 21:29:25.406858 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-47732fac-5914-4b59-99b1-4711bfa66883 tempest-VolumesBackupsTest-1716294911 None] Delete started for backup=0048f84a-8eab-4495-a615-827ef86560e9 {{(pid=135833) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1376}} May 28 21:29:25.415675 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-47732fac-5914-4b59-99b1-4711bfa66883 tempest-VolumesBackupsTest-1716294911 None] Trying diff format basename='volume-f64c199d-aafe-4b81-8b40-af7ac5fa31b5.backup.0048f84a-8eab-4495-a615-827ef86560e9' for backup base image of volume f64c199d-aafe-4b81-8b40-af7ac5fa31b5. {{(pid=135833) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:543}} May 28 21:29:25.689888 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-47732fac-5914-4b59-99b1-4711bfa66883 tempest-VolumesBackupsTest-1716294911 None] Looking for snapshot of backup base 'volume-f64c199d-aafe-4b81-8b40-af7ac5fa31b5.backup.0048f84a-8eab-4495-a615-827ef86560e9' {{(pid=135833) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:957}} May 28 21:29:25.690627 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-47732fac-5914-4b59-99b1-4711bfa66883 tempest-VolumesBackupsTest-1716294911 None] Found snapshot 'backup.0048f84a-8eab-4495-a615-827ef86560e9.snap.1653773361.172898' {{(pid=135833) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:976}} May 28 21:29:25.694675 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-47732fac-5914-4b59-99b1-4711bfa66883 tempest-VolumesBackupsTest-1716294911 None] Deleting backup snapshot='backup.0048f84a-8eab-4495-a615-827ef86560e9.snap.1653773361.172898' {{(pid=135833) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:501}} May 28 21:29:27.483210 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.drivers.ceph [None req-47732fac-5914-4b59-99b1-4711bfa66883 tempest-VolumesBackupsTest-1716294911 None] Deleting backup base image='volume-f64c199d-aafe-4b81-8b40-af7ac5fa31b5.backup.0048f84a-8eab-4495-a615-827ef86560e9' of volume f64c199d-aafe-4b81-8b40-af7ac5fa31b5. May 28 21:29:28.499275 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_db.sqlalchemy.engines [None req-1e29fabe-dc57-4ef4-ad80-ba98d322b6cd None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=135833) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 28 21:29:30.402840 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-47732fac-5914-4b59-99b1-4711bfa66883 tempest-VolumesBackupsTest-1716294911 None] Base backup image='volume-f64c199d-aafe-4b81-8b40-af7ac5fa31b5.backup.0048f84a-8eab-4495-a615-827ef86560e9' of volume f64c199d-aafe-4b81-8b40-af7ac5fa31b5 deleted. {{(pid=135833) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:591}} May 28 21:29:30.560243 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-47732fac-5914-4b59-99b1-4711bfa66883 tempest-VolumesBackupsTest-1716294911 None] Delete of backup '0048f84a-8eab-4495-a615-827ef86560e9' for volume 'f64c199d-aafe-4b81-8b40-af7ac5fa31b5' finished. {{(pid=135833) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1404}} May 28 21:29:30.677039 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.quota [None req-47732fac-5914-4b59-99b1-4711bfa66883 tempest-VolumesBackupsTest-1716294911 None] Created reservations ['d30d99e9-a2b8-4f2c-a019-65b1d8372e1f', '992ef3c1-02a8-47a2-8c63-2430bac2c6e2'] {{(pid=135833) reserve /opt/stack/cinder/cinder/quota.py:755}} May 28 21:29:30.859028 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-47732fac-5914-4b59-99b1-4711bfa66883 tempest-VolumesBackupsTest-1716294911 None] Delete backup finished, backup 0048f84a-8eab-4495-a615-827ef86560e9 deleted. May 28 21:29:35.245541 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:29:35.246166 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:29:48.514441 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=15,cinder:UPDATE=5,cinder:INSERT=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:29:51.156935 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=24,cinder:UPDATE=7,cinder:INSERT=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:29:51.166717 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] Create backup started, backup: efc30e7d-29b7-4042-973b-169c70808f0d volume: dd38b1bb-0ed4-4cb0-b594-5f3aa84501bd. May 28 21:29:51.292580 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-28T21:29:50Z,data_timestamp=2022-05-28T21:29:50Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsTest-Backup-48790400',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=efc30e7d-29b7-4042-973b-169c70808f0d,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='071f3ee8ccba4f4a891cac48aaa54d0a',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='38d2c184e2ad446cb0481b1421439771',volume_id=dd38b1bb-0ed4-4cb0-b594-5f3aa84501bd) May 28 21:29:51.371722 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.utils [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.33', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:29:51.373763 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:29:51.403563 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.029s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:29:51.403949 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[75f0c626-5512-4d0b-839e-31ca78c2dcc5]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:a90a241cea8\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:29:51.410282 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:29:51.418785 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 28 21:29:51.421978 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[fe2b0dbd-8dd2-4ebc-a99c-f8d08c04c2d9]: [Errno 2] No such file or directory: 'systool' {{(pid=167755) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 28 21:29:51.421978 ncdv-master-1 cinder-backup[135833]: Traceback (most recent call last): May 28 21:29:51.421978 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 28 21:29:51.421978 ncdv-master-1 cinder-backup[135833]: ret = func(*f_args, **f_kwargs) May 28 21:29:51.421978 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 28 21:29:51.421978 ncdv-master-1 cinder-backup[135833]: return func(*args, **kwargs) May 28 21:29:51.421978 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 28 21:29:51.421978 ncdv-master-1 cinder-backup[135833]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 28 21:29:51.421978 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 28 21:29:51.421978 ncdv-master-1 cinder-backup[135833]: return putils.execute(on_execute=on_execute, May 28 21:29:51.421978 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 28 21:29:51.421978 ncdv-master-1 cinder-backup[135833]: obj = subprocess.Popen(cmd, May 28 21:29:51.421978 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 28 21:29:51.421978 ncdv-master-1 cinder-backup[135833]: self._execute_child(args, executable, preexec_fn, close_fds, May 28 21:29:51.421978 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 28 21:29:51.421978 ncdv-master-1 cinder-backup[135833]: raise child_exception_type(errno_num, err_msg, err_filename) May 28 21:29:51.421978 ncdv-master-1 cinder-backup[135833]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 28 21:29:51.421978 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[fe2b0dbd-8dd2-4ebc-a99c-f8d08c04c2d9]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:29:51.421978 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:29:51.433547 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 28 21:29:51.434746 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[f52c6d1c-b8bb-4832-87c7-ac1a5aed9766]: [Errno 2] No such file or directory: 'systool' {{(pid=167755) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 28 21:29:51.434746 ncdv-master-1 cinder-backup[135833]: Traceback (most recent call last): May 28 21:29:51.434746 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 28 21:29:51.434746 ncdv-master-1 cinder-backup[135833]: ret = func(*f_args, **f_kwargs) May 28 21:29:51.434746 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 28 21:29:51.434746 ncdv-master-1 cinder-backup[135833]: return func(*args, **kwargs) May 28 21:29:51.434746 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 28 21:29:51.434746 ncdv-master-1 cinder-backup[135833]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 28 21:29:51.434746 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 28 21:29:51.434746 ncdv-master-1 cinder-backup[135833]: return putils.execute(on_execute=on_execute, May 28 21:29:51.434746 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 28 21:29:51.434746 ncdv-master-1 cinder-backup[135833]: obj = subprocess.Popen(cmd, May 28 21:29:51.434746 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 28 21:29:51.434746 ncdv-master-1 cinder-backup[135833]: self._execute_child(args, executable, preexec_fn, close_fds, May 28 21:29:51.434746 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 28 21:29:51.434746 ncdv-master-1 cinder-backup[135833]: raise child_exception_type(errno_num, err_msg, err_filename) May 28 21:29:51.434746 ncdv-master-1 cinder-backup[135833]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 28 21:29:51.436102 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[f52c6d1c-b8bb-4832-87c7-ac1a5aed9766]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:29:51.436870 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:29:51.455191 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.017s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:29:51.456057 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[d1cced75-f424-4c76-ac32-686e800d29b1]: (4, ('/dev/sda1\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:29:51.458328 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:29:51.475282 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.016s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:29:51.475608 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[0a66e19f-4b1c-4116-87a5-2e4656c0eea9]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:29:51.477769 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:29:51.489742 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.012s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:29:51.490059 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[edfa481c-13e3-474a-9786-4734dbeabfb5]: (4, ('fa6a5dbb-8967-894e-8b8d-5384b73a89bf\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:29:51.490967 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] Running cmd (subprocess): nvme version {{(pid=135833) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:29:51.512407 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] CMD "nvme version" returned: 0 in 0.021s {{(pid=135833) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:29:51.514119 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=135833) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 28 21:29:51.514492 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=135833) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 28 21:29:51.515094 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:41d0083e-6728-42e9-88e9-cd92e21a8271 dsc: {{(pid=135833) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 28 21:29:51.515736 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.utils [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] <== get_connector_properties: return (143ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.33', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:a90a241cea8', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'fa6a5dbb-8967-894e-8b8d-5384b73a89bf', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:41d0083e-6728-42e9-88e9-cd92e21a8271', 'found_dsc': ''} {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:29:52.329909 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connector [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] Factory for rbd on None {{(pid=135833) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 28 21:29:52.331530 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-dd38b1bb-0ed4-4cb0-b594-5f3aa84501bd', 'hosts': ['10.222.0.33'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'dd38b1bb-0ed4-4cb0-b594-5f3aa84501bd', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:29:52.333206 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.linuxrbd [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] opening connection to ceph cluster (timeout=-1). {{(pid=135833) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 28 21:29:52.545224 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] <== connect_volume: return (213ms) {'path': } {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:29:52.557370 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_db.sqlalchemy.engines [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=135833) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 28 21:29:52.871116 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] Starting backup of volume='dd38b1bb-0ed4-4cb0-b594-5f3aa84501bd'. {{(pid=135833) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1028}} May 28 21:29:52.871116 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] Volume file is RBD: attempting optimized backup {{(pid=135833) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1039}} May 28 21:29:52.871116 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] Creating base image 'volume-dd38b1bb-0ed4-4cb0-b594-5f3aa84501bd.backup.efc30e7d-29b7-4042-973b-169c70808f0d' {{(pid=135833) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:469}} May 28 21:29:52.871116 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=135833) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:277}} May 28 21:29:53.241782 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] Using --from-snap 'None' for incremental backup of volume dd38b1bb-0ed4-4cb0-b594-5f3aa84501bd. {{(pid=135833) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:811}} May 28 21:29:53.242912 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] Creating backup snapshot='backup.efc30e7d-29b7-4042-973b-169c70808f0d.snap.1653773393.2418056' {{(pid=135833) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:817}} May 28 21:29:54.477217 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] Performing differential transfer from 'volume-dd38b1bb-0ed4-4cb0-b594-5f3aa84501bd' to 'volume-dd38b1bb-0ed4-4cb0-b594-5f3aa84501bd.backup.efc30e7d-29b7-4042-973b-169c70808f0d' {{(pid=135833) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:658}} May 28 21:29:54.479241 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_vrrxh4rj --pool volumes volumes/volume-dd38b1bb-0ed4-4cb0-b594-5f3aa84501bd@backup.efc30e7d-29b7-4042-973b-169c70808f0d.snap.1653773393.2418056 -' into... {{(pid=135833) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:615}} May 28 21:29:54.483063 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-dd38b1bb-0ed4-4cb0-b594-5f3aa84501bd.backup.efc30e7d-29b7-4042-973b-169c70808f0d' {{(pid=135833) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:616}} May 28 21:30:08.507327 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=9,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:30:18.515641 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:30:28.502427 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:30:35.242126 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:30:35.242802 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:30:41.750103 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] Differential backup transfer completed in 47.2735s {{(pid=135833) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:837}} May 28 21:30:41.750748 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] Getting metadata type 'volume-base-metadata' {{(pid=135833) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 28 21:30:41.763403 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=135833) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 28 21:30:41.763799 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] Getting metadata type 'volume-metadata' {{(pid=135833) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 28 21:30:41.771119 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] No metadata type 'volume-metadata' available {{(pid=135833) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 28 21:30:41.771503 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] Getting metadata type 'volume-glance-metadata' {{(pid=135833) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 28 21:30:41.778557 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=135833) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}} May 28 21:30:41.779117 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] Backing up metadata for volume dd38b1bb-0ed4-4cb0-b594-5f3aa84501bd. {{(pid=135833) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:1003}} May 28 21:30:42.105949 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] Backup 'efc30e7d-29b7-4042-973b-169c70808f0d' of volume dd38b1bb-0ed4-4cb0-b594-5f3aa84501bd finished. {{(pid=135833) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1069}} May 28 21:30:42.107657 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-dd38b1bb-0ed4-4cb0-b594-5f3aa84501bd', 'hosts': ['10.222.0.33'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'dd38b1bb-0ed4-4cb0-b594-5f3aa84501bd', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:30:42.390714 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] <== disconnect_volume: return (282ms) None {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:30:42.641208 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-b550fd32-75d3-4aa4-bb52-1dffdf4e3e18 tempest-VolumesBackupsTest-1716294911 None] Create backup finished. backup: efc30e7d-29b7-4042-973b-169c70808f0d. May 28 21:30:45.891760 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] Restore backup started, backup: efc30e7d-29b7-4042-973b-169c70808f0d volume: 6589398c-11d6-483a-b206-0202e0f288ce. May 28 21:30:45.907067 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.utils [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.33', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:30:45.908689 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:30:45.923418 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.014s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:30:45.923794 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[cafb6cab-ef5f-412f-adfd-ea4f46114c21]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:a90a241cea8\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:30:45.925991 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:30:45.932418 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 28 21:30:45.933146 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[d51b73cf-069b-486a-80f6-2b7f77443057]: [Errno 2] No such file or directory: 'systool' {{(pid=167755) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 28 21:30:45.933146 ncdv-master-1 cinder-backup[135833]: Traceback (most recent call last): May 28 21:30:45.933146 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 28 21:30:45.933146 ncdv-master-1 cinder-backup[135833]: ret = func(*f_args, **f_kwargs) May 28 21:30:45.933146 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 28 21:30:45.933146 ncdv-master-1 cinder-backup[135833]: return func(*args, **kwargs) May 28 21:30:45.933146 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 28 21:30:45.933146 ncdv-master-1 cinder-backup[135833]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 28 21:30:45.933146 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 28 21:30:45.933146 ncdv-master-1 cinder-backup[135833]: return putils.execute(on_execute=on_execute, May 28 21:30:45.933146 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 28 21:30:45.933146 ncdv-master-1 cinder-backup[135833]: obj = subprocess.Popen(cmd, May 28 21:30:45.933146 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 28 21:30:45.933146 ncdv-master-1 cinder-backup[135833]: self._execute_child(args, executable, preexec_fn, close_fds, May 28 21:30:45.933146 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 28 21:30:45.933146 ncdv-master-1 cinder-backup[135833]: raise child_exception_type(errno_num, err_msg, err_filename) May 28 21:30:45.933146 ncdv-master-1 cinder-backup[135833]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 28 21:30:45.934307 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[d51b73cf-069b-486a-80f6-2b7f77443057]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:30:45.935144 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:30:45.943266 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 28 21:30:45.943901 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[006ffba1-701a-4df3-8bca-802279f7862b]: [Errno 2] No such file or directory: 'systool' {{(pid=167755) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 28 21:30:45.943901 ncdv-master-1 cinder-backup[135833]: Traceback (most recent call last): May 28 21:30:45.943901 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 28 21:30:45.943901 ncdv-master-1 cinder-backup[135833]: ret = func(*f_args, **f_kwargs) May 28 21:30:45.943901 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 28 21:30:45.943901 ncdv-master-1 cinder-backup[135833]: return func(*args, **kwargs) May 28 21:30:45.943901 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 28 21:30:45.943901 ncdv-master-1 cinder-backup[135833]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 28 21:30:45.943901 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 28 21:30:45.943901 ncdv-master-1 cinder-backup[135833]: return putils.execute(on_execute=on_execute, May 28 21:30:45.943901 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 28 21:30:45.943901 ncdv-master-1 cinder-backup[135833]: obj = subprocess.Popen(cmd, May 28 21:30:45.943901 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 28 21:30:45.943901 ncdv-master-1 cinder-backup[135833]: self._execute_child(args, executable, preexec_fn, close_fds, May 28 21:30:45.943901 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 28 21:30:45.943901 ncdv-master-1 cinder-backup[135833]: raise child_exception_type(errno_num, err_msg, err_filename) May 28 21:30:45.943901 ncdv-master-1 cinder-backup[135833]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 28 21:30:45.945370 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[006ffba1-701a-4df3-8bca-802279f7862b]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:30:45.945966 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:30:45.955194 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.009s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:30:45.955604 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[9eeef8fd-8992-4f28-81cf-db8c3fbf73b7]: (4, ('/dev/sda1\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:30:45.956737 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:30:45.966699 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.009s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:30:45.967140 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[1f5d0992-38bc-40ce-8b3b-63ac2b830b9e]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:30:45.968675 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:30:45.976703 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.008s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:30:45.977052 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[b71dc941-7d11-4eb2-b5c3-c2f5e4da2f32]: (4, ('fa6a5dbb-8967-894e-8b8d-5384b73a89bf\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:30:45.978030 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] Running cmd (subprocess): nvme version {{(pid=135833) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:30:46.009112 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] CMD "nvme version" returned: 0 in 0.031s {{(pid=135833) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:30:46.011210 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=135833) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 28 21:30:46.011667 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=135833) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 28 21:30:46.012158 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:41d0083e-6728-42e9-88e9-cd92e21a8271 dsc: {{(pid=135833) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 28 21:30:46.012807 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.utils [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] <== get_connector_properties: return (104ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.33', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:a90a241cea8', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'fa6a5dbb-8967-894e-8b8d-5384b73a89bf', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:41d0083e-6728-42e9-88e9-cd92e21a8271', 'found_dsc': ''} {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:30:46.816412 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connector [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] Factory for rbd on None {{(pid=135833) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 28 21:30:46.818282 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-6589398c-11d6-483a-b206-0202e0f288ce', 'hosts': ['10.222.0.33'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '6589398c-11d6-483a-b206-0202e0f288ce', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:30:46.819514 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.linuxrbd [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] opening connection to ceph cluster (timeout=-1). {{(pid=135833) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 28 21:30:46.946603 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] <== connect_volume: return (127ms) {'path': } {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:30:46.957131 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_db.sqlalchemy.engines [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=135833) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 28 21:30:46.970511 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] Starting restore from Ceph backup=efc30e7d-29b7-4042-973b-169c70808f0d to volume=volume-6589398c-11d6-483a-b206-0202e0f288ce {{(pid=135833) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1349}} May 28 21:30:47.080577 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] Looking for snapshot of backup base 'volume-dd38b1bb-0ed4-4cb0-b594-5f3aa84501bd.backup.efc30e7d-29b7-4042-973b-169c70808f0d' {{(pid=135833) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:957}} May 28 21:30:47.081047 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] Found snapshot 'backup.efc30e7d-29b7-4042-973b-169c70808f0d.snap.1653773393.2418056' {{(pid=135833) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:976}} May 28 21:30:47.086977 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] Volume file is RBD. {{(pid=135833) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1251}} May 28 21:30:47.163631 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] Attempting differential restore. {{(pid=135833) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1304}} May 28 21:30:47.165654 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] Attempting incremental restore from base='volume-dd38b1bb-0ed4-4cb0-b594-5f3aa84501bd.backup.efc30e7d-29b7-4042-973b-169c70808f0d' snap='backup.efc30e7d-29b7-4042-973b-169c70808f0d.snap.1653773393.2418056' {{(pid=135833) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1150}} May 28 21:30:47.166084 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] Performing differential transfer from 'volume-dd38b1bb-0ed4-4cb0-b594-5f3aa84501bd.backup.efc30e7d-29b7-4042-973b-169c70808f0d' to 'volume-6589398c-11d6-483a-b206-0202e0f288ce' {{(pid=135833) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:658}} May 28 21:30:47.168165 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-dd38b1bb-0ed4-4cb0-b594-5f3aa84501bd.backup.efc30e7d-29b7-4042-973b-169c70808f0d@backup.efc30e7d-29b7-4042-973b-169c70808f0d.snap.1653773393.2418056 -' into... {{(pid=135833) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:615}} May 28 21:30:47.168551 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_0adlbelf --pool volumes - volumes/volume-6589398c-11d6-483a-b206-0202e0f288ce' {{(pid=135833) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:616}} May 28 21:30:58.510387 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=12,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:30:58.511139 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=16,cinder:UPDATE=4 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:31:28.502635 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:31:34.006897 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=10,cinder:UPDATE=4 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:31:34.017662 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] Create backup started, backup: 7039d769-0e89-476f-a644-872f3ecd222f volume: bac5fe86-e590-414b-bff3-906130d40c5b. May 28 21:31:34.083894 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-28T21:31:33Z,data_timestamp=2022-05-28T21:31:33Z,deleted=False,deleted_at=None,display_description=None,display_name=None,encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=7039d769-0e89-476f-a644-872f3ecd222f,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='5241f87816134389b836a8eae1d53bd3',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='c93401a279c9403ca62627e829082968',volume_id=bac5fe86-e590-414b-bff3-906130d40c5b) May 28 21:31:34.125771 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.utils [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.33', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:31:34.127205 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:31:34.135557 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.008s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:31:34.135956 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[19abf622-6bc7-4c2f-bcbb-f0692fd26340]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:a90a241cea8\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:31:34.137187 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:31:34.144141 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 28 21:31:34.144719 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[6684c8fe-f86c-4cfb-ae4b-bf38d4d2c066]: [Errno 2] No such file or directory: 'systool' {{(pid=167755) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 28 21:31:34.144719 ncdv-master-1 cinder-backup[135833]: Traceback (most recent call last): May 28 21:31:34.144719 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 28 21:31:34.144719 ncdv-master-1 cinder-backup[135833]: ret = func(*f_args, **f_kwargs) May 28 21:31:34.144719 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 28 21:31:34.144719 ncdv-master-1 cinder-backup[135833]: return func(*args, **kwargs) May 28 21:31:34.144719 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 28 21:31:34.144719 ncdv-master-1 cinder-backup[135833]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 28 21:31:34.144719 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 28 21:31:34.144719 ncdv-master-1 cinder-backup[135833]: return putils.execute(on_execute=on_execute, May 28 21:31:34.144719 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 28 21:31:34.144719 ncdv-master-1 cinder-backup[135833]: obj = subprocess.Popen(cmd, May 28 21:31:34.144719 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 28 21:31:34.144719 ncdv-master-1 cinder-backup[135833]: self._execute_child(args, executable, preexec_fn, close_fds, May 28 21:31:34.144719 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 28 21:31:34.144719 ncdv-master-1 cinder-backup[135833]: raise child_exception_type(errno_num, err_msg, err_filename) May 28 21:31:34.144719 ncdv-master-1 cinder-backup[135833]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 28 21:31:34.145942 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[6684c8fe-f86c-4cfb-ae4b-bf38d4d2c066]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:31:34.146226 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:31:34.152097 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 28 21:31:34.152611 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[56167b45-63eb-4f2f-99ae-9b4ec3cacb0c]: [Errno 2] No such file or directory: 'systool' {{(pid=167755) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 28 21:31:34.152611 ncdv-master-1 cinder-backup[135833]: Traceback (most recent call last): May 28 21:31:34.152611 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 28 21:31:34.152611 ncdv-master-1 cinder-backup[135833]: ret = func(*f_args, **f_kwargs) May 28 21:31:34.152611 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 28 21:31:34.152611 ncdv-master-1 cinder-backup[135833]: return func(*args, **kwargs) May 28 21:31:34.152611 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 28 21:31:34.152611 ncdv-master-1 cinder-backup[135833]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 28 21:31:34.152611 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 28 21:31:34.152611 ncdv-master-1 cinder-backup[135833]: return putils.execute(on_execute=on_execute, May 28 21:31:34.152611 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 28 21:31:34.152611 ncdv-master-1 cinder-backup[135833]: obj = subprocess.Popen(cmd, May 28 21:31:34.152611 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 28 21:31:34.152611 ncdv-master-1 cinder-backup[135833]: self._execute_child(args, executable, preexec_fn, close_fds, May 28 21:31:34.152611 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 28 21:31:34.152611 ncdv-master-1 cinder-backup[135833]: raise child_exception_type(errno_num, err_msg, err_filename) May 28 21:31:34.152611 ncdv-master-1 cinder-backup[135833]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 28 21:31:34.153723 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[56167b45-63eb-4f2f-99ae-9b4ec3cacb0c]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:31:34.154385 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:31:34.175934 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.021s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:31:34.176269 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[0013138a-167f-4ffd-9776-859047c7d242]: (4, ('/dev/sda1\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:31:34.180229 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:31:34.193145 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.015s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:31:34.194125 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[150c7454-8c5f-471c-84f9-50872e2fc03f]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:31:34.195837 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:31:34.208231 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.012s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:31:34.208564 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[6c9d31ec-435d-4d64-9ac9-d3fa527fd21d]: (4, ('fa6a5dbb-8967-894e-8b8d-5384b73a89bf\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:31:34.209441 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] Running cmd (subprocess): nvme version {{(pid=135833) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:31:34.238720 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] CMD "nvme version" returned: 0 in 0.029s {{(pid=135833) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:31:34.246322 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=135833) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 28 21:31:34.246322 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=135833) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 28 21:31:34.246322 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:41d0083e-6728-42e9-88e9-cd92e21a8271 dsc: {{(pid=135833) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 28 21:31:34.246322 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.utils [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] <== get_connector_properties: return (116ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.33', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:a90a241cea8', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'fa6a5dbb-8967-894e-8b8d-5384b73a89bf', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:41d0083e-6728-42e9-88e9-cd92e21a8271', 'found_dsc': ''} {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:31:34.974438 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connector [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] Factory for rbd on None {{(pid=135833) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 28 21:31:34.976010 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-bac5fe86-e590-414b-bff3-906130d40c5b', 'hosts': ['10.222.0.33'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'bac5fe86-e590-414b-bff3-906130d40c5b', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:31:34.981311 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.linuxrbd [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] opening connection to ceph cluster (timeout=-1). {{(pid=135833) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 28 21:31:35.330130 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] <== connect_volume: return (352ms) {'path': } {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:31:35.335883 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:31:35.341191 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:31:35.343528 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_db.sqlalchemy.engines [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=135833) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 28 21:31:35.536056 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] Starting backup of volume='bac5fe86-e590-414b-bff3-906130d40c5b'. {{(pid=135833) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1028}} May 28 21:31:35.537717 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] Volume file is RBD: attempting optimized backup {{(pid=135833) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1039}} May 28 21:31:35.572334 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] Creating base image 'volume-bac5fe86-e590-414b-bff3-906130d40c5b.backup.7039d769-0e89-476f-a644-872f3ecd222f' {{(pid=135833) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:469}} May 28 21:31:35.577403 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=135833) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:277}} May 28 21:31:36.408909 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] Using --from-snap 'None' for incremental backup of volume bac5fe86-e590-414b-bff3-906130d40c5b. {{(pid=135833) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:811}} May 28 21:31:36.409546 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] Creating backup snapshot='backup.7039d769-0e89-476f-a644-872f3ecd222f.snap.1653773496.4089315' {{(pid=135833) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:817}} May 28 21:31:37.966288 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] Performing differential transfer from 'volume-bac5fe86-e590-414b-bff3-906130d40c5b' to 'volume-bac5fe86-e590-414b-bff3-906130d40c5b.backup.7039d769-0e89-476f-a644-872f3ecd222f' {{(pid=135833) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:658}} May 28 21:31:37.966955 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_hujy4esg --pool volumes volumes/volume-bac5fe86-e590-414b-bff3-906130d40c5b@backup.7039d769-0e89-476f-a644-872f3ecd222f.snap.1653773496.4089315 -' into... {{(pid=135833) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:615}} May 28 21:31:37.967229 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-bac5fe86-e590-414b-bff3-906130d40c5b.backup.7039d769-0e89-476f-a644-872f3ecd222f' {{(pid=135833) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:616}} May 28 21:31:38.407628 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] Restore transfer completed in 51.2410s {{(pid=135833) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1172}} May 28 21:31:38.408650 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=135833) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1360}} May 28 21:31:38.462329 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] Restoring volume base metadata {{(pid=135833) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} May 28 21:31:38.594138 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] No metadata of type 'volume-metadata' to restore {{(pid=135833) put /opt/stack/cinder/cinder/backup/driver.py:345}} May 28 21:31:38.594435 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] Restoring volume glance metadata {{(pid=135833) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}} May 28 21:31:39.933239 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] Restore to volume 6589398c-11d6-483a-b206-0202e0f288ce finished successfully. {{(pid=135833) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1367}} May 28 21:31:39.935456 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-6589398c-11d6-483a-b206-0202e0f288ce', 'hosts': ['10.222.0.33'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '6589398c-11d6-483a-b206-0202e0f288ce', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:31:40.355214 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] <== disconnect_volume: return (418ms) None {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:31:40.404332 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.manager [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] Encryption key ID for volume 6589398c-11d6-483a-b206-0202e0f288ce already matches encryption key ID in backup efc30e7d-29b7-4042-973b-169c70808f0d. {{(pid=135833) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} May 28 21:31:40.707227 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-62cfa036-8152-4889-89d6-bab314a69fc2 tempest-VolumesBackupsTest-1716294911 None] Finished restoring backup efc30e7d-29b7-4042-973b-169c70808f0d to volume 6589398c-11d6-483a-b206-0202e0f288ce. May 28 21:31:57.428041 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] Create backup started, backup: ab1c0376-6cb5-410e-a6a5-39226d83b255 volume: b31650fd-bab6-499a-bca7-af6b551f165e snapshot: c08d393b-3af5-4bf1-a0d6-9899b9419fa0. May 28 21:31:57.530477 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-28T21:31:56Z,data_timestamp=2022-05-28T21:31:50Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Backup-1184533321',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=ab1c0376-6cb5-410e-a6a5-39226d83b255,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='6ed82f578e214948a9311cbdbb55a027',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id='c08d393b-3af5-4bf1-a0d6-9899b9419fa0',status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='8d033e99f9294e348b01c59a970931d5',volume_id=b31650fd-bab6-499a-bca7-af6b551f165e) May 28 21:32:00.200326 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=47,cinder:UPDATE=6,cinder:INSERT=6 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:32:00.221924 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.utils [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.33', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:32:00.221924 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:32:00.234432 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.013s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:32:00.234760 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[75385552-0448-43a1-bac6-e0765ad516eb]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:a90a241cea8\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:32:00.236092 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:32:00.255571 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 28 21:32:00.255571 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[2b53308b-95db-44d6-bb32-458f7b8b6398]: [Errno 2] No such file or directory: 'systool' {{(pid=167755) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 28 21:32:00.255571 ncdv-master-1 cinder-backup[135833]: Traceback (most recent call last): May 28 21:32:00.255571 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 28 21:32:00.255571 ncdv-master-1 cinder-backup[135833]: ret = func(*f_args, **f_kwargs) May 28 21:32:00.255571 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 28 21:32:00.255571 ncdv-master-1 cinder-backup[135833]: return func(*args, **kwargs) May 28 21:32:00.255571 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 28 21:32:00.255571 ncdv-master-1 cinder-backup[135833]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 28 21:32:00.255571 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 28 21:32:00.255571 ncdv-master-1 cinder-backup[135833]: return putils.execute(on_execute=on_execute, May 28 21:32:00.255571 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 28 21:32:00.255571 ncdv-master-1 cinder-backup[135833]: obj = subprocess.Popen(cmd, May 28 21:32:00.255571 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 28 21:32:00.255571 ncdv-master-1 cinder-backup[135833]: self._execute_child(args, executable, preexec_fn, close_fds, May 28 21:32:00.255571 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 28 21:32:00.255571 ncdv-master-1 cinder-backup[135833]: raise child_exception_type(errno_num, err_msg, err_filename) May 28 21:32:00.255571 ncdv-master-1 cinder-backup[135833]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 28 21:32:00.255571 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[2b53308b-95db-44d6-bb32-458f7b8b6398]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:32:00.260314 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:32:00.267973 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 28 21:32:00.269129 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[dfa5506b-1dbf-420e-a426-162ed2a18610]: [Errno 2] No such file or directory: 'systool' {{(pid=167755) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 28 21:32:00.269129 ncdv-master-1 cinder-backup[135833]: Traceback (most recent call last): May 28 21:32:00.269129 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 28 21:32:00.269129 ncdv-master-1 cinder-backup[135833]: ret = func(*f_args, **f_kwargs) May 28 21:32:00.269129 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 28 21:32:00.269129 ncdv-master-1 cinder-backup[135833]: return func(*args, **kwargs) May 28 21:32:00.269129 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 28 21:32:00.269129 ncdv-master-1 cinder-backup[135833]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 28 21:32:00.269129 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 28 21:32:00.269129 ncdv-master-1 cinder-backup[135833]: return putils.execute(on_execute=on_execute, May 28 21:32:00.269129 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 28 21:32:00.269129 ncdv-master-1 cinder-backup[135833]: obj = subprocess.Popen(cmd, May 28 21:32:00.269129 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 28 21:32:00.269129 ncdv-master-1 cinder-backup[135833]: self._execute_child(args, executable, preexec_fn, close_fds, May 28 21:32:00.269129 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 28 21:32:00.269129 ncdv-master-1 cinder-backup[135833]: raise child_exception_type(errno_num, err_msg, err_filename) May 28 21:32:00.269129 ncdv-master-1 cinder-backup[135833]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 28 21:32:00.270630 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[dfa5506b-1dbf-420e-a426-162ed2a18610]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:32:00.274373 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:32:00.293382 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.020s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:32:00.293705 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[b3f7f2a4-d7d6-4340-b619-b9d5bb7856b1]: (4, ('/dev/sda1\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:32:00.296511 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:32:00.310045 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.014s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:32:00.310575 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[2e0d2559-2831-4c2c-bd96-1d790e75cc33]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:32:00.313736 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:32:00.334597 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.020s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:32:00.334847 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[7c04a2ed-4471-4092-859d-c9fc6e4b8d3a]: (4, ('fa6a5dbb-8967-894e-8b8d-5384b73a89bf\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:32:00.335781 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] Running cmd (subprocess): nvme version {{(pid=135833) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:32:00.361088 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] CMD "nvme version" returned: 0 in 0.025s {{(pid=135833) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:32:00.362866 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=135833) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 28 21:32:00.363235 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=135833) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 28 21:32:00.363593 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:41d0083e-6728-42e9-88e9-cd92e21a8271 dsc: {{(pid=135833) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 28 21:32:00.364217 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.utils [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] <== get_connector_properties: return (144ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.33', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:a90a241cea8', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'fa6a5dbb-8967-894e-8b8d-5384b73a89bf', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:41d0083e-6728-42e9-88e9-cd92e21a8271', 'found_dsc': ''} {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:32:01.082246 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connector [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] Factory for rbd on None {{(pid=135833) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 28 21:32:01.083864 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-efe7548f-0d36-4ab9-a529-aadaf622854f', 'hosts': ['10.222.0.33'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'efe7548f-0d36-4ab9-a529-aadaf622854f', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:32:01.084903 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.linuxrbd [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] opening connection to ceph cluster (timeout=-1). {{(pid=135833) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 28 21:32:01.534162 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] <== connect_volume: return (449ms) {'path': } {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:32:01.544716 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_db.sqlalchemy.engines [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=135833) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 28 21:32:01.621818 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] Starting backup of volume='b31650fd-bab6-499a-bca7-af6b551f165e'. {{(pid=135833) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1028}} May 28 21:32:01.652528 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=135833) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:277}} May 28 21:32:01.652946 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] Creating backup base image='volume-b31650fd-bab6-499a-bca7-af6b551f165e.backup.base' for volume b31650fd-bab6-499a-bca7-af6b551f165e. {{(pid=135833) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:880}} May 28 21:32:02.519460 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] Copying data from volume b31650fd-bab6-499a-bca7-af6b551f165e. {{(pid=135833) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:892}} May 28 21:32:02.643859 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] Transferring data between 'volume-b31650fd-bab6-499a-bca7-af6b551f165e' and 'volume-b31650fd-bab6-499a-bca7-af6b551f165e.backup.base' {{(pid=135833) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:421}} May 28 21:32:02.644399 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] 8 chunks of 134217728 bytes to be transferred {{(pid=135833) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:425}} May 28 21:32:03.609667 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-f54df77b-5b12-4e78-b204-88cc02b6ab65 tempest-VolumesBackupsTest-1716294911 None] Delete backup started, backup: efc30e7d-29b7-4042-973b-169c70808f0d. May 28 21:32:03.611336 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-f54df77b-5b12-4e78-b204-88cc02b6ab65 tempest-VolumesBackupsTest-1716294911 None] Delete started for backup=efc30e7d-29b7-4042-973b-169c70808f0d {{(pid=135833) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1376}} May 28 21:32:03.611937 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-f54df77b-5b12-4e78-b204-88cc02b6ab65 tempest-VolumesBackupsTest-1716294911 None] Trying diff format basename='volume-dd38b1bb-0ed4-4cb0-b594-5f3aa84501bd.backup.efc30e7d-29b7-4042-973b-169c70808f0d' for backup base image of volume dd38b1bb-0ed4-4cb0-b594-5f3aa84501bd. {{(pid=135833) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:543}} May 28 21:32:03.982095 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-f54df77b-5b12-4e78-b204-88cc02b6ab65 tempest-VolumesBackupsTest-1716294911 None] Looking for snapshot of backup base 'volume-dd38b1bb-0ed4-4cb0-b594-5f3aa84501bd.backup.efc30e7d-29b7-4042-973b-169c70808f0d' {{(pid=135833) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:957}} May 28 21:32:03.982740 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-f54df77b-5b12-4e78-b204-88cc02b6ab65 tempest-VolumesBackupsTest-1716294911 None] Found snapshot 'backup.efc30e7d-29b7-4042-973b-169c70808f0d.snap.1653773393.2418056' {{(pid=135833) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:976}} May 28 21:32:03.983118 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-f54df77b-5b12-4e78-b204-88cc02b6ab65 tempest-VolumesBackupsTest-1716294911 None] Deleting backup snapshot='backup.efc30e7d-29b7-4042-973b-169c70808f0d.snap.1653773393.2418056' {{(pid=135833) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:501}} May 28 21:32:07.251635 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.drivers.ceph [None req-f54df77b-5b12-4e78-b204-88cc02b6ab65 tempest-VolumesBackupsTest-1716294911 None] Deleting backup base image='volume-dd38b1bb-0ed4-4cb0-b594-5f3aa84501bd.backup.efc30e7d-29b7-4042-973b-169c70808f0d' of volume dd38b1bb-0ed4-4cb0-b594-5f3aa84501bd. May 28 21:32:08.506757 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_db.sqlalchemy.engines [None req-1e29fabe-dc57-4ef4-ad80-ba98d322b6cd None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=135833) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 28 21:32:08.514071 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=45,cinder:UPDATE=5,cinder:INSERT=7 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:32:12.633074 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] Transferred chunk 1 of 8 (13122K/s) {{(pid=135833) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} May 28 21:32:18.522451 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:32:22.604737 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] Transferred chunk 2 of 8 (13145K/s) {{(pid=135833) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} May 28 21:32:22.714631 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-f54df77b-5b12-4e78-b204-88cc02b6ab65 tempest-VolumesBackupsTest-1716294911 None] Base backup image='volume-dd38b1bb-0ed4-4cb0-b594-5f3aa84501bd.backup.efc30e7d-29b7-4042-973b-169c70808f0d' of volume dd38b1bb-0ed4-4cb0-b594-5f3aa84501bd deleted. {{(pid=135833) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:591}} May 28 21:32:22.871828 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-f54df77b-5b12-4e78-b204-88cc02b6ab65 tempest-VolumesBackupsTest-1716294911 None] Delete of backup 'efc30e7d-29b7-4042-973b-169c70808f0d' for volume 'dd38b1bb-0ed4-4cb0-b594-5f3aa84501bd' finished. {{(pid=135833) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1404}} May 28 21:32:22.876524 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] Differential backup transfer completed in 44.9104s {{(pid=135833) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:837}} May 28 21:32:22.877752 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] Getting metadata type 'volume-base-metadata' {{(pid=135833) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 28 21:32:22.899762 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=135833) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 28 21:32:22.900436 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] Getting metadata type 'volume-metadata' {{(pid=135833) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 28 21:32:22.914289 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] No metadata type 'volume-metadata' available {{(pid=135833) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 28 21:32:22.914929 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] Getting metadata type 'volume-glance-metadata' {{(pid=135833) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 28 21:32:22.924267 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=135833) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}} May 28 21:32:22.924976 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] Backing up metadata for volume bac5fe86-e590-414b-bff3-906130d40c5b. {{(pid=135833) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:1003}} May 28 21:32:23.049042 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.quota [None req-f54df77b-5b12-4e78-b204-88cc02b6ab65 tempest-VolumesBackupsTest-1716294911 None] Created reservations ['872b8992-c2e6-4170-a953-42a2f03d8256', 'ef72a995-1d6d-4662-81ed-b073a78fafba'] {{(pid=135833) reserve /opt/stack/cinder/cinder/quota.py:755}} May 28 21:32:23.121459 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] Backup '7039d769-0e89-476f-a644-872f3ecd222f' of volume bac5fe86-e590-414b-bff3-906130d40c5b finished. {{(pid=135833) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1069}} May 28 21:32:23.123018 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-bac5fe86-e590-414b-bff3-906130d40c5b', 'hosts': ['10.222.0.33'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'bac5fe86-e590-414b-bff3-906130d40c5b', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:32:23.745104 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] <== disconnect_volume: return (620ms) None {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:32:23.836554 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-f54df77b-5b12-4e78-b204-88cc02b6ab65 tempest-VolumesBackupsTest-1716294911 None] Delete backup finished, backup efc30e7d-29b7-4042-973b-169c70808f0d deleted. May 28 21:32:26.798282 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-29aaa688-ade9-4bcb-96f9-0bbe475ff858 tempest-TestVolumeBackupRestore-1923413318 None] Create backup finished. backup: 7039d769-0e89-476f-a644-872f3ecd222f. May 28 21:32:30.414210 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] Restore backup started, backup: 7039d769-0e89-476f-a644-872f3ecd222f volume: 66ff15ab-0cf7-464f-a399-0ea2ae0a2aa0. May 28 21:32:30.424925 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.utils [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.33', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:32:30.430221 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:32:30.442163 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.009s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:32:30.442163 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[392b0a84-37f9-4fb6-8267-e3669ed0c046]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:a90a241cea8\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:32:30.442163 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:32:30.444069 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 28 21:32:30.444689 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[acf073b6-f7bb-477b-bcfd-1e0b3fb2e8a2]: [Errno 2] No such file or directory: 'systool' {{(pid=167755) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 28 21:32:30.444689 ncdv-master-1 cinder-backup[135833]: Traceback (most recent call last): May 28 21:32:30.444689 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 28 21:32:30.444689 ncdv-master-1 cinder-backup[135833]: ret = func(*f_args, **f_kwargs) May 28 21:32:30.444689 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 28 21:32:30.444689 ncdv-master-1 cinder-backup[135833]: return func(*args, **kwargs) May 28 21:32:30.444689 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 28 21:32:30.444689 ncdv-master-1 cinder-backup[135833]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 28 21:32:30.444689 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 28 21:32:30.444689 ncdv-master-1 cinder-backup[135833]: return putils.execute(on_execute=on_execute, May 28 21:32:30.444689 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 28 21:32:30.444689 ncdv-master-1 cinder-backup[135833]: obj = subprocess.Popen(cmd, May 28 21:32:30.444689 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 28 21:32:30.444689 ncdv-master-1 cinder-backup[135833]: self._execute_child(args, executable, preexec_fn, close_fds, May 28 21:32:30.444689 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 28 21:32:30.444689 ncdv-master-1 cinder-backup[135833]: raise child_exception_type(errno_num, err_msg, err_filename) May 28 21:32:30.444689 ncdv-master-1 cinder-backup[135833]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 28 21:32:30.445660 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[acf073b6-f7bb-477b-bcfd-1e0b3fb2e8a2]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:32:30.446131 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:32:30.452265 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 28 21:32:30.452918 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[69497856-8f00-4578-bcc5-0ef7ba92def4]: [Errno 2] No such file or directory: 'systool' {{(pid=167755) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 28 21:32:30.452918 ncdv-master-1 cinder-backup[135833]: Traceback (most recent call last): May 28 21:32:30.452918 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 28 21:32:30.452918 ncdv-master-1 cinder-backup[135833]: ret = func(*f_args, **f_kwargs) May 28 21:32:30.452918 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 28 21:32:30.452918 ncdv-master-1 cinder-backup[135833]: return func(*args, **kwargs) May 28 21:32:30.452918 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 28 21:32:30.452918 ncdv-master-1 cinder-backup[135833]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 28 21:32:30.452918 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 28 21:32:30.452918 ncdv-master-1 cinder-backup[135833]: return putils.execute(on_execute=on_execute, May 28 21:32:30.452918 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 28 21:32:30.452918 ncdv-master-1 cinder-backup[135833]: obj = subprocess.Popen(cmd, May 28 21:32:30.452918 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 28 21:32:30.452918 ncdv-master-1 cinder-backup[135833]: self._execute_child(args, executable, preexec_fn, close_fds, May 28 21:32:30.452918 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 28 21:32:30.452918 ncdv-master-1 cinder-backup[135833]: raise child_exception_type(errno_num, err_msg, err_filename) May 28 21:32:30.452918 ncdv-master-1 cinder-backup[135833]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 28 21:32:30.454017 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[69497856-8f00-4578-bcc5-0ef7ba92def4]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:32:30.455322 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:32:30.486354 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.023s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:32:30.486354 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[b71945a4-e66e-4d62-b302-6d4dbc5adb62]: (4, ('/dev/sda1\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:32:30.486354 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:32:30.491036 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.010s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:32:30.491370 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[e02513a2-bab6-48f4-b254-8b51b0dfc357]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:32:30.492698 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:32:30.504578 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.012s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:32:30.505352 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[ce068de4-1e49-4175-838f-be451e1d33bd]: (4, ('fa6a5dbb-8967-894e-8b8d-5384b73a89bf\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:32:30.506153 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] Running cmd (subprocess): nvme version {{(pid=135833) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:32:30.549669 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] CMD "nvme version" returned: 0 in 0.043s {{(pid=135833) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:32:30.551518 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=135833) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 28 21:32:30.551892 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=135833) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 28 21:32:30.552254 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:41d0083e-6728-42e9-88e9-cd92e21a8271 dsc: {{(pid=135833) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 28 21:32:30.553158 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.utils [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] <== get_connector_properties: return (127ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.33', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:a90a241cea8', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'fa6a5dbb-8967-894e-8b8d-5384b73a89bf', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:41d0083e-6728-42e9-88e9-cd92e21a8271', 'found_dsc': ''} {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:32:30.607456 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] Transferred chunk 3 of 8 (16379K/s) {{(pid=135833) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} May 28 21:32:31.325961 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connector [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] Factory for rbd on None {{(pid=135833) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 28 21:32:31.327795 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-66ff15ab-0cf7-464f-a399-0ea2ae0a2aa0', 'hosts': ['10.222.0.33'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '66ff15ab-0cf7-464f-a399-0ea2ae0a2aa0', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:32:31.328977 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.linuxrbd [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] opening connection to ceph cluster (timeout=-1). {{(pid=135833) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 28 21:32:31.705911 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] <== connect_volume: return (377ms) {'path': } {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:32:31.715039 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_db.sqlalchemy.engines [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=135833) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 28 21:32:31.728301 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] Starting restore from Ceph backup=7039d769-0e89-476f-a644-872f3ecd222f to volume=volume-66ff15ab-0cf7-464f-a399-0ea2ae0a2aa0 {{(pid=135833) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1349}} May 28 21:32:31.784887 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] Looking for snapshot of backup base 'volume-bac5fe86-e590-414b-bff3-906130d40c5b.backup.7039d769-0e89-476f-a644-872f3ecd222f' {{(pid=135833) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:957}} May 28 21:32:31.785257 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] Found snapshot 'backup.7039d769-0e89-476f-a644-872f3ecd222f.snap.1653773496.4089315' {{(pid=135833) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:976}} May 28 21:32:31.789669 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] Volume file is RBD. {{(pid=135833) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1251}} May 28 21:32:31.825233 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] Attempting differential restore. {{(pid=135833) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1304}} May 28 21:32:31.825729 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] Attempting incremental restore from base='volume-bac5fe86-e590-414b-bff3-906130d40c5b.backup.7039d769-0e89-476f-a644-872f3ecd222f' snap='backup.7039d769-0e89-476f-a644-872f3ecd222f.snap.1653773496.4089315' {{(pid=135833) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1150}} May 28 21:32:31.826332 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] Performing differential transfer from 'volume-bac5fe86-e590-414b-bff3-906130d40c5b.backup.7039d769-0e89-476f-a644-872f3ecd222f' to 'volume-66ff15ab-0cf7-464f-a399-0ea2ae0a2aa0' {{(pid=135833) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:658}} May 28 21:32:31.826747 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-bac5fe86-e590-414b-bff3-906130d40c5b.backup.7039d769-0e89-476f-a644-872f3ecd222f@backup.7039d769-0e89-476f-a644-872f3ecd222f.snap.1653773496.4089315 -' into... {{(pid=135833) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:615}} May 28 21:32:31.827205 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_dyp9v8_a --pool volumes - volumes/volume-66ff15ab-0cf7-464f-a399-0ea2ae0a2aa0' {{(pid=135833) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:616}} May 28 21:32:38.767393 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] Transferred chunk 4 of 8 (16065K/s) {{(pid=135833) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} May 28 21:32:48.511468 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=27,cinder:UPDATE=6,cinder:INSERT=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:32:57.140381 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] Transferred chunk 5 of 8 (7134K/s) {{(pid=135833) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} May 28 21:32:58.518177 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=16,cinder:INSERT=1,cinder:UPDATE=6 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:33:17.172374 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] Transferred chunk 6 of 8 (6543K/s) {{(pid=135833) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} May 28 21:33:18.522139 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:33:35.342342 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:33:35.343038 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:33:38.521806 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:33:38.523834 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:33:39.348868 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] Transferred chunk 7 of 8 (5910K/s) {{(pid=135833) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} May 28 21:33:51.506307 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] Restore transfer completed in 79.6798s {{(pid=135833) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1172}} May 28 21:33:51.507339 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=135833) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1360}} May 28 21:33:51.821165 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] Restoring volume base metadata {{(pid=135833) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} May 28 21:33:51.939148 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] No metadata of type 'volume-metadata' to restore {{(pid=135833) put /opt/stack/cinder/cinder/backup/driver.py:345}} May 28 21:33:51.939523 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] Restoring volume glance metadata {{(pid=135833) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}} May 28 21:33:52.586314 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] Transferred chunk 8 of 8 (9902K/s) {{(pid=135833) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:444}} May 28 21:33:53.024536 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] Getting metadata type 'volume-base-metadata' {{(pid=135833) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 28 21:33:53.039914 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=135833) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 28 21:33:53.040378 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] Getting metadata type 'volume-metadata' {{(pid=135833) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 28 21:33:53.051191 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] No metadata type 'volume-metadata' available {{(pid=135833) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 28 21:33:53.054103 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] Getting metadata type 'volume-glance-metadata' {{(pid=135833) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 28 21:33:53.058877 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] No metadata type 'volume-glance-metadata' available {{(pid=135833) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} May 28 21:33:53.059537 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] Backing up metadata for volume b31650fd-bab6-499a-bca7-af6b551f165e. {{(pid=135833) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:1003}} May 28 21:33:53.114554 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] Restore to volume 66ff15ab-0cf7-464f-a399-0ea2ae0a2aa0 finished successfully. {{(pid=135833) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1367}} May 28 21:33:53.116203 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-66ff15ab-0cf7-464f-a399-0ea2ae0a2aa0', 'hosts': ['10.222.0.33'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '66ff15ab-0cf7-464f-a399-0ea2ae0a2aa0', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:33:53.258618 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] <== disconnect_volume: return (141ms) None {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:33:53.271091 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] Backup 'ab1c0376-6cb5-410e-a6a5-39226d83b255' of volume b31650fd-bab6-499a-bca7-af6b551f165e finished. {{(pid=135833) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1069}} May 28 21:33:53.272785 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-efe7548f-0d36-4ab9-a529-aadaf622854f', 'hosts': ['10.222.0.33'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'efe7548f-0d36-4ab9-a529-aadaf622854f', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:33:53.396270 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] <== disconnect_volume: return (122ms) None {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:33:53.451948 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.manager [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] Encryption key ID for volume 66ff15ab-0cf7-464f-a399-0ea2ae0a2aa0 already matches encryption key ID in backup 7039d769-0e89-476f-a644-872f3ecd222f. {{(pid=135833) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} May 28 21:33:53.763290 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-1497bf42-a2f5-4e0c-ad17-e1919a913240 tempest-VolumesSnapshotTestJSON-961770574 None] Create backup finished. backup: ab1c0376-6cb5-410e-a6a5-39226d83b255. May 28 21:33:53.786013 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-a04865ad-c8ba-46f6-9f7e-2f4250d4cbc5 tempest-TestVolumeBackupRestore-1923413318 None] Finished restoring backup 7039d769-0e89-476f-a644-872f3ecd222f to volume 66ff15ab-0cf7-464f-a399-0ea2ae0a2aa0. May 28 21:33:54.756623 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-bbc33bf8-caf9-4e0d-ae7e-6ed19da1da04 tempest-VolumesSnapshotTestJSON-961770574 None] Delete backup started, backup: ab1c0376-6cb5-410e-a6a5-39226d83b255. May 28 21:33:54.756623 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-bbc33bf8-caf9-4e0d-ae7e-6ed19da1da04 tempest-VolumesSnapshotTestJSON-961770574 None] Delete started for backup=ab1c0376-6cb5-410e-a6a5-39226d83b255 {{(pid=135833) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1376}} May 28 21:33:54.756623 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-bbc33bf8-caf9-4e0d-ae7e-6ed19da1da04 tempest-VolumesSnapshotTestJSON-961770574 None] Trying diff format basename='volume-b31650fd-bab6-499a-bca7-af6b551f165e.backup.ab1c0376-6cb5-410e-a6a5-39226d83b255' for backup base image of volume b31650fd-bab6-499a-bca7-af6b551f165e. {{(pid=135833) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:543}} May 28 21:33:54.785814 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-bbc33bf8-caf9-4e0d-ae7e-6ed19da1da04 tempest-VolumesSnapshotTestJSON-961770574 None] Image 'volume-b31650fd-bab6-499a-bca7-af6b551f165e.backup.ab1c0376-6cb5-410e-a6a5-39226d83b255' not found - trying diff format name {{(pid=135833) _rbd_image_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:697}} May 28 21:33:55.021211 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-bbc33bf8-caf9-4e0d-ae7e-6ed19da1da04 tempest-VolumesSnapshotTestJSON-961770574 None] Looking for snapshot of backup base 'volume-b31650fd-bab6-499a-bca7-af6b551f165e.backup.base' {{(pid=135833) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:957}} May 28 21:33:55.021655 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-bbc33bf8-caf9-4e0d-ae7e-6ed19da1da04 tempest-VolumesSnapshotTestJSON-961770574 None] Backup base 'volume-b31650fd-bab6-499a-bca7-af6b551f165e.backup.base' has no snapshots {{(pid=135833) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:960}} May 28 21:33:55.022054 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-bbc33bf8-caf9-4e0d-ae7e-6ed19da1da04 tempest-VolumesSnapshotTestJSON-961770574 None] No backup snapshot to delete {{(pid=135833) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:504}} May 28 21:33:55.203751 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.drivers.ceph [None req-bbc33bf8-caf9-4e0d-ae7e-6ed19da1da04 tempest-VolumesSnapshotTestJSON-961770574 None] Deleting backup base image='volume-b31650fd-bab6-499a-bca7-af6b551f165e.backup.base' of volume b31650fd-bab6-499a-bca7-af6b551f165e. May 28 21:33:58.529127 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_db.sqlalchemy.engines [None req-1e29fabe-dc57-4ef4-ad80-ba98d322b6cd None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=135833) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 28 21:34:03.714843 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=48,cinder:INSERT=4,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:34:08.042775 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-bbc33bf8-caf9-4e0d-ae7e-6ed19da1da04 tempest-VolumesSnapshotTestJSON-961770574 None] Base backup image='volume-b31650fd-bab6-499a-bca7-af6b551f165e.backup.base' of volume b31650fd-bab6-499a-bca7-af6b551f165e deleted. {{(pid=135833) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:591}} May 28 21:34:08.219694 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-bbc33bf8-caf9-4e0d-ae7e-6ed19da1da04 tempest-VolumesSnapshotTestJSON-961770574 None] Delete of backup 'ab1c0376-6cb5-410e-a6a5-39226d83b255' for volume 'b31650fd-bab6-499a-bca7-af6b551f165e' finished. {{(pid=135833) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1404}} May 28 21:34:08.357244 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.quota [None req-bbc33bf8-caf9-4e0d-ae7e-6ed19da1da04 tempest-VolumesSnapshotTestJSON-961770574 None] Created reservations ['8d37a6e4-95f3-403d-854a-b5ce9504fecd', 'e6ebb456-b746-484f-a019-7ee77c075268'] {{(pid=135833) reserve /opt/stack/cinder/cinder/quota.py:755}} May 28 21:34:08.609189 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-bbc33bf8-caf9-4e0d-ae7e-6ed19da1da04 tempest-VolumesSnapshotTestJSON-961770574 None] Delete backup finished, backup ab1c0376-6cb5-410e-a6a5-39226d83b255 deleted. May 28 21:34:18.533647 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=51,cinder:UPDATE=11,cinder:INSERT=11 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:34:28.540908 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=10,cinder:UPDATE=5 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:34:35.338540 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:34:35.339202 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:34:58.520098 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:35:08.527694 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:35:18.538007 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:35:35.335514 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:35:35.336258 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:35:38.529738 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:35:58.527174 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:36:08.528537 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:36:28.404879 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-b0e6f295-0bce-4761-bda2-856e248597af tempest-TestVolumeBackupRestore-1923413318 None] Delete backup started, backup: 7039d769-0e89-476f-a644-872f3ecd222f. May 28 21:36:28.407398 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-b0e6f295-0bce-4761-bda2-856e248597af tempest-TestVolumeBackupRestore-1923413318 None] Delete started for backup=7039d769-0e89-476f-a644-872f3ecd222f {{(pid=135833) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1376}} May 28 21:36:28.407922 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-b0e6f295-0bce-4761-bda2-856e248597af tempest-TestVolumeBackupRestore-1923413318 None] Trying diff format basename='volume-bac5fe86-e590-414b-bff3-906130d40c5b.backup.7039d769-0e89-476f-a644-872f3ecd222f' for backup base image of volume bac5fe86-e590-414b-bff3-906130d40c5b. {{(pid=135833) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:543}} May 28 21:36:28.638837 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:36:28.640111 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:36:28.644730 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-b0e6f295-0bce-4761-bda2-856e248597af tempest-TestVolumeBackupRestore-1923413318 None] Looking for snapshot of backup base 'volume-bac5fe86-e590-414b-bff3-906130d40c5b.backup.7039d769-0e89-476f-a644-872f3ecd222f' {{(pid=135833) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:957}} May 28 21:36:28.645269 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-b0e6f295-0bce-4761-bda2-856e248597af tempest-TestVolumeBackupRestore-1923413318 None] Found snapshot 'backup.7039d769-0e89-476f-a644-872f3ecd222f.snap.1653773496.4089315' {{(pid=135833) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:976}} May 28 21:36:28.645646 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-b0e6f295-0bce-4761-bda2-856e248597af tempest-TestVolumeBackupRestore-1923413318 None] Deleting backup snapshot='backup.7039d769-0e89-476f-a644-872f3ecd222f.snap.1653773496.4089315' {{(pid=135833) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:501}} May 28 21:36:28.657888 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_db.sqlalchemy.engines [None req-1e29fabe-dc57-4ef4-ad80-ba98d322b6cd None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=135833) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 28 21:36:31.237813 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.drivers.ceph [None req-b0e6f295-0bce-4761-bda2-856e248597af tempest-TestVolumeBackupRestore-1923413318 None] Deleting backup base image='volume-bac5fe86-e590-414b-bff3-906130d40c5b.backup.7039d769-0e89-476f-a644-872f3ecd222f' of volume bac5fe86-e590-414b-bff3-906130d40c5b. May 28 21:36:35.331977 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:36:35.332584 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:36:39.420795 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-b0e6f295-0bce-4761-bda2-856e248597af tempest-TestVolumeBackupRestore-1923413318 None] Base backup image='volume-bac5fe86-e590-414b-bff3-906130d40c5b.backup.7039d769-0e89-476f-a644-872f3ecd222f' of volume bac5fe86-e590-414b-bff3-906130d40c5b deleted. {{(pid=135833) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:591}} May 28 21:36:39.601676 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-b0e6f295-0bce-4761-bda2-856e248597af tempest-TestVolumeBackupRestore-1923413318 None] Delete of backup '7039d769-0e89-476f-a644-872f3ecd222f' for volume 'bac5fe86-e590-414b-bff3-906130d40c5b' finished. {{(pid=135833) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1404}} May 28 21:36:39.813697 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.quota [None req-b0e6f295-0bce-4761-bda2-856e248597af tempest-TestVolumeBackupRestore-1923413318 None] Created reservations ['0be9296f-38c0-4f7f-b402-fa5afe390ed3', '76923fee-7ae0-42d1-9b27-480503c3be9b'] {{(pid=135833) reserve /opt/stack/cinder/cinder/quota.py:755}} May 28 21:36:40.026787 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-b0e6f295-0bce-4761-bda2-856e248597af tempest-TestVolumeBackupRestore-1923413318 None] Delete backup finished, backup 7039d769-0e89-476f-a644-872f3ecd222f deleted. May 28 21:36:58.638902 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3,cinder:INSERT=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:37:08.641314 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=19,cinder:UPDATE=7,cinder:INSERT=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:37:35.336573 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:37:35.337142 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:37:38.637585 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:37:38.638249 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:38:18.645038 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=10,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:38:28.659014 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:38:35.332522 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:38:35.333134 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:38:38.665629 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:38:58.654353 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:39:08.655734 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:39:28.656728 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:39:35.337820 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:39:35.338406 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:39:38.648909 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:39:58.654485 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=3,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:40:18.652563 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:40:35.343001 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:40:35.348880 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:40:38.665984 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:40:48.667451 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:41:08.663010 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:41:28.658727 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=9,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:41:35.339248 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:41:35.347160 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:41:48.656947 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:41:58.670762 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:42:18.658475 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:42:28.676691 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:42:35.334673 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:42:35.335293 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:42:38.140450 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] Create backup started, backup: 37375bd5-f706-494e-8ceb-f6466da4fac9 volume: 4bdda978-6c53-441e-936a-9adb0994478b. May 28 21:42:38.221447 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-28T21:42:37Z,data_timestamp=2022-05-28T21:42:37Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsAdminTest-Backup-1613682014',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=37375bd5-f706-494e-8ceb-f6466da4fac9,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='201bd6cbfd144d5c9ff30c81341773c6',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='a69e3e8e9a7c4eda80a8c110a62511e2',volume_id=4bdda978-6c53-441e-936a-9adb0994478b) May 28 21:42:38.262705 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.utils [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.33', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:42:38.264075 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:42:38.272078 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.008s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:42:38.272472 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[5189c698-e1ee-4212-b904-ff1e99395dd5]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:a90a241cea8\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:42:38.273670 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:42:38.282112 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 28 21:42:38.282112 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[75f52e6d-9f0a-4fd0-a13b-58255bd06513]: [Errno 2] No such file or directory: 'systool' {{(pid=167755) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 28 21:42:38.282112 ncdv-master-1 cinder-backup[135833]: Traceback (most recent call last): May 28 21:42:38.282112 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 28 21:42:38.282112 ncdv-master-1 cinder-backup[135833]: ret = func(*f_args, **f_kwargs) May 28 21:42:38.282112 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 28 21:42:38.282112 ncdv-master-1 cinder-backup[135833]: return func(*args, **kwargs) May 28 21:42:38.282112 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 28 21:42:38.282112 ncdv-master-1 cinder-backup[135833]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 28 21:42:38.282112 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 28 21:42:38.282112 ncdv-master-1 cinder-backup[135833]: return putils.execute(on_execute=on_execute, May 28 21:42:38.282112 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 28 21:42:38.282112 ncdv-master-1 cinder-backup[135833]: obj = subprocess.Popen(cmd, May 28 21:42:38.282112 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 28 21:42:38.282112 ncdv-master-1 cinder-backup[135833]: self._execute_child(args, executable, preexec_fn, close_fds, May 28 21:42:38.282112 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 28 21:42:38.282112 ncdv-master-1 cinder-backup[135833]: raise child_exception_type(errno_num, err_msg, err_filename) May 28 21:42:38.282112 ncdv-master-1 cinder-backup[135833]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 28 21:42:38.282112 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[75f52e6d-9f0a-4fd0-a13b-58255bd06513]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:42:38.283797 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:42:38.288416 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 28 21:42:38.289112 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[801f02f4-e50b-419a-a60b-cf193c5a0bb8]: [Errno 2] No such file or directory: 'systool' {{(pid=167755) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 28 21:42:38.289112 ncdv-master-1 cinder-backup[135833]: Traceback (most recent call last): May 28 21:42:38.289112 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 28 21:42:38.289112 ncdv-master-1 cinder-backup[135833]: ret = func(*f_args, **f_kwargs) May 28 21:42:38.289112 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 28 21:42:38.289112 ncdv-master-1 cinder-backup[135833]: return func(*args, **kwargs) May 28 21:42:38.289112 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 28 21:42:38.289112 ncdv-master-1 cinder-backup[135833]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 28 21:42:38.289112 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 28 21:42:38.289112 ncdv-master-1 cinder-backup[135833]: return putils.execute(on_execute=on_execute, May 28 21:42:38.289112 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 28 21:42:38.289112 ncdv-master-1 cinder-backup[135833]: obj = subprocess.Popen(cmd, May 28 21:42:38.289112 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 28 21:42:38.289112 ncdv-master-1 cinder-backup[135833]: self._execute_child(args, executable, preexec_fn, close_fds, May 28 21:42:38.289112 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 28 21:42:38.289112 ncdv-master-1 cinder-backup[135833]: raise child_exception_type(errno_num, err_msg, err_filename) May 28 21:42:38.289112 ncdv-master-1 cinder-backup[135833]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 28 21:42:38.290361 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[801f02f4-e50b-419a-a60b-cf193c5a0bb8]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:42:38.290794 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:42:38.300314 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.009s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:42:38.305090 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[0376f4d7-57ee-4178-9d1b-1af3550cf9b2]: (4, ('/dev/sda1\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:42:38.305090 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:42:38.311745 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.009s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:42:38.312085 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[a35fae24-b1fd-483e-b49b-1cc9fad28d53]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:42:38.313140 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:42:38.319840 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.007s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:42:38.320156 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[d2ee40ea-3555-4c1b-b988-612e22ad1289]: (4, ('fa6a5dbb-8967-894e-8b8d-5384b73a89bf\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:42:38.320894 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] Running cmd (subprocess): nvme version {{(pid=135833) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:42:38.345536 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] CMD "nvme version" returned: 0 in 0.024s {{(pid=135833) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:42:38.347546 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=135833) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 28 21:42:38.347922 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=135833) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 28 21:42:38.348305 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:41d0083e-6728-42e9-88e9-cd92e21a8271 dsc: {{(pid=135833) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 28 21:42:38.348940 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.utils [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] <== get_connector_properties: return (85ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.33', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:a90a241cea8', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'fa6a5dbb-8967-894e-8b8d-5384b73a89bf', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:41d0083e-6728-42e9-88e9-cd92e21a8271', 'found_dsc': ''} {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:42:38.664014 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_db.sqlalchemy.engines [None req-1e29fabe-dc57-4ef4-ad80-ba98d322b6cd None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=135833) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 28 21:42:39.005982 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connector [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] Factory for rbd on None {{(pid=135833) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 28 21:42:39.007586 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-4bdda978-6c53-441e-936a-9adb0994478b', 'hosts': ['10.222.0.33'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '4bdda978-6c53-441e-936a-9adb0994478b', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:42:39.008720 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.linuxrbd [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] opening connection to ceph cluster (timeout=-1). {{(pid=135833) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 28 21:42:39.232605 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] <== connect_volume: return (224ms) {'path': } {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:42:39.329680 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] Starting backup of volume='4bdda978-6c53-441e-936a-9adb0994478b'. {{(pid=135833) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1028}} May 28 21:42:39.330696 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] Volume file is RBD: attempting optimized backup {{(pid=135833) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1039}} May 28 21:42:39.351673 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] Creating base image 'volume-4bdda978-6c53-441e-936a-9adb0994478b.backup.37375bd5-f706-494e-8ceb-f6466da4fac9' {{(pid=135833) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:469}} May 28 21:42:39.352248 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=135833) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:277}} May 28 21:42:40.011053 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] Using --from-snap 'None' for incremental backup of volume 4bdda978-6c53-441e-936a-9adb0994478b. {{(pid=135833) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:811}} May 28 21:42:40.011965 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] Creating backup snapshot='backup.37375bd5-f706-494e-8ceb-f6466da4fac9.snap.1653774160.011077' {{(pid=135833) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:817}} May 28 21:42:42.240157 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] Performing differential transfer from 'volume-4bdda978-6c53-441e-936a-9adb0994478b' to 'volume-4bdda978-6c53-441e-936a-9adb0994478b.backup.37375bd5-f706-494e-8ceb-f6466da4fac9' {{(pid=135833) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:658}} May 28 21:42:42.240796 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_04bx9816 --pool volumes volumes/volume-4bdda978-6c53-441e-936a-9adb0994478b@backup.37375bd5-f706-494e-8ceb-f6466da4fac9.snap.1653774160.011077 -' into... {{(pid=135833) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:615}} May 28 21:42:42.240933 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-4bdda978-6c53-441e-936a-9adb0994478b.backup.37375bd5-f706-494e-8ceb-f6466da4fac9' {{(pid=135833) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:616}} May 28 21:42:45.553097 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] Differential backup transfer completed in 3.3126s {{(pid=135833) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:837}} May 28 21:42:45.553631 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] Getting metadata type 'volume-base-metadata' {{(pid=135833) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 28 21:42:45.568959 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=135833) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 28 21:42:45.569366 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] Getting metadata type 'volume-metadata' {{(pid=135833) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 28 21:42:45.576428 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] No metadata type 'volume-metadata' available {{(pid=135833) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 28 21:42:45.576754 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] Getting metadata type 'volume-glance-metadata' {{(pid=135833) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 28 21:42:45.583016 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] No metadata type 'volume-glance-metadata' available {{(pid=135833) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} May 28 21:42:45.585393 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] Backing up metadata for volume 4bdda978-6c53-441e-936a-9adb0994478b. {{(pid=135833) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:1003}} May 28 21:42:45.732774 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] Backup '37375bd5-f706-494e-8ceb-f6466da4fac9' of volume 4bdda978-6c53-441e-936a-9adb0994478b finished. {{(pid=135833) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1069}} May 28 21:42:45.734599 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-4bdda978-6c53-441e-936a-9adb0994478b', 'hosts': ['10.222.0.33'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '4bdda978-6c53-441e-936a-9adb0994478b', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:42:46.318001 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] <== disconnect_volume: return (582ms) None {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:42:46.499350 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-dbdac7db-da12-4977-8be1-76258d857134 tempest-VolumesBackupsAdminTest-1665785387 None] Create backup finished. backup: 37375bd5-f706-494e-8ceb-f6466da4fac9. May 28 21:42:47.656346 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-7d620dba-e76f-4c4d-975e-e827f8b7000a tempest-VolumesBackupsAdminTest-1214246761 None] Export record started, backup: 37375bd5-f706-494e-8ceb-f6466da4fac9. May 28 21:42:47.658310 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-7d620dba-e76f-4c4d-975e-e827f8b7000a tempest-VolumesBackupsAdminTest-1214246761 None] Export record finished, backup 37375bd5-f706-494e-8ceb-f6466da4fac9 exported. May 28 21:42:48.186539 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-5720ce1e-e1fe-40a4-ad73-dc7da4f5644b tempest-VolumesBackupsAdminTest-1214246761 None] Import record started, backup_url: eyJkcml2ZXJfaW5mbyI6IHt9LCAiaWQiOiAiMzU2ZGExOWQtNzVmNS00ZjY0LWI2YTQtY2ZlM2QwZWRmYjJmIiwgInVzZXJfaWQiOiAiYTY5ZTNlOGU5YTdjNGVkYTgwYThjMTEwYTYyNTExZTIiLCAicHJvamVjdF9pZCI6ICIyMDFiZDZjYmZkMTQ0ZDVjOWZmMzBjODEzNDE3NzNjNiIsICJ2b2x1bWVfaWQiOiAiNGJkZGE5NzgtNmM1My00NDFlLTkzNmEtOWFkYjA5OTQ0NzhiIiwgImhvc3QiOiAibmNkdi1tYXN0ZXItMSIsICJhdmFpbGFiaWxpdHlfem9uZSI6IG51bGwsICJjb250YWluZXIiOiAiYmFja3VwcyIsICJwYXJlbnRfaWQiOiBudWxsLCAic3RhdHVzIjogImF2YWlsYWJsZSIsICJmYWlsX3JlYXNvbiI6IG51bGwsICJzaXplIjogMSwgImRpc3BsYXlfbmFtZSI6ICJ0ZW1wZXN0LVZvbHVtZXNCYWNrdXBzQWRtaW5UZXN0LUJhY2t1cC0xNjEzNjgyMDE0IiwgImRpc3BsYXlfZGVzY3JpcHRpb24iOiBudWxsLCAic2VydmljZV9tZXRhZGF0YSI6ICJ7XCJiYXNlXCI6IFwidm9sdW1lLTRiZGRhOTc4LTZjNTMtNDQxZS05MzZhLTlhZGIwOTk0NDc4Yi5iYWNrdXAuMzczNzViZDUtZjcwNi00OTRlLThjZWItZjY0NjZkYTRmYWM5XCJ9IiwgInNlcnZpY2UiOiAiY2luZGVyLmJhY2t1cC5kcml2ZXJzLmNlcGguQ2VwaEJhY2t1cERyaXZlciIsICJvYmplY3RfY291bnQiOiAwLCAidGVtcF92b2x1bWVfaWQiOiBudWxsLCAidGVtcF9zbmFwc2hvdF9pZCI6IG51bGwsICJudW1fZGVwZW5kZW50X2JhY2t1cHMiOiAwLCAic25hcHNob3RfaWQiOiBudWxsLCAiZGF0YV90aW1lc3RhbXAiOiAiMjAyMi0wNS0yOFQyMTo0MjozOFoiLCAicmVzdG9yZV92b2x1bWVfaWQiOiBudWxsLCAibWV0YWRhdGEiOiB7fSwgImVuY3J5cHRpb25fa2V5X2lkIjogbnVsbCwgImNyZWF0ZWRfYXQiOiAiMjAyMi0wNS0yOFQyMTo0MjozOFoiLCAidXBkYXRlZF9hdCI6ICIyMDIyLTA1LTI4VDIxOjQyOjQ2WiIsICJkZWxldGVkX2F0IjogbnVsbCwgImRlbGV0ZWQiOiBmYWxzZX0=. May 28 21:42:48.194452 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_db.sqlalchemy.engines [None req-5720ce1e-e1fe-40a4-ad73-dc7da4f5644b tempest-VolumesBackupsAdminTest-1214246761 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=135833) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 28 21:42:48.453312 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-5720ce1e-e1fe-40a4-ad73-dc7da4f5644b tempest-VolumesBackupsAdminTest-1214246761 None] Import record id 356da19d-75f5-4f64-b6a4-cfe3d0edfb2f metadata from driver finished. May 28 21:42:52.050049 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] Restore backup started, backup: 37375bd5-f706-494e-8ceb-f6466da4fac9 volume: 1cd0b801-96f3-4e1c-b0ce-3742852cf0e7. May 28 21:42:52.053158 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=19,cinder:UPDATE=4 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:42:52.065242 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.utils [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.33', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:42:52.066699 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:42:52.078852 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.008s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:42:52.079150 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[bd5648e1-f2a8-403b-bf17-01c0a23b24a4]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:a90a241cea8\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:42:52.080418 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:42:52.087177 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 28 21:42:52.088393 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[3badcddc-dcfe-41b0-bcf8-b96c0858bb4a]: [Errno 2] No such file or directory: 'systool' {{(pid=167755) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 28 21:42:52.088393 ncdv-master-1 cinder-backup[135833]: Traceback (most recent call last): May 28 21:42:52.088393 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 28 21:42:52.088393 ncdv-master-1 cinder-backup[135833]: ret = func(*f_args, **f_kwargs) May 28 21:42:52.088393 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 28 21:42:52.088393 ncdv-master-1 cinder-backup[135833]: return func(*args, **kwargs) May 28 21:42:52.088393 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 28 21:42:52.088393 ncdv-master-1 cinder-backup[135833]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 28 21:42:52.088393 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 28 21:42:52.088393 ncdv-master-1 cinder-backup[135833]: return putils.execute(on_execute=on_execute, May 28 21:42:52.088393 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 28 21:42:52.088393 ncdv-master-1 cinder-backup[135833]: obj = subprocess.Popen(cmd, May 28 21:42:52.088393 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 28 21:42:52.088393 ncdv-master-1 cinder-backup[135833]: self._execute_child(args, executable, preexec_fn, close_fds, May 28 21:42:52.088393 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 28 21:42:52.088393 ncdv-master-1 cinder-backup[135833]: raise child_exception_type(errno_num, err_msg, err_filename) May 28 21:42:52.088393 ncdv-master-1 cinder-backup[135833]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 28 21:42:52.092729 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[3badcddc-dcfe-41b0-bcf8-b96c0858bb4a]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:42:52.092729 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:42:52.096137 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 28 21:42:52.096658 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[5939f9be-2678-4cf5-8a0c-753c3a9b97f0]: [Errno 2] No such file or directory: 'systool' {{(pid=167755) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 28 21:42:52.096658 ncdv-master-1 cinder-backup[135833]: Traceback (most recent call last): May 28 21:42:52.096658 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 28 21:42:52.096658 ncdv-master-1 cinder-backup[135833]: ret = func(*f_args, **f_kwargs) May 28 21:42:52.096658 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 28 21:42:52.096658 ncdv-master-1 cinder-backup[135833]: return func(*args, **kwargs) May 28 21:42:52.096658 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 28 21:42:52.096658 ncdv-master-1 cinder-backup[135833]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 28 21:42:52.096658 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 28 21:42:52.096658 ncdv-master-1 cinder-backup[135833]: return putils.execute(on_execute=on_execute, May 28 21:42:52.096658 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 28 21:42:52.096658 ncdv-master-1 cinder-backup[135833]: obj = subprocess.Popen(cmd, May 28 21:42:52.096658 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 28 21:42:52.096658 ncdv-master-1 cinder-backup[135833]: self._execute_child(args, executable, preexec_fn, close_fds, May 28 21:42:52.096658 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 28 21:42:52.096658 ncdv-master-1 cinder-backup[135833]: raise child_exception_type(errno_num, err_msg, err_filename) May 28 21:42:52.096658 ncdv-master-1 cinder-backup[135833]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 28 21:42:52.098161 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[5939f9be-2678-4cf5-8a0c-753c3a9b97f0]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:42:52.098486 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:42:52.106369 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.008s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:42:52.107619 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[5e185070-7be9-4ab8-ac0e-cf1bacb9c419]: (4, ('/dev/sda1\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:42:52.108684 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:42:52.116960 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.008s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:42:52.117289 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[3262a235-7d4f-455a-ab8c-10f7f5773c42]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:42:52.118399 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:42:52.124667 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.006s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:42:52.124986 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[8fd2d20e-8176-49a6-8ea9-980a5ded4d78]: (4, ('fa6a5dbb-8967-894e-8b8d-5384b73a89bf\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:42:52.125758 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] Running cmd (subprocess): nvme version {{(pid=135833) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:42:52.151449 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] CMD "nvme version" returned: 0 in 0.025s {{(pid=135833) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:42:52.153173 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=135833) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 28 21:42:52.154076 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=135833) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 28 21:42:52.154450 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:41d0083e-6728-42e9-88e9-cd92e21a8271 dsc: {{(pid=135833) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 28 21:42:52.155134 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.utils [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] <== get_connector_properties: return (88ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.33', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:a90a241cea8', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'fa6a5dbb-8967-894e-8b8d-5384b73a89bf', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:41d0083e-6728-42e9-88e9-cd92e21a8271', 'found_dsc': ''} {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:42:52.811327 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connector [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] Factory for rbd on None {{(pid=135833) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 28 21:42:52.812928 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-1cd0b801-96f3-4e1c-b0ce-3742852cf0e7', 'hosts': ['10.222.0.33'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '1cd0b801-96f3-4e1c-b0ce-3742852cf0e7', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:42:52.814015 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.linuxrbd [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] opening connection to ceph cluster (timeout=-1). {{(pid=135833) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 28 21:42:53.078061 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] <== connect_volume: return (264ms) {'path': } {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:42:53.086587 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_db.sqlalchemy.engines [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=135833) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 28 21:42:53.099024 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] Starting restore from Ceph backup=37375bd5-f706-494e-8ceb-f6466da4fac9 to volume=volume-1cd0b801-96f3-4e1c-b0ce-3742852cf0e7 {{(pid=135833) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1349}} May 28 21:42:53.155797 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] Looking for snapshot of backup base 'volume-4bdda978-6c53-441e-936a-9adb0994478b.backup.37375bd5-f706-494e-8ceb-f6466da4fac9' {{(pid=135833) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:957}} May 28 21:42:53.156200 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] Found snapshot 'backup.37375bd5-f706-494e-8ceb-f6466da4fac9.snap.1653774160.011077' {{(pid=135833) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:976}} May 28 21:42:53.160425 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] Volume file is RBD. {{(pid=135833) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1251}} May 28 21:42:53.189399 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] Attempting differential restore. {{(pid=135833) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1304}} May 28 21:42:53.189901 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] Attempting incremental restore from base='volume-4bdda978-6c53-441e-936a-9adb0994478b.backup.37375bd5-f706-494e-8ceb-f6466da4fac9' snap='backup.37375bd5-f706-494e-8ceb-f6466da4fac9.snap.1653774160.011077' {{(pid=135833) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1150}} May 28 21:42:53.190295 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] Performing differential transfer from 'volume-4bdda978-6c53-441e-936a-9adb0994478b.backup.37375bd5-f706-494e-8ceb-f6466da4fac9' to 'volume-1cd0b801-96f3-4e1c-b0ce-3742852cf0e7' {{(pid=135833) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:658}} May 28 21:42:53.190696 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-4bdda978-6c53-441e-936a-9adb0994478b.backup.37375bd5-f706-494e-8ceb-f6466da4fac9@backup.37375bd5-f706-494e-8ceb-f6466da4fac9.snap.1653774160.011077 -' into... {{(pid=135833) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:615}} May 28 21:42:53.191061 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_yisf9kt3 --pool volumes - volumes/volume-1cd0b801-96f3-4e1c-b0ce-3742852cf0e7' {{(pid=135833) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:616}} May 28 21:42:55.797180 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] Restore transfer completed in 2.6062s {{(pid=135833) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1172}} May 28 21:42:55.798240 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=135833) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1360}} May 28 21:42:55.826442 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] Restoring volume base metadata {{(pid=135833) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} May 28 21:42:55.889560 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] No metadata of type 'volume-metadata' to restore {{(pid=135833) put /opt/stack/cinder/cinder/backup/driver.py:345}} May 28 21:42:55.889983 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] No metadata of type 'volume-glance-metadata' to restore {{(pid=135833) put /opt/stack/cinder/cinder/backup/driver.py:345}} May 28 21:42:55.893223 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] Restore to volume 1cd0b801-96f3-4e1c-b0ce-3742852cf0e7 finished successfully. {{(pid=135833) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1367}} May 28 21:42:55.894993 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-1cd0b801-96f3-4e1c-b0ce-3742852cf0e7', 'hosts': ['10.222.0.33'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '1cd0b801-96f3-4e1c-b0ce-3742852cf0e7', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:42:56.063560 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] <== disconnect_volume: return (167ms) None {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:42:56.114135 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.manager [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] Encryption key ID for volume 1cd0b801-96f3-4e1c-b0ce-3742852cf0e7 already matches encryption key ID in backup 37375bd5-f706-494e-8ceb-f6466da4fac9. {{(pid=135833) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} May 28 21:42:56.403605 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-60deed66-4cac-44bb-8cef-8765a60673a2 tempest-VolumesBackupsAdminTest-1665785387 None] Finished restoring backup 37375bd5-f706-494e-8ceb-f6466da4fac9 to volume 1cd0b801-96f3-4e1c-b0ce-3742852cf0e7. May 28 21:42:57.678229 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-198e3a39-54f4-4d1f-88df-891e2bf5d13d tempest-VolumesBackupsAdminTest-1214246761 None] Delete backup started, backup: 356da19d-75f5-4f64-b6a4-cfe3d0edfb2f. May 28 21:42:57.681541 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-198e3a39-54f4-4d1f-88df-891e2bf5d13d tempest-VolumesBackupsAdminTest-1214246761 None] Delete started for backup=356da19d-75f5-4f64-b6a4-cfe3d0edfb2f {{(pid=135833) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1376}} May 28 21:42:57.681541 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-198e3a39-54f4-4d1f-88df-891e2bf5d13d tempest-VolumesBackupsAdminTest-1214246761 None] Trying diff format basename='volume-4bdda978-6c53-441e-936a-9adb0994478b.backup.37375bd5-f706-494e-8ceb-f6466da4fac9' for backup base image of volume 4bdda978-6c53-441e-936a-9adb0994478b. {{(pid=135833) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:543}} May 28 21:42:57.894657 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-198e3a39-54f4-4d1f-88df-891e2bf5d13d tempest-VolumesBackupsAdminTest-1214246761 None] Looking for snapshot of backup base 'volume-4bdda978-6c53-441e-936a-9adb0994478b.backup.37375bd5-f706-494e-8ceb-f6466da4fac9' {{(pid=135833) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:957}} May 28 21:42:57.894946 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-198e3a39-54f4-4d1f-88df-891e2bf5d13d tempest-VolumesBackupsAdminTest-1214246761 None] Backup '356da19d-75f5-4f64-b6a4-cfe3d0edfb2f' has no snapshot {{(pid=135833) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:967}} May 28 21:42:57.895308 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-198e3a39-54f4-4d1f-88df-891e2bf5d13d tempest-VolumesBackupsAdminTest-1214246761 None] No backup snapshot to delete {{(pid=135833) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:504}} May 28 21:42:58.180828 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.drivers.ceph [None req-198e3a39-54f4-4d1f-88df-891e2bf5d13d tempest-VolumesBackupsAdminTest-1214246761 None] Backup base image of volume 4bdda978-6c53-441e-936a-9adb0994478b still has 1 snapshots so skipping base image delete. May 28 21:42:58.333526 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-198e3a39-54f4-4d1f-88df-891e2bf5d13d tempest-VolumesBackupsAdminTest-1214246761 None] Metadata backup object 'backup.356da19d-75f5-4f64-b6a4-cfe3d0edfb2f.meta' not found - ignoring {{(pid=135833) remove_if_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:162}} May 28 21:42:58.337065 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-198e3a39-54f4-4d1f-88df-891e2bf5d13d tempest-VolumesBackupsAdminTest-1214246761 None] Delete of backup '356da19d-75f5-4f64-b6a4-cfe3d0edfb2f' for volume '4bdda978-6c53-441e-936a-9adb0994478b' finished. {{(pid=135833) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1404}} May 28 21:42:58.343654 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_db.sqlalchemy.engines [None req-198e3a39-54f4-4d1f-88df-891e2bf5d13d tempest-VolumesBackupsAdminTest-1214246761 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=135833) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 28 21:42:58.431920 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.quota [None req-198e3a39-54f4-4d1f-88df-891e2bf5d13d tempest-VolumesBackupsAdminTest-1214246761 None] Created reservations ['02a693b0-a31a-4e99-b34f-b3c160b97929', 'eed7aebe-e72b-4315-9a0f-e88a46dccddf'] {{(pid=135833) reserve /opt/stack/cinder/cinder/quota.py:755}} May 28 21:42:58.624597 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-198e3a39-54f4-4d1f-88df-891e2bf5d13d tempest-VolumesBackupsAdminTest-1214246761 None] Delete backup finished, backup 356da19d-75f5-4f64-b6a4-cfe3d0edfb2f deleted. May 28 21:42:58.854607 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-59b9a3d1-ba71-4cdc-9019-4227d489471f tempest-VolumesBackupsAdminTest-1665785387 None] Delete backup started, backup: 37375bd5-f706-494e-8ceb-f6466da4fac9. May 28 21:42:58.856274 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-59b9a3d1-ba71-4cdc-9019-4227d489471f tempest-VolumesBackupsAdminTest-1665785387 None] Delete started for backup=37375bd5-f706-494e-8ceb-f6466da4fac9 {{(pid=135833) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1376}} May 28 21:42:58.856714 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-59b9a3d1-ba71-4cdc-9019-4227d489471f tempest-VolumesBackupsAdminTest-1665785387 None] Trying diff format basename='volume-4bdda978-6c53-441e-936a-9adb0994478b.backup.37375bd5-f706-494e-8ceb-f6466da4fac9' for backup base image of volume 4bdda978-6c53-441e-936a-9adb0994478b. {{(pid=135833) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:543}} May 28 21:42:58.999844 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-59b9a3d1-ba71-4cdc-9019-4227d489471f tempest-VolumesBackupsAdminTest-1665785387 None] Looking for snapshot of backup base 'volume-4bdda978-6c53-441e-936a-9adb0994478b.backup.37375bd5-f706-494e-8ceb-f6466da4fac9' {{(pid=135833) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:957}} May 28 21:42:59.000226 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-59b9a3d1-ba71-4cdc-9019-4227d489471f tempest-VolumesBackupsAdminTest-1665785387 None] Found snapshot 'backup.37375bd5-f706-494e-8ceb-f6466da4fac9.snap.1653774160.011077' {{(pid=135833) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:976}} May 28 21:42:59.000598 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-59b9a3d1-ba71-4cdc-9019-4227d489471f tempest-VolumesBackupsAdminTest-1665785387 None] Deleting backup snapshot='backup.37375bd5-f706-494e-8ceb-f6466da4fac9.snap.1653774160.011077' {{(pid=135833) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:501}} May 28 21:43:01.505469 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.drivers.ceph [None req-59b9a3d1-ba71-4cdc-9019-4227d489471f tempest-VolumesBackupsAdminTest-1665785387 None] Deleting backup base image='volume-4bdda978-6c53-441e-936a-9adb0994478b.backup.37375bd5-f706-494e-8ceb-f6466da4fac9' of volume 4bdda978-6c53-441e-936a-9adb0994478b. May 28 21:43:05.918590 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-59b9a3d1-ba71-4cdc-9019-4227d489471f tempest-VolumesBackupsAdminTest-1665785387 None] Base backup image='volume-4bdda978-6c53-441e-936a-9adb0994478b.backup.37375bd5-f706-494e-8ceb-f6466da4fac9' of volume 4bdda978-6c53-441e-936a-9adb0994478b deleted. {{(pid=135833) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:591}} May 28 21:43:06.034906 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-59b9a3d1-ba71-4cdc-9019-4227d489471f tempest-VolumesBackupsAdminTest-1665785387 None] Delete of backup '37375bd5-f706-494e-8ceb-f6466da4fac9' for volume '4bdda978-6c53-441e-936a-9adb0994478b' finished. {{(pid=135833) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1404}} May 28 21:43:06.040983 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_db.sqlalchemy.engines [None req-59b9a3d1-ba71-4cdc-9019-4227d489471f tempest-VolumesBackupsAdminTest-1665785387 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=135833) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 28 21:43:06.043907 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=40,cinder:UPDATE=10,cinder:INSERT=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:43:06.141291 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.quota [None req-59b9a3d1-ba71-4cdc-9019-4227d489471f tempest-VolumesBackupsAdminTest-1665785387 None] Created reservations ['b8e0b927-803a-4585-827d-5b6ececcf851', 'bdfae663-0806-44bf-9be5-53d658c35848'] {{(pid=135833) reserve /opt/stack/cinder/cinder/quota.py:755}} May 28 21:43:06.363638 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-59b9a3d1-ba71-4cdc-9019-4227d489471f tempest-VolumesBackupsAdminTest-1665785387 None] Delete backup finished, backup 37375bd5-f706-494e-8ceb-f6466da4fac9 deleted. May 28 21:43:10.238541 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] Create backup started, backup: 91dc451b-cfbb-497c-915d-e958a807681b volume: 29650c1b-4d9b-4ef4-87e6-128966066aa2. May 28 21:43:10.318061 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-28T21:43:09Z,data_timestamp=2022-05-28T21:43:09Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsAdminTest-Backup-1710208003',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=91dc451b-cfbb-497c-915d-e958a807681b,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='201bd6cbfd144d5c9ff30c81341773c6',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='a69e3e8e9a7c4eda80a8c110a62511e2',volume_id=29650c1b-4d9b-4ef4-87e6-128966066aa2) May 28 21:43:10.360775 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.utils [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.33', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:43:10.362156 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:43:10.369805 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.008s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:43:10.370205 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[366440b3-8a5b-4999-bcfb-6dc4332af799]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:a90a241cea8\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:43:10.371412 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:43:10.377222 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 28 21:43:10.377895 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[566436a3-c10f-4680-b5d9-bde7c613307e]: [Errno 2] No such file or directory: 'systool' {{(pid=167755) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 28 21:43:10.377895 ncdv-master-1 cinder-backup[135833]: Traceback (most recent call last): May 28 21:43:10.377895 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 28 21:43:10.377895 ncdv-master-1 cinder-backup[135833]: ret = func(*f_args, **f_kwargs) May 28 21:43:10.377895 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 28 21:43:10.377895 ncdv-master-1 cinder-backup[135833]: return func(*args, **kwargs) May 28 21:43:10.377895 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 28 21:43:10.377895 ncdv-master-1 cinder-backup[135833]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 28 21:43:10.377895 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 28 21:43:10.377895 ncdv-master-1 cinder-backup[135833]: return putils.execute(on_execute=on_execute, May 28 21:43:10.377895 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 28 21:43:10.377895 ncdv-master-1 cinder-backup[135833]: obj = subprocess.Popen(cmd, May 28 21:43:10.377895 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 28 21:43:10.377895 ncdv-master-1 cinder-backup[135833]: self._execute_child(args, executable, preexec_fn, close_fds, May 28 21:43:10.377895 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 28 21:43:10.377895 ncdv-master-1 cinder-backup[135833]: raise child_exception_type(errno_num, err_msg, err_filename) May 28 21:43:10.377895 ncdv-master-1 cinder-backup[135833]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 28 21:43:10.379347 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[566436a3-c10f-4680-b5d9-bde7c613307e]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:43:10.379347 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:43:10.385003 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 28 21:43:10.385623 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[a58462fd-e5ce-4b8f-8ae6-f51910d0ba8e]: [Errno 2] No such file or directory: 'systool' {{(pid=167755) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 28 21:43:10.385623 ncdv-master-1 cinder-backup[135833]: Traceback (most recent call last): May 28 21:43:10.385623 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 28 21:43:10.385623 ncdv-master-1 cinder-backup[135833]: ret = func(*f_args, **f_kwargs) May 28 21:43:10.385623 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 28 21:43:10.385623 ncdv-master-1 cinder-backup[135833]: return func(*args, **kwargs) May 28 21:43:10.385623 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 28 21:43:10.385623 ncdv-master-1 cinder-backup[135833]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 28 21:43:10.385623 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 28 21:43:10.385623 ncdv-master-1 cinder-backup[135833]: return putils.execute(on_execute=on_execute, May 28 21:43:10.385623 ncdv-master-1 cinder-backup[135833]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 28 21:43:10.385623 ncdv-master-1 cinder-backup[135833]: obj = subprocess.Popen(cmd, May 28 21:43:10.385623 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 28 21:43:10.385623 ncdv-master-1 cinder-backup[135833]: self._execute_child(args, executable, preexec_fn, close_fds, May 28 21:43:10.385623 ncdv-master-1 cinder-backup[135833]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 28 21:43:10.385623 ncdv-master-1 cinder-backup[135833]: raise child_exception_type(errno_num, err_msg, err_filename) May 28 21:43:10.385623 ncdv-master-1 cinder-backup[135833]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 28 21:43:10.387038 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[a58462fd-e5ce-4b8f-8ae6-f51910d0ba8e]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:43:10.387307 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:43:10.396044 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.008s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:43:10.396373 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[49862149-ecd5-4d5f-ad14-92ccccdeccc2]: (4, ('/dev/sda1\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:43:10.399474 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:43:10.406966 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.009s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:43:10.407290 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[5107f143-c9fe-494a-a2f7-24da3c5188fe]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:43:10.408368 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:43:10.415441 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.007s {{(pid=167755) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:43:10.415793 ncdv-master-1 cinder-backup[135833]: DEBUG oslo.privsep.daemon [-] privsep: reply[a8a289b1-739c-445a-ab6b-18755623cf25]: (4, ('fa6a5dbb-8967-894e-8b8d-5384b73a89bf\n', '')) {{(pid=167755) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 28 21:43:10.416972 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] Running cmd (subprocess): nvme version {{(pid=135833) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 28 21:43:10.441782 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_concurrency.processutils [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] CMD "nvme version" returned: 0 in 0.024s {{(pid=135833) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 28 21:43:10.443588 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=135833) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 28 21:43:10.443966 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=135833) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 28 21:43:10.444339 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.lightos [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:41d0083e-6728-42e9-88e9-cd92e21a8271 dsc: {{(pid=135833) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 28 21:43:10.444976 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.utils [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] <== get_connector_properties: return (83ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.33', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:a90a241cea8', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': 'fa6a5dbb-8967-894e-8b8d-5384b73a89bf', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:41d0083e-6728-42e9-88e9-cd92e21a8271', 'found_dsc': ''} {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:43:11.084955 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connector [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] Factory for rbd on None {{(pid=135833) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 28 21:43:11.086611 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-29650c1b-4d9b-4ef4-87e6-128966066aa2', 'hosts': ['10.222.0.33'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '29650c1b-4d9b-4ef4-87e6-128966066aa2', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:43:11.087650 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.linuxrbd [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] opening connection to ceph cluster (timeout=-1). {{(pid=135833) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 28 21:43:11.294900 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] <== connect_volume: return (207ms) {'path': } {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:43:11.304154 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_db.sqlalchemy.engines [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=135833) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 28 21:43:11.374877 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] Starting backup of volume='29650c1b-4d9b-4ef4-87e6-128966066aa2'. {{(pid=135833) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1028}} May 28 21:43:11.376385 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] Volume file is RBD: attempting optimized backup {{(pid=135833) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1039}} May 28 21:43:11.450139 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] Creating base image 'volume-29650c1b-4d9b-4ef4-87e6-128966066aa2.backup.91dc451b-cfbb-497c-915d-e958a807681b' {{(pid=135833) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:469}} May 28 21:43:11.450567 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=135833) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:277}} May 28 21:43:12.223844 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] Using --from-snap 'None' for incremental backup of volume 29650c1b-4d9b-4ef4-87e6-128966066aa2. {{(pid=135833) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:811}} May 28 21:43:12.224255 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] Creating backup snapshot='backup.91dc451b-cfbb-497c-915d-e958a807681b.snap.1653774192.2238672' {{(pid=135833) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:817}} May 28 21:43:13.312570 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] Performing differential transfer from 'volume-29650c1b-4d9b-4ef4-87e6-128966066aa2' to 'volume-29650c1b-4d9b-4ef4-87e6-128966066aa2.backup.91dc451b-cfbb-497c-915d-e958a807681b' {{(pid=135833) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:658}} May 28 21:43:13.313058 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_lz5zi5rj --pool volumes volumes/volume-29650c1b-4d9b-4ef4-87e6-128966066aa2@backup.91dc451b-cfbb-497c-915d-e958a807681b.snap.1653774192.2238672 -' into... {{(pid=135833) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:615}} May 28 21:43:13.313259 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-29650c1b-4d9b-4ef4-87e6-128966066aa2.backup.91dc451b-cfbb-497c-915d-e958a807681b' {{(pid=135833) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:616}} May 28 21:43:15.387333 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] Differential backup transfer completed in 2.0743s {{(pid=135833) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:837}} May 28 21:43:15.387938 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] Getting metadata type 'volume-base-metadata' {{(pid=135833) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 28 21:43:15.402372 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=135833) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 28 21:43:15.402787 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] Getting metadata type 'volume-metadata' {{(pid=135833) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 28 21:43:15.410276 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] No metadata type 'volume-metadata' available {{(pid=135833) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 28 21:43:15.411585 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] Getting metadata type 'volume-glance-metadata' {{(pid=135833) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 28 21:43:15.416882 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.driver [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] No metadata type 'volume-glance-metadata' available {{(pid=135833) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} May 28 21:43:15.417418 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] Backing up metadata for volume 29650c1b-4d9b-4ef4-87e6-128966066aa2. {{(pid=135833) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:1003}} May 28 21:43:15.570096 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] Backup '91dc451b-cfbb-497c-915d-e958a807681b' of volume 29650c1b-4d9b-4ef4-87e6-128966066aa2 finished. {{(pid=135833) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1069}} May 28 21:43:15.571939 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-29650c1b-4d9b-4ef4-87e6-128966066aa2', 'hosts': ['10.222.0.33'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '29650c1b-4d9b-4ef4-87e6-128966066aa2', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 28 21:43:16.088140 ncdv-master-1 cinder-backup[135833]: DEBUG os_brick.initiator.connectors.rbd [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] <== disconnect_volume: return (515ms) None {{(pid=135833) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 28 21:43:16.301064 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-eb69de26-5ac3-44f4-92f8-3782776dcdc3 tempest-VolumesBackupsAdminTest-1665785387 None] Create backup finished. backup: 91dc451b-cfbb-497c-915d-e958a807681b. May 28 21:43:17.503013 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-5f120fcc-944d-4a64-84af-522083743e33 tempest-VolumesBackupsAdminTest-1214246761 None] Reset backup status started, backup_id: 91dc451b-cfbb-497c-915d-e958a807681b, status: error. May 28 21:43:17.503013 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-5f120fcc-944d-4a64-84af-522083743e33 tempest-VolumesBackupsAdminTest-1214246761 None] Backup service: cinder.backup.drivers.ceph.CephBackupDriver. May 28 21:43:18.670152 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-8a3a70da-1dd3-40c5-94d6-3aa3d9698b6c tempest-VolumesBackupsAdminTest-1665785387 None] Delete backup started, backup: 91dc451b-cfbb-497c-915d-e958a807681b. May 28 21:43:18.671594 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-8a3a70da-1dd3-40c5-94d6-3aa3d9698b6c tempest-VolumesBackupsAdminTest-1665785387 None] Delete started for backup=91dc451b-cfbb-497c-915d-e958a807681b {{(pid=135833) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1376}} May 28 21:43:18.672004 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-8a3a70da-1dd3-40c5-94d6-3aa3d9698b6c tempest-VolumesBackupsAdminTest-1665785387 None] Trying diff format basename='volume-29650c1b-4d9b-4ef4-87e6-128966066aa2.backup.91dc451b-cfbb-497c-915d-e958a807681b' for backup base image of volume 29650c1b-4d9b-4ef4-87e6-128966066aa2. {{(pid=135833) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:543}} May 28 21:43:18.691266 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_db.sqlalchemy.engines [None req-1e29fabe-dc57-4ef4-ad80-ba98d322b6cd None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=135833) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 28 21:43:18.873086 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-8a3a70da-1dd3-40c5-94d6-3aa3d9698b6c tempest-VolumesBackupsAdminTest-1665785387 None] Looking for snapshot of backup base 'volume-29650c1b-4d9b-4ef4-87e6-128966066aa2.backup.91dc451b-cfbb-497c-915d-e958a807681b' {{(pid=135833) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:957}} May 28 21:43:18.873390 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-8a3a70da-1dd3-40c5-94d6-3aa3d9698b6c tempest-VolumesBackupsAdminTest-1665785387 None] Found snapshot 'backup.91dc451b-cfbb-497c-915d-e958a807681b.snap.1653774192.2238672' {{(pid=135833) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:976}} May 28 21:43:18.873761 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-8a3a70da-1dd3-40c5-94d6-3aa3d9698b6c tempest-VolumesBackupsAdminTest-1665785387 None] Deleting backup snapshot='backup.91dc451b-cfbb-497c-915d-e958a807681b.snap.1653774192.2238672' {{(pid=135833) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:501}} May 28 21:43:21.808600 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.drivers.ceph [None req-8a3a70da-1dd3-40c5-94d6-3aa3d9698b6c tempest-VolumesBackupsAdminTest-1665785387 None] Deleting backup base image='volume-29650c1b-4d9b-4ef4-87e6-128966066aa2.backup.91dc451b-cfbb-497c-915d-e958a807681b' of volume 29650c1b-4d9b-4ef4-87e6-128966066aa2. May 28 21:43:26.267024 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-8a3a70da-1dd3-40c5-94d6-3aa3d9698b6c tempest-VolumesBackupsAdminTest-1665785387 None] Base backup image='volume-29650c1b-4d9b-4ef4-87e6-128966066aa2.backup.91dc451b-cfbb-497c-915d-e958a807681b' of volume 29650c1b-4d9b-4ef4-87e6-128966066aa2 deleted. {{(pid=135833) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:591}} May 28 21:43:26.566566 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.backup.drivers.ceph [None req-8a3a70da-1dd3-40c5-94d6-3aa3d9698b6c tempest-VolumesBackupsAdminTest-1665785387 None] Delete of backup '91dc451b-cfbb-497c-915d-e958a807681b' for volume '29650c1b-4d9b-4ef4-87e6-128966066aa2' finished. {{(pid=135833) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1404}} May 28 21:43:26.569646 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=45,cinder:UPDATE=15,cinder:INSERT=4 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:43:26.688781 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.quota [None req-8a3a70da-1dd3-40c5-94d6-3aa3d9698b6c tempest-VolumesBackupsAdminTest-1665785387 None] Created reservations ['7212db89-dbc8-4fc4-afdd-2184735fab7c', 'bd52d82a-5496-4024-a332-88726fae3311'] {{(pid=135833) reserve /opt/stack/cinder/cinder/quota.py:755}} May 28 21:43:26.912195 ncdv-master-1 cinder-backup[135833]: INFO cinder.backup.manager [None req-8a3a70da-1dd3-40c5-94d6-3aa3d9698b6c tempest-VolumesBackupsAdminTest-1665785387 None] Delete backup finished, backup 91dc451b-cfbb-497c-915d-e958a807681b deleted. May 28 21:43:35.340526 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:43:35.341075 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:43:38.669150 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:43:38.671613 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=30,cinder:UPDATE=10,cinder:INSERT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:44:18.673055 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=9,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:44:18.673055 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:44:35.345775 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:44:35.346468 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:44:58.663788 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:44:58.665337 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:45:18.675132 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:45:35.350769 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:45:35.351465 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:45:38.675862 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=10,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:45:58.667254 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:46:18.672885 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:46:35.355759 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:46:35.356524 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:46:38.671660 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:46:38.672322 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:47:18.670649 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:47:18.673558 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:47:35.360721 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:47:35.361921 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:47:38.673374 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:47:58.680610 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:48:18.678949 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:48:35.356447 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:48:35.357103 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:48:38.673648 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:48:48.678772 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:49:18.677899 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:49:28.686654 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:49:35.361192 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:49:35.361859 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:49:58.687330 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:50:08.688497 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:50:28.687489 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:50:35.366761 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:50:35.367442 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:50:48.688033 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:51:08.686818 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:51:28.687246 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:51:28.688358 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:51:35.372260 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:51:35.372805 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:52:08.678738 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:52:08.679378 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=4 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:52:28.685394 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:52:35.378200 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:52:35.378784 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:52:48.690739 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=4 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:53:08.681511 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:53:18.690998 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:53:35.383467 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:53:35.384074 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:53:48.688739 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:53:58.689499 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:54:18.692744 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:54:28.690410 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:54:35.388780 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:54:35.389417 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:54:38.697338 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:55:08.687932 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:55:18.690821 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:55:28.696425 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:55:35.393763 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:55:35.394381 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:55:38.711257 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:55:58.699645 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:56:18.708037 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:56:35.390525 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:56:35.391141 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:56:38.704348 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:56:58.704723 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:57:18.706137 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:57:35.395187 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:57:35.401184 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:57:38.713739 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:57:58.706495 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:57:58.708478 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:58:28.712609 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:58:35.400661 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:58:35.401265 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:58:38.708430 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:59:08.696430 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:59:18.699328 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 21:59:35.405092 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 21:59:35.406796 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 21:59:38.707827 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:00:08.700744 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:00:18.705571 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:00:35.410675 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:00:35.411291 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:00:38.716241 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:00:48.715911 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:00:58.727111 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:01:18.721822 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:01:35.415333 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:01:35.415990 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:01:38.724852 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:01:58.730351 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:02:18.731922 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:02:28.739721 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=3,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:02:35.420724 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:02:35.421323 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:02:48.745614 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:02:48.749620 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:02:58.783011 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:03:18.751210 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:03:28.758242 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:03:35.425691 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:03:35.426356 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:03:38.765076 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:03:48.765993 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:03:58.761962 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:04:28.756583 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:04:35.421434 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:04:35.422601 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:04:38.763867 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:04:58.769615 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:05:18.760301 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:05:35.417441 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:05:35.418124 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:05:38.761141 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:05:38.761803 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:05:58.765414 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:06:18.770192 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=9,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:06:35.422849 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:06:35.423487 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:06:38.774408 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:06:38.775202 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:07:08.773765 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:07:18.777626 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=9,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:07:28.779272 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:07:35.428045 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:07:35.428829 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:07:48.776385 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:08:08.777014 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:08:28.777369 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:08:28.779296 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:08:35.433667 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:08:35.434363 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:08:58.779480 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:09:08.779976 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:09:18.789435 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:09:35.439659 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:09:35.440304 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:09:38.790084 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:09:48.795267 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=9,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:10:08.796598 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:10:28.804935 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:10:35.444737 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:10:35.445652 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:10:48.816824 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:11:18.806508 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:11:28.805386 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:11:35.441355 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:11:35.442068 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:11:38.812645 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:11:58.818161 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:12:08.823215 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:12:28.822680 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:12:35.446563 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:12:35.447233 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:12:38.832506 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:13:08.823464 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:13:18.829133 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:13:35.451612 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:13:35.452236 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:13:38.833633 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:13:48.834746 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:14:18.839991 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:14:18.843309 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:14:35.456795 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:14:35.457406 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:14:38.849695 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:14:58.862616 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=9,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:15:18.865554 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:15:35.461906 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:15:35.462590 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:15:38.859920 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=9,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:16:08.863666 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:16:08.866225 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:16:35.466632 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:16:35.467237 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:16:48.864977 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=9,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:16:58.871651 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:17:18.871031 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:17:35.471560 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:17:35.472196 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:17:38.874785 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:18:08.881912 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:18:28.878702 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=1,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:18:35.477109 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:18:35.477763 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:18:38.878072 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:18:58.875640 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:19:18.879019 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:19:35.481638 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:19:35.482366 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:19:38.875697 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:19:58.878663 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:20:18.878692 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:20:35.486660 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:20:35.487308 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:20:38.882560 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:20:58.873551 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:21:18.875682 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:21:35.492410 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:21:35.493089 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:21:38.881224 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:22:08.878558 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:22:18.880607 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:22:35.496685 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:22:35.497330 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:22:38.879114 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=3,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:22:58.882640 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:23:18.881696 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:23:18.884528 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:23:35.501720 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:23:35.502419 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:23:38.887518 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:23:58.892911 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=10,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:24:18.893650 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:24:35.507102 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:24:35.507804 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:24:38.902804 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:24:58.901943 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:24:58.904388 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:25:18.910312 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:25:35.512435 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:25:35.513078 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:25:48.907606 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:25:48.912067 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=8,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:26:18.906997 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:26:28.910600 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:26:35.517725 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:26:35.518416 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:26:48.917075 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:27:08.915812 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:27:28.909765 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=7,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:27:35.522570 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:27:35.523206 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:27:48.909496 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:27:48.910176 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:28:08.912787 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:28:28.912347 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=9,cinder:UPDATE=3 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:28:35.527644 ncdv-master-1 cinder-backup[135833]: DEBUG oslo_service.periodic_task [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=135833) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 28 22:28:35.528304 ncdv-master-1 cinder-backup[135833]: DEBUG cinder.manager [None req-1845a122-6620-4db6-9e85-e56b07ed3d68 None None] Notifying Schedulers of capabilities ... {{(pid=135833) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 28 22:28:38.921730 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=4,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:28:58.926217 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=5,cinder:UPDATE=2 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} May 28 22:29:18.919687 ncdv-master-1 cinder-backup[135833]: DEBUG dbcounter [-] [135833] Writing DB stats cinder:SELECT=6,cinder:UPDATE=1 {{(pid=135833) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}