-- Logs begin at Sat 2022-05-14 20:01:12 UTC, end at Sat 2022-05-14 22:44:13 UTC. -- May 14 21:01:50.601847 ncdv-master-1 systemd[1]: Started Devstack devstack@c-bak.service. May 14 21:01:53.997291 ncdv-master-1 cinder-backup[123339]: INFO cinder.cmd.backup [-] Backup running with 1 processes. May 14 21:01:54.017567 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_concurrency.lockutils [-] Acquired lock "singleton_lock" {{(pid=123339) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:294}} May 14 21:01:54.017567 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_concurrency.lockutils [-] Releasing lock "singleton_lock" {{(pid=123339) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:312}} May 14 21:01:54.176987 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_db.sqlalchemy.engines [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123339) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 14 21:01:54.417152 ncdv-master-1 cinder-backup[123339]: INFO cinder.rpc [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] Automatically selected cinder-backup objects version 1.39 as minimum service version. May 14 21:01:54.421824 ncdv-master-1 cinder-backup[123339]: INFO cinder.rpc [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] Automatically selected cinder-backup RPC version 2.3 as minimum service version. May 14 21:01:54.427257 ncdv-master-1 cinder-backup[123339]: INFO cinder.rpc [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] Automatically selected cinder-volume objects version 1.39 as minimum service version. May 14 21:01:54.431834 ncdv-master-1 cinder-backup[123339]: INFO cinder.rpc [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] Automatically selected cinder-volume RPC version 3.18 as minimum service version. May 14 21:01:54.437456 ncdv-master-1 cinder-backup[123339]: INFO cinder.rpc [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] Automatically selected cinder-scheduler objects version 1.39 as minimum service version. May 14 21:01:54.442470 ncdv-master-1 cinder-backup[123339]: INFO cinder.rpc [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] Automatically selected cinder-scheduler RPC version 3.12 as minimum service version. May 14 21:01:54.503714 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_db.sqlalchemy.engines [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123339) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 14 21:01:54.598083 ncdv-master-1 cinder-backup[123339]: INFO oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] Starting 1 workers May 14 21:01:54.605341 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] Started child 123784 {{(pid=123339) _start_child /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:575}} May 14 21:01:54.605341 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] Full set of CONF: {{(pid=123339) wait /usr/local/lib/python3.8/dist-packages/oslo_service/service.py:649}} May 14 21:01:54.605341 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] ******************************************************************************** {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} May 14 21:01:54.605772 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] Configuration options gathered from: {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} May 14 21:01:54.606232 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] command line args: ['--config-file', '/etc/cinder/cinder.conf'] {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} May 14 21:01:54.609381 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] config files: ['/etc/cinder/cinder.conf'] {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} May 14 21:01:54.609381 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] ================================================================================ {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} May 14 21:01:54.609381 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] allow_availability_zone_fallback = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.609381 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] allow_compression_on_image_upload = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.613754 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] allowed_direct_url_schemes = [] {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.613754 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] api_rate_limit = True {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.613754 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] auth_strategy = keystone {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.613754 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_availability_zone = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.613754 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backup_api_class = cinder.backup.api.API {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.613754 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backup_ceph_chunk_size = 134217728 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.615852 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backup_ceph_conf = /etc/ceph/ceph.conf {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.615852 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backup_ceph_image_journals = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.615852 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backup_ceph_pool = backups {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.615852 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backup_ceph_stripe_count = 0 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.621514 ncdv-master-1 cinder-backup[123784]: INFO cinder.service [-] Starting cinder-backup node (version 20.1.0) May 14 21:01:54.624361 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backup_ceph_stripe_unit = 0 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.624826 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backup_ceph_user = cinder-bak {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.625302 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backup_driver = cinder.backup.drivers.ceph.CephBackupDriver {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.626061 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backup_driver_init_check_interval = 60 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.627422 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backup_driver_stats_polling_interval = 60 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.628991 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backup_manager = cinder.backup.manager.BackupManager {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.629727 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backup_max_operations = 15 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.630288 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backup_metadata_version = 2 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.630817 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backup_name_template = backup-%s {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.631293 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backup_native_threads_pool_size = 60 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.631831 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backup_object_number_per_notification = 10 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.632321 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backup_service_inithost_offload = True {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.632863 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backup_timer_interval = 120 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.633335 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backup_use_same_host = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.633822 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backup_use_temp_snapshot = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.634230 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backup_workers = 1 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.634655 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] chap_password = **** {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.640663 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] chap_username = {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.640663 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.640663 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cinder_internal_tenant_project_id = 0911a024287e430ea6ee015389f0c440 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.640663 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cinder_internal_tenant_user_id = 198271d1c0d44cb58bace41a65f4b94c {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.640663 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] compression_format = gzip {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.640663 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] config_dir = [] {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.641156 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] config_file = ['/etc/cinder/cinder.conf'] {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.641156 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] config_source = [] {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.641156 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] consistencygroup_api_class = cinder.consistencygroup.api.API {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.641156 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] control_exchange = openstack {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.641156 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] debug = True {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.641156 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] default_availability_zone = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.641555 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] default_group_type = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.641555 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'swiftclient=WARN', 'botocore=WARN'] {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.642615 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] default_volume_type = ceph {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.643055 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] driver_client_cert = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.647968 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] driver_client_cert_key = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.647968 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] driver_data_namespace = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.647968 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] driver_ssl_cert_path = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.647968 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] driver_ssl_cert_verify = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.647968 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] driver_use_ssl = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.647968 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] enable_new_services = True {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.647968 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] enable_unsupported_driver = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.648779 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] enabled_backends = ['ceph'] {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.648779 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] enforce_multipath_for_image_xfer = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.648779 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] filter_function = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.648779 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] glance_api_insecure = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.648779 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] glance_api_servers = ['http://10.222.0.52/image'] {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.648779 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] glance_api_ssl_compression = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.649297 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] glance_ca_certificates_file = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.767698 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] glance_catalog_info = image:glance:publicURL {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.767698 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] glance_certfile = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.767698 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] glance_core_properties = ['checksum', 'container_format', 'disk_format', 'image_name', 'image_id', 'min_disk', 'min_ram', 'name', 'size'] {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.767698 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] glance_keyfile = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.767698 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] glance_num_retries = 3 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.767698 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] glance_request_timeout = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.812535 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] goodness_function = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.812535 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] graceful_shutdown_timeout = 5 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.812535 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] group_api_class = cinder.group.api.API {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.812535 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] host = ncdv-master-1 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.812535 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] iet_conf = /etc/iet/ietd.conf {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.812535 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] image_compress_on_upload = True {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.813174 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] image_conversion_address_space_limit = 1 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.813174 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] image_conversion_cpu_limit = 60 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.813174 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] image_conversion_dir = /opt/stack/data/cinder/conversion {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.813174 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] image_upload_use_cinder_backend = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.813174 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] image_upload_use_internal_tenant = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.813174 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] image_volume_cache_enabled = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.813665 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] image_volume_cache_max_count = 0 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.813665 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] image_volume_cache_max_size_gb = 0 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.813665 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] instance_format = [instance: %(uuid)s] {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.813665 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] instance_uuid_format = [instance: %(uuid)s] {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.813999 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.lockutils [None req-6ac56b3a-184a-4236-a8ea-61c85dcf03d1 None None] Lock "cleanup_incomplete_backups_123339" acquired by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: waited 0.001s {{(pid=123784) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:386}} May 14 21:01:54.814107 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] iscsi_iotype = fileio {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.814196 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-6ac56b3a-184a-4236-a8ea-61c85dcf03d1 None None] Cleaning up incomplete backup operations. May 14 21:01:54.814286 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] iscsi_secondary_ip_addresses = [] {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.814286 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] iscsi_target_flags = {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.814286 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] iscsi_write_cache = on {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.814286 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] iser_helper = tgtadm {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.814286 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] iser_ip_address = 10.222.0.52 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.814286 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] iser_port = 3260 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.814286 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.814815 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] log_config_append = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.814815 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.814815 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] log_dir = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.814815 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] log_file = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.814815 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] log_options = True {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.814815 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] log_rotate_interval = 1 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.814815 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] log_rotate_interval_type = days {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.815381 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_db.sqlalchemy.engines [None req-6ac56b3a-184a-4236-a8ea-61c85dcf03d1 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123784) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 14 21:01:54.815496 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] log_rotation_type = none {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.815496 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.815496 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.815496 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.815496 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.815935 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] logging_user_identity_format = %(user)s %(project)s %(domain)s %(user_domain)s %(project_domain)s {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.815935 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] max_age = 0 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.815935 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] max_logfile_count = 30 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.815935 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] max_logfile_size_mb = 200 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.815935 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] max_over_subscription_ratio = 20.0 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.815935 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] message_reap_interval = 86400 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.816443 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] message_ttl = 2592000 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.816443 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] monkey_patch = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.816443 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] monkey_patch_modules = [] {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.816443 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] my_ip = 10.222.0.52 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.816443 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] no_snapshot_gb_quota = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.816443 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] num_iser_scan_tries = 3 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.816443 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] num_shell_tries = 3 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.816966 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] num_volume_device_scan_tries = 3 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.816966 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] nvmet_ns_id = 10 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.816966 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] nvmet_port_id = 1 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.816966 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] osapi_max_limit = 1000 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.816966 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] osapi_volume_ext_list = [] {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.816966 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] osapi_volume_extension = ['cinder.api.contrib.standard_extensions'] {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.816966 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] osapi_volume_listen = 0.0.0.0 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.817494 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] osapi_volume_listen_port = 8776 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.817494 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] osapi_volume_use_ssl = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.817494 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] osapi_volume_workers = 2 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.817494 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] per_volume_size_limit = -1 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.817494 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] periodic_fuzzy_delay = 60 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.817494 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] periodic_interval = 60 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.817494 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] project_id_regex = [0-9a-f\-]+ {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.818065 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] public_endpoint = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.818065 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] publish_errors = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.818065 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] quota_backup_gigabytes = 1000 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.818065 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] quota_backups = 10 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.818065 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] quota_consistencygroups = 10 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.818065 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] quota_driver = cinder.quota.DbQuotaDriver {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.818065 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] quota_gigabytes = 1000 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.818573 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] quota_groups = 10 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.818573 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] quota_snapshots = 10 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.818573 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] quota_volumes = 10 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.818573 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] rate_limit_burst = 0 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.818573 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] rate_limit_except_level = CRITICAL {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.818573 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] rate_limit_interval = 0 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.818573 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] replication_device = **** {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.819308 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] report_discard_supported = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.819308 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] report_interval = 10 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.819308 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] reservation_clean_interval = 86400 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.819308 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] reservation_expire = 86400 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.819308 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] reserved_percentage = 0 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.819308 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] resource_query_filters_file = /etc/cinder/resource_filters.json {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.819308 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] restore_discard_excess_bytes = True {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.819879 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] rootwrap_config = /etc/cinder/rootwrap.conf {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.819879 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] rpc_response_timeout = 120 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.819879 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] run_external_periodic_tasks = True {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.819879 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] scheduler_manager = cinder.scheduler.manager.SchedulerManager {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.819879 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] scst_target_driver = iscsi {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.819879 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] scst_target_iqn_name = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.821433 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] service_down_time = 60 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.821433 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] snapshot_name_template = snapshot-%s {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.821433 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] split_loggers = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.821433 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] state_path = /opt/stack/data/cinder {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.821433 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] storage_availability_zone = nova {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.821433 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] storage_protocol = iSCSI {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.821433 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] syslog_log_facility = LOG_USER {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.821986 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] target_helper = lioadm {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.821986 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] target_ip_address = 10.222.0.52 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.821986 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] target_port = 3260 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.821986 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] target_prefix = iqn.2010-10.org.openstack: {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.821986 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] target_protocol = iscsi {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.821986 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] trace_flags = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.823977 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] transfer_api_class = cinder.transfer.api.API {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.823977 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] transport_url = **** {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.823977 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] until_refresh = 0 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.823977 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] use_chap_auth = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.823977 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] use_default_quota_class = True {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.823977 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] use_eventlog = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.823977 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] use_journal = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.824576 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] use_json = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.824576 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] use_multipath_for_image_xfer = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.824576 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] use_stderr = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.824576 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] use_syslog = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.824576 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] verify_glance_signatures = enabled {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.824576 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] volume_api_class = cinder.volume.api.API {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.824576 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] volume_backend_name = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.825134 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] volume_clear = zero {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.825134 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] volume_clear_ionice = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.825134 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] volume_clear_size = 0 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.825134 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.825134 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] volume_copy_bps_limit = 0 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.825134 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] volume_dd_blocksize = 1M {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.825618 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] volume_manager = cinder.volume.manager.VolumeManager {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.825618 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] volume_name_template = volume-%s {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.825618 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] volume_usage_audit_period = month {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.825618 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] volumes_dir = /opt/stack/data/cinder/volumes {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.825618 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] watch_log_file = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} May 14 21:01:54.825618 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.826152 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.826152 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_concurrency.disable_process_locking = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.826152 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_concurrency.lock_path = /opt/stack/data/cinder {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.826152 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.826152 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.826152 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_metrics.metrics_process_name = {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.826633 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.826633 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.826633 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.admin_password = **** {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.826633 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.admin_tenant_name = admin {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.826633 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.admin_token = **** {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.826633 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.admin_user = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.827749 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.auth_admin_prefix = {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.827749 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.827749 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.auth_port = 35357 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.827749 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.auth_protocol = https {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.827749 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.auth_section = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.827749 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.auth_type = password {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.827749 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.auth_uri = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.828408 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.auth_version = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.828408 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.cache = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.828408 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.cafile = /opt/stack/data/ca-bundle.pem {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.828408 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.certfile = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.828408 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.delay_auth_decision = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.828408 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.enforce_token_bind = permissive {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.828944 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.http_connect_timeout = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.828944 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.http_request_max_retries = 3 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.828944 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.identity_uri = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.828944 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.include_service_catalog = True {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.828944 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.insecure = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.828944 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.interface = public {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.829413 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.keyfile = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.829413 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.829413 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.829413 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.829413 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.829413 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.829907 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.memcache_secret_key = **** {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.829907 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.memcache_security_strategy = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.829907 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.829907 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.memcached_servers = ['localhost:11211'] {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.829907 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.region_name = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.829907 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.service_token_roles = ['service'] {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.830368 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.service_token_roles_required = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.830368 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.service_type = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.830368 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.token_cache_time = 300 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.830368 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] keystone_authtoken.www_authenticate_uri = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.830368 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.backend = dogpile.cache.null {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.830368 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.backend_argument = **** {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.830812 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.config_prefix = cache.oslo {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.830812 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.dead_timeout = 60.0 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.830812 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.debug_cache_backend = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.830812 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.enable_retry_client = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.830812 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.enable_socket_keepalive = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.830812 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.enabled = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.830812 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.expiration_time = 600 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.831367 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.hashclient_retry_attempts = 2 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.831367 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.hashclient_retry_delay = 1.0 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.831367 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.memcache_dead_retry = 300 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.831367 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.memcache_pool_connection_get_timeout = 10 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.831367 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.memcache_pool_flush_on_reconnect = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.831367 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.memcache_pool_maxsize = 10 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.831367 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.memcache_pool_unused_timeout = 60 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.841758 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.memcache_servers = ['localhost:11211'] {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.841758 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.memcache_socket_timeout = 1.0 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.841758 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.proxies = [] {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.841758 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.retry_attempts = 2 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.841758 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.retry_delay = 0.0 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.841758 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.socket_keepalive_count = 1 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.841758 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.socket_keepalive_idle = 1 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.843450 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.socket_keepalive_interval = 1 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.843450 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.tls_allowed_ciphers = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.843450 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.tls_cafile = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.843450 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.tls_certfile = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.843450 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.tls_enabled = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.843450 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cache.tls_keyfile = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.843450 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] coordination.backend_url = etcd3+http://10.222.0.52:2379 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.843990 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] database.backend = sqlalchemy {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.843990 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] database.connection = **** {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.843990 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] database.connection_debug = 0 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.843990 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] database.connection_parameters = {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.843990 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] database.connection_recycle_time = 3600 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.843990 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] database.connection_trace = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.843990 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] database.db_inc_retry_interval = True {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.844430 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] database.db_max_retries = 20 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.844430 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] database.db_max_retry_interval = 10 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.844430 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] database.db_retry_interval = 1 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.844430 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] database.max_overflow = 50 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.844430 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] database.max_pool_size = 5 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.844430 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] database.max_retries = 10 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.844430 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] database.mysql_enable_ndb = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.844883 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] database.mysql_sql_mode = TRADITIONAL {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.844883 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] database.pool_timeout = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.844883 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] database.retry_interval = 10 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.844883 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] database.slave_connection = **** {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.844883 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] database.sqlite_synchronous = True {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.844883 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] database.use_db_reconnect = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.844883 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_policy.enforce_new_defaults = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.845340 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_policy.enforce_scope = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.845340 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_policy.policy_default_rule = default {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.845340 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_policy.policy_dirs = ['policy.d'] {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.845340 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_policy.policy_file = policy.yaml {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.845340 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.845340 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.845717 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.845717 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.845717 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.845717 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] profiler.connection_string = messaging:// {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.845717 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] profiler.enabled = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.845717 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] profiler.es_doc_type = notification {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.846086 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] profiler.es_scroll_size = 10000 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.846086 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] profiler.es_scroll_time = 2m {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.846086 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] profiler.filter_error_trace = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.846086 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] profiler.hmac_keys = SECRET_KEY {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.846086 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] profiler.sentinel_service_name = mymaster {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.846086 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] profiler.socket_timeout = 0.1 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.846086 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] profiler.trace_sqlalchemy = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.846533 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] privsep_osbrick.capabilities = [21] {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.846533 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] privsep_osbrick.group = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.846533 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] privsep_osbrick.helper_command = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.846533 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] privsep_osbrick.logger_name = oslo_privsep.daemon {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.846533 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] privsep_osbrick.thread_pool_size = 4 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.846533 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] privsep_osbrick.user = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.846533 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cinder_sys_admin.capabilities = [0, 1, 2, 3, 12, 21] {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.846977 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cinder_sys_admin.group = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.846977 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cinder_sys_admin.helper_command = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.846977 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cinder_sys_admin.logger_name = oslo_privsep.daemon {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.846977 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cinder_sys_admin.thread_pool_size = 4 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.846977 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] cinder_sys_admin.user = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.846977 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] service_user.auth_section = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.846977 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] service_user.auth_type = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.847431 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] service_user.cafile = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.847431 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] service_user.certfile = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.847431 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] service_user.collect_timing = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.847431 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] service_user.insecure = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.847431 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] service_user.keyfile = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.847431 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] service_user.send_service_user_token = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.847431 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] service_user.split_loggers = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.849519 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] service_user.timeout = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.849519 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.backend_availability_zone = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.849519 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.chap_password = **** {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.849519 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.chap_username = {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.849519 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.chiscsi_conf = /etc/chelsio-iscsi/chiscsi.conf {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.849519 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.deferred_deletion_delay = 0 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.850395 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.deferred_deletion_purge_interval = 60 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.850395 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.driver_client_cert = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.850395 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.driver_client_cert_key = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.850395 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.driver_data_namespace = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.850395 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.driver_ssl_cert_path = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.850395 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.driver_ssl_cert_verify = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.859329 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.driver_use_ssl = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.859329 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.enable_deferred_deletion = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.859329 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.enable_unsupported_driver = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.859329 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.enforce_multipath_for_image_xfer = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.859329 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.filter_function = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.859329 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.goodness_function = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.859830 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.iet_conf = /etc/iet/ietd.conf {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.859830 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.image_upload_use_cinder_backend = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.859830 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.image_upload_use_internal_tenant = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.859830 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.image_volume_cache_enabled = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.859830 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.image_volume_cache_max_count = 0 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.859830 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.image_volume_cache_max_size_gb = 0 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.860234 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.iscsi_iotype = fileio {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.860234 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.iscsi_secondary_ip_addresses = [] {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.860234 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.iscsi_target_flags = {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.860234 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.iscsi_write_cache = on {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.860234 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.iser_helper = tgtadm {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.860234 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.iser_ip_address = 10.222.0.52 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.860234 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.iser_port = 3260 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.860688 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.iser_target_prefix = iqn.2010-10.org.openstack: {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.860688 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.max_over_subscription_ratio = 20.0 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.860688 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.num_iser_scan_tries = 3 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.860688 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.num_shell_tries = 3 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.860688 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.num_volume_device_scan_tries = 3 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.860688 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.nvmet_ns_id = 10 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.861101 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.nvmet_port_id = 1 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.861101 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.rados_connect_timeout = -1 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.861101 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.rados_connection_interval = 5 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.861101 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.rados_connection_retries = 3 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.861101 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.rbd_ceph_conf = {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.861101 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.rbd_cluster_name = ceph {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.861101 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.rbd_exclusive_cinder_pool = True {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.861555 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.rbd_flatten_volume_from_snapshot = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.861555 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.rbd_max_clone_depth = 5 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.861555 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.rbd_pool = rbd {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.861555 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.rbd_secret_uuid = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.861555 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.rbd_store_chunk_size = 4 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.861555 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.rbd_user = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.861950 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.replication_connect_timeout = 5 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.861950 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.replication_device = **** {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.862099 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.lockutils [None req-6ac56b3a-184a-4236-a8ea-61c85dcf03d1 None None] Lock "cleanup_incomplete_backups_123339" "released" by "cinder.backup.manager.BackupManager._cleanup_incomplete_backup_operations" :: held 0.152s {{(pid=123784) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:400}} May 14 21:01:54.862177 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.report_discard_supported = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.862177 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.report_dynamic_total_capacity = True {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.862177 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.reserved_percentage = 0 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.862177 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.scst_target_driver = iscsi {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.862177 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.scst_target_iqn_name = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.862177 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.storage_protocol = iSCSI {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.862566 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.target_helper = tgtadm {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.862566 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.target_ip_address = 10.222.0.52 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.862566 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.target_port = 3260 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.862566 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.target_prefix = iqn.2010-10.org.openstack: {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.862566 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.target_protocol = iscsi {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.862566 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.trace_flags = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.862944 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.unique_fqdn_network = True {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.862944 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.use_chap_auth = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.862944 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.use_multipath_for_image_xfer = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.862944 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.volume_backend_name = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.862944 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.volume_clear = zero {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.862944 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.volume_clear_ionice = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.863376 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.volume_clear_size = 0 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.863376 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.volume_copy_blkio_cgroup_name = cinder-volume-copy {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.863602 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_db.sqlalchemy.engines [None req-6ac56b3a-184a-4236-a8ea-61c85dcf03d1 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123784) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 14 21:01:54.863704 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.volume_copy_bps_limit = 0 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.863704 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.volume_dd_blocksize = 1M {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.863704 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] backend_defaults.volumes_dir = /opt/stack/data/cinder/volumes {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.863704 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_reports.file_event_handler = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.863704 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_reports.file_event_handler_interval = 1 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.863704 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_reports.log_dir = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.864131 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.864131 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.864131 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.864131 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.864131 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.864131 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.864523 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.heartbeat_in_pthread = True {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.864523 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.heartbeat_rate = 2 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.864523 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.864523 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.864523 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.864863 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-6ac56b3a-184a-4236-a8ea-61c85dcf03d1 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:01:54.869023 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.869921 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.870536 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.871143 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.871787 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.872358 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.872969 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.873545 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.874696 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.875424 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.876553 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.877198 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.ssl = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.877804 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.ssl_ca_file = {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.878391 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.ssl_cert_file = {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.879022 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.ssl_enforce_fips_mode = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.879632 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.ssl_key_file = {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.883610 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_rabbit.ssl_version = {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.884432 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_notifications.driver = [] {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.885100 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_notifications.retry = -1 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.885758 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.886484 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] oslo_messaging_notifications.transport_url = **** {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.887176 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] key_manager.backend = cinder.keymgr.conf_key_mgr.ConfKeyManager {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.889703 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] barbican.auth_endpoint = http://localhost/identity/v3 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.890701 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] barbican.barbican_api_version = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.891280 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] barbican.barbican_endpoint = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.892080 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] barbican.barbican_endpoint_type = public {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.892712 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] barbican.barbican_region_name = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.893396 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] barbican.cafile = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.894133 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] barbican.certfile = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.894812 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] barbican.collect_timing = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.895540 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] barbican.insecure = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.896193 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] barbican.keyfile = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.896962 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] barbican.number_of_retries = 60 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.897717 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] barbican.retry_delay = 1 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.898414 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] barbican.send_service_user_token = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.899104 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] barbican.split_loggers = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.908916 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] barbican.timeout = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.908916 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] barbican.verify_ssl = True {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.908916 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] barbican.verify_ssl_path = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.908916 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] barbican_service_user.auth_section = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.908916 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] barbican_service_user.auth_type = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.908916 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] barbican_service_user.cafile = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.908916 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] barbican_service_user.certfile = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.909721 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] barbican_service_user.collect_timing = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.909721 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] barbican_service_user.insecure = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.909721 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] barbican_service_user.keyfile = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.909721 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] barbican_service_user.split_loggers = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.909721 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] barbican_service_user.timeout = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.909721 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] vault.approle_role_id = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.909721 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] vault.approle_secret_id = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.910282 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] vault.cafile = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.910282 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] vault.certfile = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.910844 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] vault.collect_timing = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.911660 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] vault.insecure = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.912387 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] vault.keyfile = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.913169 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] vault.kv_mountpoint = secret {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.913894 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] vault.kv_version = 2 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.914633 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] vault.namespace = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.915374 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] vault.root_token_id = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.915990 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] vault.split_loggers = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.916660 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] vault.ssl_ca_crt_file = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.917511 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] vault.timeout = None {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.918114 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] vault.use_ssl = False {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.918729 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] vault.vault_url = http://127.0.0.1:8200 {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} May 14 21:01:54.919273 ncdv-master-1 cinder-backup[123339]: DEBUG oslo_service.service [None req-46b1940b-c64c-43a7-abd1-f2d1a6bd9231 None None] ******************************************************************************** {{(pid=123339) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} May 14 21:01:54.922080 ncdv-master-1 cinder-backup[123784]: INFO cinder.keymgr.migration [None req-93541cc8-c186-4371-bda4-b0e4aba8416c None None] Not migrating encryption keys because the ConfKeyManager is still in use. May 14 21:01:54.951640 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.service [None req-6ac56b3a-184a-4236-a8ea-61c85dcf03d1 None None] Creating RPC server for service cinder-backup {{(pid=123784) start /opt/stack/cinder/cinder/service.py:231}} May 14 21:01:54.967758 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.service [None req-6ac56b3a-184a-4236-a8ea-61c85dcf03d1 None None] Pinning object versions for RPC server serializer to 1.39 {{(pid=123784) start /opt/stack/cinder/cinder/service.py:237}} May 14 21:03:17.016427 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:03:17.017063 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:04:17.012319 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:04:17.012996 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:05:17.017143 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:05:17.017772 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:06:17.022468 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:06:17.023107 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:07:17.019644 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:07:17.020320 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:08:17.024313 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:08:17.025005 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:09:17.029591 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:09:17.030259 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:10:17.029644 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:10:17.030278 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:11:17.035109 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:11:17.035926 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:12:17.039571 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:12:17.040710 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:13:17.039318 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:13:17.040060 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:14:17.045148 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:14:17.045828 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:15:17.042215 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:15:17.047505 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:16:17.046139 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:16:17.054821 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:17:17.048183 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:17:17.049441 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:18:17.045401 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:18:17.046369 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:20:17.045525 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:20:17.046530 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:21:17.046438 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:21:17.046438 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:22:17.047138 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:22:17.048055 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:23:17.044186 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:23:17.044988 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:24:17.050390 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:24:17.050390 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:25:17.048166 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:25:17.049211 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:26:17.045367 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:26:17.046074 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:27:17.051189 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:27:17.051919 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:28:17.049951 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:28:17.050682 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:29:17.054292 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:29:17.055199 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:29:53.996211 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] Create backup started, backup: fe83b34c-76e8-421b-a507-f43e0f25e8ce volume: c221be08-99e4-4f09-a858-6fffe81fd931 snapshot: f9966700-2a70-49c4-a5bc-d0c19d8baa1a. May 14 21:29:54.086645 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-14T21:29:53Z,data_timestamp=2022-05-14T21:29:49Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesSnapshotTestJSON-Backup-188718327',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=fe83b34c-76e8-421b-a507-f43e0f25e8ce,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='162610ac2697486d8a05f41713cc5ba2',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id='f9966700-2a70-49c4-a5bc-d0c19d8baa1a',status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='a8e967d05266477dbc5748ed77b8c69e',volume_id=c221be08-99e4-4f09-a858-6fffe81fd931) May 14 21:29:56.584024 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.utils [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.52', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:29:56.595720 ncdv-master-1 cinder-backup[123784]: INFO oslo.privsep.daemon [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] Running privsep helper: ['sudo', 'cinder-rootwrap', '/etc/cinder/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/cinder/cinder.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmp2p6hzh_t/privsep.sock'] May 14 21:29:56.694354 ncdv-master-1 sudo[146738]: ubuntu : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf privsep-helper --config-file /etc/cinder/cinder.conf --privsep_context os_brick.privileged.default --privsep_sock_path /tmp/tmp2p6hzh_t/privsep.sock May 14 21:29:56.703554 ncdv-master-1 sudo[146738]: pam_unix(sudo:session): session opened for user root by (uid=0) May 14 21:30:01.410504 ncdv-master-1 sudo[146738]: pam_unix(sudo:session): session closed for user root May 14 21:30:01.416557 ncdv-master-1 cinder-backup[123784]: INFO oslo.privsep.daemon [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] Spawned new privsep daemon via rootwrap May 14 21:30:01.417179 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] Accepted privsep connection to /tmp/tmp2p6hzh_t/privsep.sock {{(pid=123784) __init__ /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:362}} May 14 21:30:01.418674 ncdv-master-1 cinder-backup[123784]: INFO oslo.privsep.daemon [-] privsep daemon starting May 14 21:30:01.418947 ncdv-master-1 cinder-backup[123784]: INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 May 14 21:30:01.419212 ncdv-master-1 cinder-backup[123784]: INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_SYS_ADMIN/CAP_SYS_ADMIN/none May 14 21:30:01.419493 ncdv-master-1 cinder-backup[123784]: INFO oslo.privsep.daemon [-] privsep daemon running as pid 146745 May 14 21:30:01.421051 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[35b90804-75eb-47e7-a8dc-e77464de29be]: (2,) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:30:01.714150 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:30:01.749474 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.035s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:30:01.749816 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[03991d96-9e95-467f-aa51-502ba8b94f81]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:828e4c9f8c8c\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:30:01.752199 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:30:01.759650 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 14 21:30:01.792765 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[019d3c88-3549-4769-bfe3-40ad4202a1e3]: [Errno 2] No such file or directory: 'systool' {{(pid=146745) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 14 21:30:01.792765 ncdv-master-1 cinder-backup[123784]: Traceback (most recent call last): May 14 21:30:01.792765 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 14 21:30:01.792765 ncdv-master-1 cinder-backup[123784]: ret = func(*f_args, **f_kwargs) May 14 21:30:01.792765 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 14 21:30:01.792765 ncdv-master-1 cinder-backup[123784]: return func(*args, **kwargs) May 14 21:30:01.792765 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 14 21:30:01.792765 ncdv-master-1 cinder-backup[123784]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 14 21:30:01.792765 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 14 21:30:01.792765 ncdv-master-1 cinder-backup[123784]: return putils.execute(on_execute=on_execute, May 14 21:30:01.792765 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 14 21:30:01.792765 ncdv-master-1 cinder-backup[123784]: obj = subprocess.Popen(cmd, May 14 21:30:01.792765 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 14 21:30:01.792765 ncdv-master-1 cinder-backup[123784]: self._execute_child(args, executable, preexec_fn, close_fds, May 14 21:30:01.792765 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 14 21:30:01.792765 ncdv-master-1 cinder-backup[123784]: raise child_exception_type(errno_num, err_msg, err_filename) May 14 21:30:01.792765 ncdv-master-1 cinder-backup[123784]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 14 21:30:01.794381 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[019d3c88-3549-4769-bfe3-40ad4202a1e3]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:30:01.794487 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:30:01.801427 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 14 21:30:01.801971 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[03160388-0d86-430f-8a05-c36d482a0309]: [Errno 2] No such file or directory: 'systool' {{(pid=146745) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 14 21:30:01.801971 ncdv-master-1 cinder-backup[123784]: Traceback (most recent call last): May 14 21:30:01.801971 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 14 21:30:01.801971 ncdv-master-1 cinder-backup[123784]: ret = func(*f_args, **f_kwargs) May 14 21:30:01.801971 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 14 21:30:01.801971 ncdv-master-1 cinder-backup[123784]: return func(*args, **kwargs) May 14 21:30:01.801971 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 14 21:30:01.801971 ncdv-master-1 cinder-backup[123784]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 14 21:30:01.801971 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 14 21:30:01.801971 ncdv-master-1 cinder-backup[123784]: return putils.execute(on_execute=on_execute, May 14 21:30:01.801971 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 14 21:30:01.801971 ncdv-master-1 cinder-backup[123784]: obj = subprocess.Popen(cmd, May 14 21:30:01.801971 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 14 21:30:01.801971 ncdv-master-1 cinder-backup[123784]: self._execute_child(args, executable, preexec_fn, close_fds, May 14 21:30:01.801971 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 14 21:30:01.801971 ncdv-master-1 cinder-backup[123784]: raise child_exception_type(errno_num, err_msg, err_filename) May 14 21:30:01.801971 ncdv-master-1 cinder-backup[123784]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 14 21:30:01.803423 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[03160388-0d86-430f-8a05-c36d482a0309]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:30:01.804079 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:30:01.844775 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.039s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:30:01.846412 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[cefa509b-f599-4bea-b6ff-de8569bd8877]: (4, ('/dev/sda1\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:30:01.846924 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:30:01.917372 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.070s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:30:01.917765 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[40d97ee0-9928-419a-a784-f7d6258ff605]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:30:01.919298 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:30:01.930248 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.011s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:30:01.931985 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[076a06f9-b960-48ae-b893-0e331cd18fa5]: (4, ('04b32ac0-335b-a141-b9ee-83ea10e826d9\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:30:01.932670 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] Running cmd (subprocess): nvme version {{(pid=123784) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:30:01.984838 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] CMD "nvme version" returned: 0 in 0.052s {{(pid=123784) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:30:02.016685 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=123784) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 14 21:30:02.017065 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=123784) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 14 21:30:02.017437 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:5e2f7208-21be-4c5f-84d7-22d3a9f29e66 dsc: {{(pid=123784) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 14 21:30:02.018739 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.utils [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] <== get_connector_properties: return (5432ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.52', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:828e4c9f8c8c', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '04b32ac0-335b-a141-b9ee-83ea10e826d9', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:5e2f7208-21be-4c5f-84d7-22d3a9f29e66', 'found_dsc': ''} {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:30:02.845564 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connector [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] Factory for rbd on None {{(pid=123784) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 14 21:30:02.848145 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-d4a3485e-9088-48b6-ac8b-2122497b6eae', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'd4a3485e-9088-48b6-ac8b-2122497b6eae', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:30:02.849333 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.linuxrbd [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] opening connection to ceph cluster (timeout=-1). {{(pid=123784) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 14 21:30:03.115278 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] <== connect_volume: return (265ms) {'path': } {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:30:03.130378 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_db.sqlalchemy.engines [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123784) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 14 21:30:03.206530 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] Starting backup of volume='c221be08-99e4-4f09-a858-6fffe81fd931'. {{(pid=123784) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} May 14 21:30:03.239080 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=123784) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} May 14 21:30:03.239817 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] Creating backup base image='volume-c221be08-99e4-4f09-a858-6fffe81fd931.backup.base' for volume c221be08-99e4-4f09-a858-6fffe81fd931. {{(pid=123784) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:875}} May 14 21:30:06.969887 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] Copying data from volume c221be08-99e4-4f09-a858-6fffe81fd931. {{(pid=123784) _full_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:887}} May 14 21:30:07.245557 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] Transferring data between 'volume-c221be08-99e4-4f09-a858-6fffe81fd931' and 'volume-c221be08-99e4-4f09-a858-6fffe81fd931.backup.base' {{(pid=123784) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:417}} May 14 21:30:07.245944 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] 8 chunks of 134217728 bytes to be transferred {{(pid=123784) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:421}} May 14 21:30:17.052363 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:30:17.053027 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:30:19.377410 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] Transferred chunk 1 of 8 (10804K/s) {{(pid=123784) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} May 14 21:30:32.724182 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] Transferred chunk 2 of 8 (9821K/s) {{(pid=123784) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} May 14 21:30:45.868611 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] Transferred chunk 3 of 8 (9973K/s) {{(pid=123784) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} May 14 21:30:58.702581 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] Transferred chunk 4 of 8 (10214K/s) {{(pid=123784) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} May 14 21:31:10.090303 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] Transferred chunk 5 of 8 (11510K/s) {{(pid=123784) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} May 14 21:31:17.051092 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:31:17.051664 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:31:22.261335 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] Transferred chunk 6 of 8 (10769K/s) {{(pid=123784) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} May 14 21:31:34.256516 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] Transferred chunk 7 of 8 (10927K/s) {{(pid=123784) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} May 14 21:31:48.707870 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] Transferred chunk 8 of 8 (9070K/s) {{(pid=123784) _transfer_data /opt/stack/cinder/cinder/backup/drivers/ceph.py:440}} May 14 21:31:49.724110 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] Getting metadata type 'volume-base-metadata' {{(pid=123784) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 14 21:31:49.739680 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=123784) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 14 21:31:49.740100 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] Getting metadata type 'volume-metadata' {{(pid=123784) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 14 21:31:49.752080 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] No metadata type 'volume-metadata' available {{(pid=123784) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 14 21:31:49.752471 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] Getting metadata type 'volume-glance-metadata' {{(pid=123784) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 14 21:31:49.762218 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] No metadata type 'volume-glance-metadata' available {{(pid=123784) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} May 14 21:31:49.762771 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] Backing up metadata for volume c221be08-99e4-4f09-a858-6fffe81fd931. {{(pid=123784) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} May 14 21:31:50.006220 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] Backup 'fe83b34c-76e8-421b-a507-f43e0f25e8ce' of volume c221be08-99e4-4f09-a858-6fffe81fd931 finished. {{(pid=123784) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} May 14 21:31:50.008146 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-d4a3485e-9088-48b6-ac8b-2122497b6eae', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'd4a3485e-9088-48b6-ac8b-2122497b6eae', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:31:50.259551 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] <== disconnect_volume: return (245ms) None {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:31:50.827097 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-a5a75b09-1e06-4293-883f-ba15683bb6e6 tempest-VolumesSnapshotTestJSON-1533504236 None] Create backup finished. backup: fe83b34c-76e8-421b-a507-f43e0f25e8ce. May 14 21:31:51.724356 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-15b47af6-5b6a-44b7-a756-221e98741d28 tempest-VolumesSnapshotTestJSON-1533504236 None] Delete backup started, backup: fe83b34c-76e8-421b-a507-f43e0f25e8ce. May 14 21:31:51.726079 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-15b47af6-5b6a-44b7-a756-221e98741d28 tempest-VolumesSnapshotTestJSON-1533504236 None] Delete started for backup=fe83b34c-76e8-421b-a507-f43e0f25e8ce {{(pid=123784) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} May 14 21:31:51.726833 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-15b47af6-5b6a-44b7-a756-221e98741d28 tempest-VolumesSnapshotTestJSON-1533504236 None] Trying diff format basename='volume-c221be08-99e4-4f09-a858-6fffe81fd931.backup.fe83b34c-76e8-421b-a507-f43e0f25e8ce' for backup base image of volume c221be08-99e4-4f09-a858-6fffe81fd931. {{(pid=123784) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} May 14 21:31:51.796611 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-15b47af6-5b6a-44b7-a756-221e98741d28 tempest-VolumesSnapshotTestJSON-1533504236 None] Image 'volume-c221be08-99e4-4f09-a858-6fffe81fd931.backup.fe83b34c-76e8-421b-a507-f43e0f25e8ce' not found - trying diff format name {{(pid=123784) _rbd_image_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:693}} May 14 21:31:52.143209 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-15b47af6-5b6a-44b7-a756-221e98741d28 tempest-VolumesSnapshotTestJSON-1533504236 None] Looking for snapshot of backup base 'volume-c221be08-99e4-4f09-a858-6fffe81fd931.backup.base' {{(pid=123784) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 14 21:31:52.145057 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-15b47af6-5b6a-44b7-a756-221e98741d28 tempest-VolumesSnapshotTestJSON-1533504236 None] Backup base 'volume-c221be08-99e4-4f09-a858-6fffe81fd931.backup.base' has no snapshots {{(pid=123784) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:955}} May 14 21:31:52.145057 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-15b47af6-5b6a-44b7-a756-221e98741d28 tempest-VolumesSnapshotTestJSON-1533504236 None] No backup snapshot to delete {{(pid=123784) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:500}} May 14 21:31:52.451821 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.drivers.ceph [None req-15b47af6-5b6a-44b7-a756-221e98741d28 tempest-VolumesSnapshotTestJSON-1533504236 None] Deleting backup base image='volume-c221be08-99e4-4f09-a858-6fffe81fd931.backup.base' of volume c221be08-99e4-4f09-a858-6fffe81fd931. May 14 21:31:55.188073 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_db.sqlalchemy.engines [None req-85b1c192-0a07-4fed-b3a8-7524167e72b8 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123784) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 14 21:32:09.388154 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-15b47af6-5b6a-44b7-a756-221e98741d28 tempest-VolumesSnapshotTestJSON-1533504236 None] Base backup image='volume-c221be08-99e4-4f09-a858-6fffe81fd931.backup.base' of volume c221be08-99e4-4f09-a858-6fffe81fd931 deleted. {{(pid=123784) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} May 14 21:32:09.734035 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-15b47af6-5b6a-44b7-a756-221e98741d28 tempest-VolumesSnapshotTestJSON-1533504236 None] Delete of backup 'fe83b34c-76e8-421b-a507-f43e0f25e8ce' for volume 'c221be08-99e4-4f09-a858-6fffe81fd931' finished. {{(pid=123784) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} May 14 21:32:09.785761 ncdv-master-1 cinder-backup[123784]: WARNING cinder.quota [None req-15b47af6-5b6a-44b7-a756-221e98741d28 tempest-VolumesSnapshotTestJSON-1533504236 None] Deprecated: Default quota for resource: backups is set by the default quota flag: quota_backups, it is now deprecated. Please use the default quota class for default quota. May 14 21:32:09.787015 ncdv-master-1 cinder-backup[123784]: WARNING cinder.quota [None req-15b47af6-5b6a-44b7-a756-221e98741d28 tempest-VolumesSnapshotTestJSON-1533504236 None] Deprecated: Default quota for resource: backup_gigabytes is set by the default quota flag: quota_backup_gigabytes, it is now deprecated. Please use the default quota class for default quota. May 14 21:32:09.894934 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.quota [None req-15b47af6-5b6a-44b7-a756-221e98741d28 tempest-VolumesSnapshotTestJSON-1533504236 None] Created reservations ['d6a6b023-5578-4eaf-bad0-8e25396a7108', '307bd7dc-a30e-469d-bd59-b7c6b224f867'] {{(pid=123784) reserve /opt/stack/cinder/cinder/quota.py:755}} May 14 21:32:10.212677 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-15b47af6-5b6a-44b7-a756-221e98741d28 tempest-VolumesSnapshotTestJSON-1533504236 None] Delete backup finished, backup fe83b34c-76e8-421b-a507-f43e0f25e8ce deleted. May 14 21:32:17.056472 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:32:17.057152 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:33:17.053603 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:33:17.054236 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:34:17.052279 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:34:17.053561 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:35:17.050712 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:35:17.052014 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:36:17.053719 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:36:17.054352 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:37:17.056184 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:37:17.057058 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:38:17.060132 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:38:17.061823 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:39:17.060201 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:39:17.061257 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:40:17.064348 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:40:17.064999 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:40:26.673034 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] Create backup started, backup: 68940f14-10a2-4082-adee-123b6a3059f8 volume: af0dc711-750a-44a0-b81b-0754b812e053. May 14 21:40:26.745830 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-14T21:40:26Z,data_timestamp=2022-05-14T21:40:26Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsV39Test-Backup-1007209795',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=68940f14-10a2-4082-adee-123b6a3059f8,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='537c4ac112c846fcbbd4aa124542ab64',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='69c5e3ab4e494335839ffd7afb2b9c52',volume_id=af0dc711-750a-44a0-b81b-0754b812e053) May 14 21:40:26.784973 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.utils [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.52', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:40:26.786633 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:40:26.799976 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.013s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:40:26.800316 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[bb511513-458f-4598-ab10-179f1aa0d176]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:828e4c9f8c8c\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:40:26.804051 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:40:26.807876 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 14 21:40:26.808472 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[96eaaa2d-c73c-4330-91b4-0c4c4b576402]: [Errno 2] No such file or directory: 'systool' {{(pid=146745) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 14 21:40:26.808472 ncdv-master-1 cinder-backup[123784]: Traceback (most recent call last): May 14 21:40:26.808472 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 14 21:40:26.808472 ncdv-master-1 cinder-backup[123784]: ret = func(*f_args, **f_kwargs) May 14 21:40:26.808472 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 14 21:40:26.808472 ncdv-master-1 cinder-backup[123784]: return func(*args, **kwargs) May 14 21:40:26.808472 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 14 21:40:26.808472 ncdv-master-1 cinder-backup[123784]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 14 21:40:26.808472 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 14 21:40:26.808472 ncdv-master-1 cinder-backup[123784]: return putils.execute(on_execute=on_execute, May 14 21:40:26.808472 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 14 21:40:26.808472 ncdv-master-1 cinder-backup[123784]: obj = subprocess.Popen(cmd, May 14 21:40:26.808472 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 14 21:40:26.808472 ncdv-master-1 cinder-backup[123784]: self._execute_child(args, executable, preexec_fn, close_fds, May 14 21:40:26.808472 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 14 21:40:26.808472 ncdv-master-1 cinder-backup[123784]: raise child_exception_type(errno_num, err_msg, err_filename) May 14 21:40:26.808472 ncdv-master-1 cinder-backup[123784]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 14 21:40:26.809648 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[96eaaa2d-c73c-4330-91b4-0c4c4b576402]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:40:26.810302 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:40:26.817101 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 14 21:40:26.818130 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[578da044-c09a-457e-b107-f786807c5483]: [Errno 2] No such file or directory: 'systool' {{(pid=146745) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 14 21:40:26.818130 ncdv-master-1 cinder-backup[123784]: Traceback (most recent call last): May 14 21:40:26.818130 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 14 21:40:26.818130 ncdv-master-1 cinder-backup[123784]: ret = func(*f_args, **f_kwargs) May 14 21:40:26.818130 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 14 21:40:26.818130 ncdv-master-1 cinder-backup[123784]: return func(*args, **kwargs) May 14 21:40:26.818130 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 14 21:40:26.818130 ncdv-master-1 cinder-backup[123784]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 14 21:40:26.818130 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 14 21:40:26.818130 ncdv-master-1 cinder-backup[123784]: return putils.execute(on_execute=on_execute, May 14 21:40:26.818130 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 14 21:40:26.818130 ncdv-master-1 cinder-backup[123784]: obj = subprocess.Popen(cmd, May 14 21:40:26.818130 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 14 21:40:26.818130 ncdv-master-1 cinder-backup[123784]: self._execute_child(args, executable, preexec_fn, close_fds, May 14 21:40:26.818130 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 14 21:40:26.818130 ncdv-master-1 cinder-backup[123784]: raise child_exception_type(errno_num, err_msg, err_filename) May 14 21:40:26.818130 ncdv-master-1 cinder-backup[123784]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 14 21:40:26.819028 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[578da044-c09a-457e-b107-f786807c5483]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:40:26.822482 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:40:26.848739 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.028s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:40:26.849977 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[4507664b-52cd-4415-b594-a2666b5768fb]: (4, ('/dev/sda1\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:40:26.851350 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:40:26.877031 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.025s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:40:26.881981 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[6d56950a-51c0-4de9-a0cc-e2c9d5520346]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:40:26.881981 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:40:26.889792 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.010s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:40:26.890420 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[3a38bff2-d49e-4991-9556-25e607f204a4]: (4, ('04b32ac0-335b-a141-b9ee-83ea10e826d9\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:40:26.891442 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] Running cmd (subprocess): nvme version {{(pid=123784) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:40:26.913945 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] CMD "nvme version" returned: 0 in 0.022s {{(pid=123784) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:40:26.916275 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=123784) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 14 21:40:26.917055 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=123784) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 14 21:40:26.917572 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:5e2f7208-21be-4c5f-84d7-22d3a9f29e66 dsc: {{(pid=123784) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 14 21:40:26.918493 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.utils [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] <== get_connector_properties: return (133ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.52', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:828e4c9f8c8c', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '04b32ac0-335b-a141-b9ee-83ea10e826d9', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:5e2f7208-21be-4c5f-84d7-22d3a9f29e66', 'found_dsc': ''} {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:40:27.737557 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connector [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] Factory for rbd on None {{(pid=123784) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 14 21:40:27.739982 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-af0dc711-750a-44a0-b81b-0754b812e053', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'af0dc711-750a-44a0-b81b-0754b812e053', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:40:27.741096 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.linuxrbd [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] opening connection to ceph cluster (timeout=-1). {{(pid=123784) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 14 21:40:28.072805 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] <== connect_volume: return (331ms) {'path': } {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:40:28.084052 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_db.sqlalchemy.engines [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123784) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 14 21:40:28.170275 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] Starting backup of volume='af0dc711-750a-44a0-b81b-0754b812e053'. {{(pid=123784) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} May 14 21:40:28.175506 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] Volume file is RBD: attempting optimized backup {{(pid=123784) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}} May 14 21:40:28.227565 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] Creating base image 'volume-af0dc711-750a-44a0-b81b-0754b812e053.backup.68940f14-10a2-4082-adee-123b6a3059f8' {{(pid=123784) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}} May 14 21:40:28.228592 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=123784) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} May 14 21:40:29.463380 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] Using --from-snap 'None' for incremental backup of volume af0dc711-750a-44a0-b81b-0754b812e053. {{(pid=123784) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}} May 14 21:40:29.464051 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] Creating backup snapshot='backup.68940f14-10a2-4082-adee-123b6a3059f8.snap.1652564429.4633286' {{(pid=123784) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}} May 14 21:40:32.526991 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] Performing differential transfer from 'volume-af0dc711-750a-44a0-b81b-0754b812e053' to 'volume-af0dc711-750a-44a0-b81b-0754b812e053.backup.68940f14-10a2-4082-adee-123b6a3059f8' {{(pid=123784) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} May 14 21:40:32.527642 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_l_9z6kdh --pool volumes volumes/volume-af0dc711-750a-44a0-b81b-0754b812e053@backup.68940f14-10a2-4082-adee-123b6a3059f8.snap.1652564429.4633286 -' into... {{(pid=123784) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} May 14 21:40:32.527804 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-af0dc711-750a-44a0-b81b-0754b812e053.backup.68940f14-10a2-4082-adee-123b6a3059f8' {{(pid=123784) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} May 14 21:40:35.827859 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] Differential backup transfer completed in 3.3000s {{(pid=123784) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}} May 14 21:40:35.828564 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] Getting metadata type 'volume-base-metadata' {{(pid=123784) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 14 21:40:35.842133 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=123784) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 14 21:40:35.842641 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] Getting metadata type 'volume-metadata' {{(pid=123784) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 14 21:40:35.849143 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] No metadata type 'volume-metadata' available {{(pid=123784) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 14 21:40:35.849535 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] Getting metadata type 'volume-glance-metadata' {{(pid=123784) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 14 21:40:35.856501 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] No metadata type 'volume-glance-metadata' available {{(pid=123784) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} May 14 21:40:35.857058 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] Backing up metadata for volume af0dc711-750a-44a0-b81b-0754b812e053. {{(pid=123784) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} May 14 21:40:36.111506 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] Backup '68940f14-10a2-4082-adee-123b6a3059f8' of volume af0dc711-750a-44a0-b81b-0754b812e053 finished. {{(pid=123784) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} May 14 21:40:36.113587 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-af0dc711-750a-44a0-b81b-0754b812e053', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'af0dc711-750a-44a0-b81b-0754b812e053', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:40:36.864988 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] <== disconnect_volume: return (750ms) None {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:40:37.231205 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-84a90b1c-0963-413a-bb4f-7d5012d604dc tempest-VolumesBackupsV39Test-500200030 None] Create backup finished. backup: 68940f14-10a2-4082-adee-123b6a3059f8. May 14 21:40:38.532943 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-f2fab1ce-ad68-4c3e-8bb1-45df1c7e7cdb tempest-VolumesBackupsV39Test-500200030 None] Delete backup started, backup: 68940f14-10a2-4082-adee-123b6a3059f8. May 14 21:40:38.533647 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-f2fab1ce-ad68-4c3e-8bb1-45df1c7e7cdb tempest-VolumesBackupsV39Test-500200030 None] Delete started for backup=68940f14-10a2-4082-adee-123b6a3059f8 {{(pid=123784) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} May 14 21:40:38.534127 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-f2fab1ce-ad68-4c3e-8bb1-45df1c7e7cdb tempest-VolumesBackupsV39Test-500200030 None] Trying diff format basename='volume-af0dc711-750a-44a0-b81b-0754b812e053.backup.68940f14-10a2-4082-adee-123b6a3059f8' for backup base image of volume af0dc711-750a-44a0-b81b-0754b812e053. {{(pid=123784) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} May 14 21:40:38.666382 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-f2fab1ce-ad68-4c3e-8bb1-45df1c7e7cdb tempest-VolumesBackupsV39Test-500200030 None] Looking for snapshot of backup base 'volume-af0dc711-750a-44a0-b81b-0754b812e053.backup.68940f14-10a2-4082-adee-123b6a3059f8' {{(pid=123784) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 14 21:40:38.666382 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-f2fab1ce-ad68-4c3e-8bb1-45df1c7e7cdb tempest-VolumesBackupsV39Test-500200030 None] Found snapshot 'backup.68940f14-10a2-4082-adee-123b6a3059f8.snap.1652564429.4633286' {{(pid=123784) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} May 14 21:40:38.666382 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-f2fab1ce-ad68-4c3e-8bb1-45df1c7e7cdb tempest-VolumesBackupsV39Test-500200030 None] Deleting backup snapshot='backup.68940f14-10a2-4082-adee-123b6a3059f8.snap.1652564429.4633286' {{(pid=123784) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}} May 14 21:40:41.684857 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.drivers.ceph [None req-f2fab1ce-ad68-4c3e-8bb1-45df1c7e7cdb tempest-VolumesBackupsV39Test-500200030 None] Deleting backup base image='volume-af0dc711-750a-44a0-b81b-0754b812e053.backup.68940f14-10a2-4082-adee-123b6a3059f8' of volume af0dc711-750a-44a0-b81b-0754b812e053. May 14 21:40:45.215379 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_db.sqlalchemy.engines [None req-85b1c192-0a07-4fed-b3a8-7524167e72b8 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123784) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 14 21:40:48.605187 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-f2fab1ce-ad68-4c3e-8bb1-45df1c7e7cdb tempest-VolumesBackupsV39Test-500200030 None] Base backup image='volume-af0dc711-750a-44a0-b81b-0754b812e053.backup.68940f14-10a2-4082-adee-123b6a3059f8' of volume af0dc711-750a-44a0-b81b-0754b812e053 deleted. {{(pid=123784) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} May 14 21:40:48.865295 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-f2fab1ce-ad68-4c3e-8bb1-45df1c7e7cdb tempest-VolumesBackupsV39Test-500200030 None] Delete of backup '68940f14-10a2-4082-adee-123b6a3059f8' for volume 'af0dc711-750a-44a0-b81b-0754b812e053' finished. {{(pid=123784) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} May 14 21:40:49.032294 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.quota [None req-f2fab1ce-ad68-4c3e-8bb1-45df1c7e7cdb tempest-VolumesBackupsV39Test-500200030 None] Created reservations ['b2eb1118-10ad-4e1e-9d55-f7af397cd30c', 'e146032a-152a-4a2f-b681-aa3a05801f64'] {{(pid=123784) reserve /opt/stack/cinder/cinder/quota.py:755}} May 14 21:40:49.319056 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-f2fab1ce-ad68-4c3e-8bb1-45df1c7e7cdb tempest-VolumesBackupsV39Test-500200030 None] Delete backup finished, backup 68940f14-10a2-4082-adee-123b6a3059f8 deleted. May 14 21:41:17.059768 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:41:17.060411 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:42:17.065198 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:42:17.066248 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:43:17.066115 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:43:17.066761 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:43:28.764964 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] Create backup started, backup: 0b889b5f-0acf-4c35-9afa-d6d9e482fb22 volume: f2219050-2175-4ef5-b7e6-5497a584ba80. May 14 21:43:28.995877 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-14T21:43:28Z,data_timestamp=2022-05-14T21:43:28Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsTest-Backup-541977381',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=0b889b5f-0acf-4c35-9afa-d6d9e482fb22,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='8526cad5fef54772a988df44138c90fe',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='80f409708620474aa128bf8412fa54be',volume_id=f2219050-2175-4ef5-b7e6-5497a584ba80) May 14 21:43:29.064031 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.utils [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.52', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:43:29.065838 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:43:29.103133 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.037s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:43:29.104053 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[139fe1a6-0c26-4820-81ad-5867f070e4b1]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:828e4c9f8c8c\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:43:29.114179 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:43:29.134263 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 14 21:43:29.135457 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[9e7ee77a-610a-4ef7-b30e-e654ce0e68fa]: [Errno 2] No such file or directory: 'systool' {{(pid=146745) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 14 21:43:29.135457 ncdv-master-1 cinder-backup[123784]: Traceback (most recent call last): May 14 21:43:29.135457 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 14 21:43:29.135457 ncdv-master-1 cinder-backup[123784]: ret = func(*f_args, **f_kwargs) May 14 21:43:29.135457 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 14 21:43:29.135457 ncdv-master-1 cinder-backup[123784]: return func(*args, **kwargs) May 14 21:43:29.135457 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 14 21:43:29.135457 ncdv-master-1 cinder-backup[123784]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 14 21:43:29.135457 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 14 21:43:29.135457 ncdv-master-1 cinder-backup[123784]: return putils.execute(on_execute=on_execute, May 14 21:43:29.135457 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 14 21:43:29.135457 ncdv-master-1 cinder-backup[123784]: obj = subprocess.Popen(cmd, May 14 21:43:29.135457 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 14 21:43:29.135457 ncdv-master-1 cinder-backup[123784]: self._execute_child(args, executable, preexec_fn, close_fds, May 14 21:43:29.135457 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 14 21:43:29.135457 ncdv-master-1 cinder-backup[123784]: raise child_exception_type(errno_num, err_msg, err_filename) May 14 21:43:29.135457 ncdv-master-1 cinder-backup[123784]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 14 21:43:29.138227 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[9e7ee77a-610a-4ef7-b30e-e654ce0e68fa]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:43:29.138227 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:43:29.155412 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 14 21:43:29.157351 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[ede684c4-920e-4f16-9441-bdb364990774]: [Errno 2] No such file or directory: 'systool' {{(pid=146745) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 14 21:43:29.157351 ncdv-master-1 cinder-backup[123784]: Traceback (most recent call last): May 14 21:43:29.157351 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 14 21:43:29.157351 ncdv-master-1 cinder-backup[123784]: ret = func(*f_args, **f_kwargs) May 14 21:43:29.157351 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 14 21:43:29.157351 ncdv-master-1 cinder-backup[123784]: return func(*args, **kwargs) May 14 21:43:29.157351 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 14 21:43:29.157351 ncdv-master-1 cinder-backup[123784]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 14 21:43:29.157351 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 14 21:43:29.157351 ncdv-master-1 cinder-backup[123784]: return putils.execute(on_execute=on_execute, May 14 21:43:29.157351 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 14 21:43:29.157351 ncdv-master-1 cinder-backup[123784]: obj = subprocess.Popen(cmd, May 14 21:43:29.157351 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 14 21:43:29.157351 ncdv-master-1 cinder-backup[123784]: self._execute_child(args, executable, preexec_fn, close_fds, May 14 21:43:29.157351 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 14 21:43:29.157351 ncdv-master-1 cinder-backup[123784]: raise child_exception_type(errno_num, err_msg, err_filename) May 14 21:43:29.157351 ncdv-master-1 cinder-backup[123784]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 14 21:43:29.159625 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[ede684c4-920e-4f16-9441-bdb364990774]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:43:29.164162 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:43:29.183441 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.021s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:43:29.183845 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[d2944f89-5194-44b1-b121-99eb9a96c966]: (4, ('/dev/sda1\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:43:29.185320 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:43:29.233376 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.048s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:43:29.233825 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[618b2175-add6-4621-9aec-3eccebb4503a]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:43:29.235087 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:43:29.248866 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.013s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:43:29.249178 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[b2580575-5db1-4ec4-b09d-507948357ee8]: (4, ('04b32ac0-335b-a141-b9ee-83ea10e826d9\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:43:29.250138 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] Running cmd (subprocess): nvme version {{(pid=123784) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:43:29.278404 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] CMD "nvme version" returned: 0 in 0.028s {{(pid=123784) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:43:29.280364 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=123784) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 14 21:43:29.280675 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=123784) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 14 21:43:29.281060 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:5e2f7208-21be-4c5f-84d7-22d3a9f29e66 dsc: {{(pid=123784) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 14 21:43:29.281694 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.utils [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] <== get_connector_properties: return (216ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.52', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:828e4c9f8c8c', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '04b32ac0-335b-a141-b9ee-83ea10e826d9', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:5e2f7208-21be-4c5f-84d7-22d3a9f29e66', 'found_dsc': ''} {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:43:30.123410 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connector [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] Factory for rbd on None {{(pid=123784) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 14 21:43:30.125031 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-f2219050-2175-4ef5-b7e6-5497a584ba80', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'f2219050-2175-4ef5-b7e6-5497a584ba80', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:43:30.126101 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.linuxrbd [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] opening connection to ceph cluster (timeout=-1). {{(pid=123784) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 14 21:43:30.280960 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] <== connect_volume: return (154ms) {'path': } {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:43:30.290162 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_db.sqlalchemy.engines [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123784) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 14 21:43:30.399799 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] Starting backup of volume='f2219050-2175-4ef5-b7e6-5497a584ba80'. {{(pid=123784) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} May 14 21:43:30.399799 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] Volume file is RBD: attempting optimized backup {{(pid=123784) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}} May 14 21:43:30.450951 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] Creating base image 'volume-f2219050-2175-4ef5-b7e6-5497a584ba80.backup.0b889b5f-0acf-4c35-9afa-d6d9e482fb22' {{(pid=123784) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}} May 14 21:43:30.451424 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=123784) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} May 14 21:43:31.370283 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] Using --from-snap 'None' for incremental backup of volume f2219050-2175-4ef5-b7e6-5497a584ba80. {{(pid=123784) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}} May 14 21:43:31.370929 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] Creating backup snapshot='backup.0b889b5f-0acf-4c35-9afa-d6d9e482fb22.snap.1652564611.3703113' {{(pid=123784) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}} May 14 21:43:32.965399 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] Performing differential transfer from 'volume-f2219050-2175-4ef5-b7e6-5497a584ba80' to 'volume-f2219050-2175-4ef5-b7e6-5497a584ba80.backup.0b889b5f-0acf-4c35-9afa-d6d9e482fb22' {{(pid=123784) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} May 14 21:43:32.966113 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_3inbowqb --pool volumes volumes/volume-f2219050-2175-4ef5-b7e6-5497a584ba80@backup.0b889b5f-0acf-4c35-9afa-d6d9e482fb22.snap.1652564611.3703113 -' into... {{(pid=123784) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} May 14 21:43:32.966227 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-f2219050-2175-4ef5-b7e6-5497a584ba80.backup.0b889b5f-0acf-4c35-9afa-d6d9e482fb22' {{(pid=123784) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} May 14 21:43:35.561672 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] Differential backup transfer completed in 2.5960s {{(pid=123784) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}} May 14 21:43:35.562310 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] Getting metadata type 'volume-base-metadata' {{(pid=123784) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 14 21:43:35.574632 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=123784) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 14 21:43:35.575044 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] Getting metadata type 'volume-metadata' {{(pid=123784) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 14 21:43:35.581953 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] No metadata type 'volume-metadata' available {{(pid=123784) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 14 21:43:35.582534 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] Getting metadata type 'volume-glance-metadata' {{(pid=123784) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 14 21:43:35.590915 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] No metadata type 'volume-glance-metadata' available {{(pid=123784) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} May 14 21:43:35.591699 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] Backing up metadata for volume f2219050-2175-4ef5-b7e6-5497a584ba80. {{(pid=123784) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} May 14 21:43:35.839741 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] Backup '0b889b5f-0acf-4c35-9afa-d6d9e482fb22' of volume f2219050-2175-4ef5-b7e6-5497a584ba80 finished. {{(pid=123784) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} May 14 21:43:35.844621 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-f2219050-2175-4ef5-b7e6-5497a584ba80', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'f2219050-2175-4ef5-b7e6-5497a584ba80', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:43:36.110102 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] <== disconnect_volume: return (267ms) None {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:43:36.309854 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-54cd9ff4-e261-4944-89e5-ce1b9ea6714e tempest-VolumesBackupsTest-1778047115 None] Create backup finished. backup: 0b889b5f-0acf-4c35-9afa-d6d9e482fb22. May 14 21:43:37.069754 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-5619ea49-5d5f-4bd5-a516-3826d10ac5de tempest-VolumesBackupsTest-1778047115 None] Delete backup started, backup: 0b889b5f-0acf-4c35-9afa-d6d9e482fb22. May 14 21:43:37.072719 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-5619ea49-5d5f-4bd5-a516-3826d10ac5de tempest-VolumesBackupsTest-1778047115 None] Delete started for backup=0b889b5f-0acf-4c35-9afa-d6d9e482fb22 {{(pid=123784) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} May 14 21:43:37.074244 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-5619ea49-5d5f-4bd5-a516-3826d10ac5de tempest-VolumesBackupsTest-1778047115 None] Trying diff format basename='volume-f2219050-2175-4ef5-b7e6-5497a584ba80.backup.0b889b5f-0acf-4c35-9afa-d6d9e482fb22' for backup base image of volume f2219050-2175-4ef5-b7e6-5497a584ba80. {{(pid=123784) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} May 14 21:43:37.344659 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-5619ea49-5d5f-4bd5-a516-3826d10ac5de tempest-VolumesBackupsTest-1778047115 None] Looking for snapshot of backup base 'volume-f2219050-2175-4ef5-b7e6-5497a584ba80.backup.0b889b5f-0acf-4c35-9afa-d6d9e482fb22' {{(pid=123784) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 14 21:43:37.345050 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-5619ea49-5d5f-4bd5-a516-3826d10ac5de tempest-VolumesBackupsTest-1778047115 None] Found snapshot 'backup.0b889b5f-0acf-4c35-9afa-d6d9e482fb22.snap.1652564611.3703113' {{(pid=123784) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} May 14 21:43:37.345426 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-5619ea49-5d5f-4bd5-a516-3826d10ac5de tempest-VolumesBackupsTest-1778047115 None] Deleting backup snapshot='backup.0b889b5f-0acf-4c35-9afa-d6d9e482fb22.snap.1652564611.3703113' {{(pid=123784) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}} May 14 21:43:39.372582 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.drivers.ceph [None req-5619ea49-5d5f-4bd5-a516-3826d10ac5de tempest-VolumesBackupsTest-1778047115 None] Deleting backup base image='volume-f2219050-2175-4ef5-b7e6-5497a584ba80.backup.0b889b5f-0acf-4c35-9afa-d6d9e482fb22' of volume f2219050-2175-4ef5-b7e6-5497a584ba80. May 14 21:43:44.184646 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-5619ea49-5d5f-4bd5-a516-3826d10ac5de tempest-VolumesBackupsTest-1778047115 None] Base backup image='volume-f2219050-2175-4ef5-b7e6-5497a584ba80.backup.0b889b5f-0acf-4c35-9afa-d6d9e482fb22' of volume f2219050-2175-4ef5-b7e6-5497a584ba80 deleted. {{(pid=123784) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} May 14 21:43:44.375749 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-5619ea49-5d5f-4bd5-a516-3826d10ac5de tempest-VolumesBackupsTest-1778047115 None] Delete of backup '0b889b5f-0acf-4c35-9afa-d6d9e482fb22' for volume 'f2219050-2175-4ef5-b7e6-5497a584ba80' finished. {{(pid=123784) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} May 14 21:43:44.383024 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_db.sqlalchemy.engines [None req-5619ea49-5d5f-4bd5-a516-3826d10ac5de tempest-VolumesBackupsTest-1778047115 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123784) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 14 21:43:44.480676 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.quota [None req-5619ea49-5d5f-4bd5-a516-3826d10ac5de tempest-VolumesBackupsTest-1778047115 None] Created reservations ['589d2b09-e4f0-498f-af49-2a0eff148c7f', 'ae99a894-ca5b-4e46-9cbf-0726939c8387'] {{(pid=123784) reserve /opt/stack/cinder/cinder/quota.py:755}} May 14 21:43:44.639704 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-5619ea49-5d5f-4bd5-a516-3826d10ac5de tempest-VolumesBackupsTest-1778047115 None] Delete backup finished, backup 0b889b5f-0acf-4c35-9afa-d6d9e482fb22 deleted. May 14 21:44:10.263073 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] Create backup started, backup: da61f1fe-5ccb-4511-9c48-da6e422121bd volume: af532b40-a505-4682-982e-e62144e2824c. May 14 21:44:10.309050 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-14T21:44:09Z,data_timestamp=2022-05-14T21:44:09Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsTest-Backup-763507483',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=da61f1fe-5ccb-4511-9c48-da6e422121bd,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='8526cad5fef54772a988df44138c90fe',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='80f409708620474aa128bf8412fa54be',volume_id=af532b40-a505-4682-982e-e62144e2824c) May 14 21:44:10.348727 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.utils [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.52', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:44:10.350100 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:44:10.358282 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.008s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:44:10.358782 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[4fd4e432-7fc1-4b26-a643-7654f24333f6]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:828e4c9f8c8c\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:44:10.360217 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:44:10.366087 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 14 21:44:10.366697 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[6f657949-5441-47d1-9da4-9f24271aec63]: [Errno 2] No such file or directory: 'systool' {{(pid=146745) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 14 21:44:10.366697 ncdv-master-1 cinder-backup[123784]: Traceback (most recent call last): May 14 21:44:10.366697 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 14 21:44:10.366697 ncdv-master-1 cinder-backup[123784]: ret = func(*f_args, **f_kwargs) May 14 21:44:10.366697 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 14 21:44:10.366697 ncdv-master-1 cinder-backup[123784]: return func(*args, **kwargs) May 14 21:44:10.366697 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 14 21:44:10.366697 ncdv-master-1 cinder-backup[123784]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 14 21:44:10.366697 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 14 21:44:10.366697 ncdv-master-1 cinder-backup[123784]: return putils.execute(on_execute=on_execute, May 14 21:44:10.366697 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 14 21:44:10.366697 ncdv-master-1 cinder-backup[123784]: obj = subprocess.Popen(cmd, May 14 21:44:10.366697 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 14 21:44:10.366697 ncdv-master-1 cinder-backup[123784]: self._execute_child(args, executable, preexec_fn, close_fds, May 14 21:44:10.366697 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 14 21:44:10.366697 ncdv-master-1 cinder-backup[123784]: raise child_exception_type(errno_num, err_msg, err_filename) May 14 21:44:10.366697 ncdv-master-1 cinder-backup[123784]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 14 21:44:10.368036 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[6f657949-5441-47d1-9da4-9f24271aec63]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:44:10.368276 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:44:10.374516 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 14 21:44:10.374795 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[c1296955-2c02-4d6e-9058-b7a49a6f1da1]: [Errno 2] No such file or directory: 'systool' {{(pid=146745) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 14 21:44:10.374795 ncdv-master-1 cinder-backup[123784]: Traceback (most recent call last): May 14 21:44:10.374795 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 14 21:44:10.374795 ncdv-master-1 cinder-backup[123784]: ret = func(*f_args, **f_kwargs) May 14 21:44:10.374795 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 14 21:44:10.374795 ncdv-master-1 cinder-backup[123784]: return func(*args, **kwargs) May 14 21:44:10.374795 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 14 21:44:10.374795 ncdv-master-1 cinder-backup[123784]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 14 21:44:10.374795 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 14 21:44:10.374795 ncdv-master-1 cinder-backup[123784]: return putils.execute(on_execute=on_execute, May 14 21:44:10.374795 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 14 21:44:10.374795 ncdv-master-1 cinder-backup[123784]: obj = subprocess.Popen(cmd, May 14 21:44:10.374795 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 14 21:44:10.374795 ncdv-master-1 cinder-backup[123784]: self._execute_child(args, executable, preexec_fn, close_fds, May 14 21:44:10.374795 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 14 21:44:10.374795 ncdv-master-1 cinder-backup[123784]: raise child_exception_type(errno_num, err_msg, err_filename) May 14 21:44:10.374795 ncdv-master-1 cinder-backup[123784]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 14 21:44:10.376048 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[c1296955-2c02-4d6e-9058-b7a49a6f1da1]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:44:10.376543 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:44:10.384872 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.008s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:44:10.385170 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[62398f1d-c850-4f90-95ad-7dfa2d16a9f8]: (4, ('/dev/sda1\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:44:10.386524 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:44:10.395183 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.009s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:44:10.395769 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[6b4afc60-34c1-4f2f-8595-b8a5c18db5eb]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:44:10.396910 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:44:10.405533 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.008s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:44:10.408933 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[e892ffb0-cc20-410e-ada1-f47f920e3693]: (4, ('04b32ac0-335b-a141-b9ee-83ea10e826d9\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:44:10.408933 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] Running cmd (subprocess): nvme version {{(pid=123784) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:44:10.426176 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] CMD "nvme version" returned: 0 in 0.019s {{(pid=123784) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:44:10.427997 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=123784) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 14 21:44:10.428375 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=123784) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 14 21:44:10.428746 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:5e2f7208-21be-4c5f-84d7-22d3a9f29e66 dsc: {{(pid=123784) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 14 21:44:10.429397 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.utils [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] <== get_connector_properties: return (80ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.52', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:828e4c9f8c8c', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '04b32ac0-335b-a141-b9ee-83ea10e826d9', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:5e2f7208-21be-4c5f-84d7-22d3a9f29e66', 'found_dsc': ''} {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:44:11.112583 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connector [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] Factory for rbd on None {{(pid=123784) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 14 21:44:11.114165 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-af532b40-a505-4682-982e-e62144e2824c', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'af532b40-a505-4682-982e-e62144e2824c', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:44:11.115230 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.linuxrbd [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] opening connection to ceph cluster (timeout=-1). {{(pid=123784) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 14 21:44:11.331390 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] <== connect_volume: return (216ms) {'path': } {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:44:11.340101 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_db.sqlalchemy.engines [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123784) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 14 21:44:11.416891 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] Starting backup of volume='af532b40-a505-4682-982e-e62144e2824c'. {{(pid=123784) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} May 14 21:44:11.417713 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] Volume file is RBD: attempting optimized backup {{(pid=123784) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}} May 14 21:44:11.438465 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] Creating base image 'volume-af532b40-a505-4682-982e-e62144e2824c.backup.da61f1fe-5ccb-4511-9c48-da6e422121bd' {{(pid=123784) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}} May 14 21:44:11.439260 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=123784) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} May 14 21:44:12.397153 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] Using --from-snap 'None' for incremental backup of volume af532b40-a505-4682-982e-e62144e2824c. {{(pid=123784) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}} May 14 21:44:12.398026 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] Creating backup snapshot='backup.da61f1fe-5ccb-4511-9c48-da6e422121bd.snap.1652564652.3971' {{(pid=123784) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}} May 14 21:44:14.562087 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] Performing differential transfer from 'volume-af532b40-a505-4682-982e-e62144e2824c' to 'volume-af532b40-a505-4682-982e-e62144e2824c.backup.da61f1fe-5ccb-4511-9c48-da6e422121bd' {{(pid=123784) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} May 14 21:44:14.562705 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_4vimz2eo --pool volumes volumes/volume-af532b40-a505-4682-982e-e62144e2824c@backup.da61f1fe-5ccb-4511-9c48-da6e422121bd.snap.1652564652.3971 -' into... {{(pid=123784) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} May 14 21:44:14.563146 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-af532b40-a505-4682-982e-e62144e2824c.backup.da61f1fe-5ccb-4511-9c48-da6e422121bd' {{(pid=123784) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} May 14 21:44:17.069560 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:44:17.071387 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:45:02.384669 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] Differential backup transfer completed in 47.8225s {{(pid=123784) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}} May 14 21:45:02.385415 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] Getting metadata type 'volume-base-metadata' {{(pid=123784) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 14 21:45:02.397423 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=123784) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 14 21:45:02.398077 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] Getting metadata type 'volume-metadata' {{(pid=123784) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 14 21:45:02.404362 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] No metadata type 'volume-metadata' available {{(pid=123784) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 14 21:45:02.404736 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] Getting metadata type 'volume-glance-metadata' {{(pid=123784) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 14 21:45:02.412046 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=123784) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}} May 14 21:45:02.412622 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] Backing up metadata for volume af532b40-a505-4682-982e-e62144e2824c. {{(pid=123784) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} May 14 21:45:02.562038 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] Backup 'da61f1fe-5ccb-4511-9c48-da6e422121bd' of volume af532b40-a505-4682-982e-e62144e2824c finished. {{(pid=123784) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} May 14 21:45:02.564705 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-af532b40-a505-4682-982e-e62144e2824c', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'af532b40-a505-4682-982e-e62144e2824c', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:45:03.058863 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] <== disconnect_volume: return (493ms) None {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:45:03.618616 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-c67c3e81-7926-4cd4-a64c-737fb132440d tempest-VolumesBackupsTest-1778047115 None] Create backup finished. backup: da61f1fe-5ccb-4511-9c48-da6e422121bd. May 14 21:45:06.754947 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] Restore backup started, backup: da61f1fe-5ccb-4511-9c48-da6e422121bd volume: fe2b0773-b648-40ed-8148-08fa2937605a. May 14 21:45:06.776998 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.utils [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.52', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:45:06.788048 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:45:06.793770 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.012s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:45:06.796191 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[0d0549b1-330f-4549-b419-da4533e20082]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:828e4c9f8c8c\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:45:06.798909 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:45:06.828013 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 14 21:45:06.828013 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[fb037f3a-806f-4a21-b4b2-dd0d9bacc132]: [Errno 2] No such file or directory: 'systool' {{(pid=146745) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 14 21:45:06.828013 ncdv-master-1 cinder-backup[123784]: Traceback (most recent call last): May 14 21:45:06.828013 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 14 21:45:06.828013 ncdv-master-1 cinder-backup[123784]: ret = func(*f_args, **f_kwargs) May 14 21:45:06.828013 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 14 21:45:06.828013 ncdv-master-1 cinder-backup[123784]: return func(*args, **kwargs) May 14 21:45:06.828013 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 14 21:45:06.828013 ncdv-master-1 cinder-backup[123784]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 14 21:45:06.828013 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 14 21:45:06.828013 ncdv-master-1 cinder-backup[123784]: return putils.execute(on_execute=on_execute, May 14 21:45:06.828013 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 14 21:45:06.828013 ncdv-master-1 cinder-backup[123784]: obj = subprocess.Popen(cmd, May 14 21:45:06.828013 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 14 21:45:06.828013 ncdv-master-1 cinder-backup[123784]: self._execute_child(args, executable, preexec_fn, close_fds, May 14 21:45:06.828013 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 14 21:45:06.828013 ncdv-master-1 cinder-backup[123784]: raise child_exception_type(errno_num, err_msg, err_filename) May 14 21:45:06.828013 ncdv-master-1 cinder-backup[123784]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 14 21:45:06.828013 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[fb037f3a-806f-4a21-b4b2-dd0d9bacc132]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:45:06.829277 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:45:06.834160 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 14 21:45:06.834611 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[39daf300-9201-40b7-a813-b7f0caaaf8d1]: [Errno 2] No such file or directory: 'systool' {{(pid=146745) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 14 21:45:06.834611 ncdv-master-1 cinder-backup[123784]: Traceback (most recent call last): May 14 21:45:06.834611 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 14 21:45:06.834611 ncdv-master-1 cinder-backup[123784]: ret = func(*f_args, **f_kwargs) May 14 21:45:06.834611 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 14 21:45:06.834611 ncdv-master-1 cinder-backup[123784]: return func(*args, **kwargs) May 14 21:45:06.834611 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 14 21:45:06.834611 ncdv-master-1 cinder-backup[123784]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 14 21:45:06.834611 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 14 21:45:06.834611 ncdv-master-1 cinder-backup[123784]: return putils.execute(on_execute=on_execute, May 14 21:45:06.834611 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 14 21:45:06.834611 ncdv-master-1 cinder-backup[123784]: obj = subprocess.Popen(cmd, May 14 21:45:06.834611 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 14 21:45:06.834611 ncdv-master-1 cinder-backup[123784]: self._execute_child(args, executable, preexec_fn, close_fds, May 14 21:45:06.834611 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 14 21:45:06.834611 ncdv-master-1 cinder-backup[123784]: raise child_exception_type(errno_num, err_msg, err_filename) May 14 21:45:06.834611 ncdv-master-1 cinder-backup[123784]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 14 21:45:06.835933 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[39daf300-9201-40b7-a813-b7f0caaaf8d1]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:45:06.840120 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:45:06.857566 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.017s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:45:06.857885 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[3672ebc9-782f-4727-a4bf-260205d2acf3]: (4, ('/dev/sda1\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:45:06.859223 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:45:06.872296 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.013s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:45:06.872708 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[84ca6fb7-eb5a-4f5d-9c93-d964bbdfa641]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:45:06.874118 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:45:06.884122 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.010s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:45:06.884458 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[72938227-3d1a-4b51-b0c5-a07e16f30a47]: (4, ('04b32ac0-335b-a141-b9ee-83ea10e826d9\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:45:06.885350 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] Running cmd (subprocess): nvme version {{(pid=123784) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:45:06.905487 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] CMD "nvme version" returned: 0 in 0.020s {{(pid=123784) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:45:06.907872 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=123784) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 14 21:45:06.908350 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=123784) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 14 21:45:06.908718 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:5e2f7208-21be-4c5f-84d7-22d3a9f29e66 dsc: {{(pid=123784) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 14 21:45:06.909367 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.utils [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] <== get_connector_properties: return (131ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.52', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:828e4c9f8c8c', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '04b32ac0-335b-a141-b9ee-83ea10e826d9', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:5e2f7208-21be-4c5f-84d7-22d3a9f29e66', 'found_dsc': ''} {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:45:07.734019 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connector [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] Factory for rbd on None {{(pid=123784) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 14 21:45:07.736268 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-fe2b0773-b648-40ed-8148-08fa2937605a', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'fe2b0773-b648-40ed-8148-08fa2937605a', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:45:07.737334 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.linuxrbd [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] opening connection to ceph cluster (timeout=-1). {{(pid=123784) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 14 21:45:08.193103 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] <== connect_volume: return (455ms) {'path': } {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:45:08.201800 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_db.sqlalchemy.engines [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123784) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 14 21:45:08.214723 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] Starting restore from Ceph backup=da61f1fe-5ccb-4511-9c48-da6e422121bd to volume=volume-fe2b0773-b648-40ed-8148-08fa2937605a {{(pid=123784) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1343}} May 14 21:45:08.300454 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] Looking for snapshot of backup base 'volume-af532b40-a505-4682-982e-e62144e2824c.backup.da61f1fe-5ccb-4511-9c48-da6e422121bd' {{(pid=123784) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 14 21:45:08.301280 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] Found snapshot 'backup.da61f1fe-5ccb-4511-9c48-da6e422121bd.snap.1652564652.3971' {{(pid=123784) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} May 14 21:45:08.313771 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] Volume file is RBD. {{(pid=123784) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1245}} May 14 21:45:08.375061 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] Attempting differential restore. {{(pid=123784) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1298}} May 14 21:45:08.375604 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] Attempting incremental restore from base='volume-af532b40-a505-4682-982e-e62144e2824c.backup.da61f1fe-5ccb-4511-9c48-da6e422121bd' snap='backup.da61f1fe-5ccb-4511-9c48-da6e422121bd.snap.1652564652.3971' {{(pid=123784) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1145}} May 14 21:45:08.376007 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] Performing differential transfer from 'volume-af532b40-a505-4682-982e-e62144e2824c.backup.da61f1fe-5ccb-4511-9c48-da6e422121bd' to 'volume-fe2b0773-b648-40ed-8148-08fa2937605a' {{(pid=123784) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} May 14 21:45:08.376419 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-af532b40-a505-4682-982e-e62144e2824c.backup.da61f1fe-5ccb-4511-9c48-da6e422121bd@backup.da61f1fe-5ccb-4511-9c48-da6e422121bd.snap.1652564652.3971 -' into... {{(pid=123784) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} May 14 21:45:08.376777 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_xfz4skpz --pool volumes - volumes/volume-fe2b0773-b648-40ed-8148-08fa2937605a' {{(pid=123784) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} May 14 21:45:17.066676 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:45:17.067414 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:45:52.156384 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] Restore transfer completed in 43.7801s {{(pid=123784) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1167}} May 14 21:45:52.157388 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=123784) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1354}} May 14 21:45:52.180488 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] Restoring volume base metadata {{(pid=123784) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} May 14 21:45:52.266197 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] No metadata of type 'volume-metadata' to restore {{(pid=123784) put /opt/stack/cinder/cinder/backup/driver.py:345}} May 14 21:45:52.266492 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] Restoring volume glance metadata {{(pid=123784) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}} May 14 21:45:53.395469 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] Restore to volume fe2b0773-b648-40ed-8148-08fa2937605a finished successfully. {{(pid=123784) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1361}} May 14 21:45:53.397367 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-fe2b0773-b648-40ed-8148-08fa2937605a', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'fe2b0773-b648-40ed-8148-08fa2937605a', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:45:53.532726 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] <== disconnect_volume: return (134ms) None {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:45:53.589881 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.manager [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] Encryption key ID for volume fe2b0773-b648-40ed-8148-08fa2937605a already matches encryption key ID in backup da61f1fe-5ccb-4511-9c48-da6e422121bd. {{(pid=123784) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} May 14 21:45:53.889559 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-37b0f0d1-757c-43bd-b041-3b2be08496d9 tempest-VolumesBackupsTest-1778047115 None] Finished restoring backup da61f1fe-5ccb-4511-9c48-da6e422121bd to volume fe2b0773-b648-40ed-8148-08fa2937605a. May 14 21:46:05.297533 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-13bda21a-c2d2-4136-97f5-a8f6d0141ef6 tempest-VolumesBackupsTest-1778047115 None] Delete backup started, backup: da61f1fe-5ccb-4511-9c48-da6e422121bd. May 14 21:46:05.299009 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-13bda21a-c2d2-4136-97f5-a8f6d0141ef6 tempest-VolumesBackupsTest-1778047115 None] Delete started for backup=da61f1fe-5ccb-4511-9c48-da6e422121bd {{(pid=123784) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} May 14 21:46:05.299685 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-13bda21a-c2d2-4136-97f5-a8f6d0141ef6 tempest-VolumesBackupsTest-1778047115 None] Trying diff format basename='volume-af532b40-a505-4682-982e-e62144e2824c.backup.da61f1fe-5ccb-4511-9c48-da6e422121bd' for backup base image of volume af532b40-a505-4682-982e-e62144e2824c. {{(pid=123784) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} May 14 21:46:05.496090 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-13bda21a-c2d2-4136-97f5-a8f6d0141ef6 tempest-VolumesBackupsTest-1778047115 None] Looking for snapshot of backup base 'volume-af532b40-a505-4682-982e-e62144e2824c.backup.da61f1fe-5ccb-4511-9c48-da6e422121bd' {{(pid=123784) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 14 21:46:05.496562 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-13bda21a-c2d2-4136-97f5-a8f6d0141ef6 tempest-VolumesBackupsTest-1778047115 None] Found snapshot 'backup.da61f1fe-5ccb-4511-9c48-da6e422121bd.snap.1652564652.3971' {{(pid=123784) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} May 14 21:46:05.497050 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-13bda21a-c2d2-4136-97f5-a8f6d0141ef6 tempest-VolumesBackupsTest-1778047115 None] Deleting backup snapshot='backup.da61f1fe-5ccb-4511-9c48-da6e422121bd.snap.1652564652.3971' {{(pid=123784) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}} May 14 21:46:08.204930 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.drivers.ceph [None req-13bda21a-c2d2-4136-97f5-a8f6d0141ef6 tempest-VolumesBackupsTest-1778047115 None] Deleting backup base image='volume-af532b40-a505-4682-982e-e62144e2824c.backup.da61f1fe-5ccb-4511-9c48-da6e422121bd' of volume af532b40-a505-4682-982e-e62144e2824c. May 14 21:46:15.266395 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_db.sqlalchemy.engines [None req-85b1c192-0a07-4fed-b3a8-7524167e72b8 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123784) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 14 21:46:16.955940 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-13bda21a-c2d2-4136-97f5-a8f6d0141ef6 tempest-VolumesBackupsTest-1778047115 None] Base backup image='volume-af532b40-a505-4682-982e-e62144e2824c.backup.da61f1fe-5ccb-4511-9c48-da6e422121bd' of volume af532b40-a505-4682-982e-e62144e2824c deleted. {{(pid=123784) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} May 14 21:46:17.062356 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:46:17.062670 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:46:17.160621 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-13bda21a-c2d2-4136-97f5-a8f6d0141ef6 tempest-VolumesBackupsTest-1778047115 None] Delete of backup 'da61f1fe-5ccb-4511-9c48-da6e422121bd' for volume 'af532b40-a505-4682-982e-e62144e2824c' finished. {{(pid=123784) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} May 14 21:46:17.249037 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.quota [None req-13bda21a-c2d2-4136-97f5-a8f6d0141ef6 tempest-VolumesBackupsTest-1778047115 None] Created reservations ['236cd191-8ec6-44c2-b97f-edabdf63fe30', 'd43e2f43-9e2e-450e-b8c1-902b5f34973e'] {{(pid=123784) reserve /opt/stack/cinder/cinder/quota.py:755}} May 14 21:46:17.431896 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-13bda21a-c2d2-4136-97f5-a8f6d0141ef6 tempest-VolumesBackupsTest-1778047115 None] Delete backup finished, backup da61f1fe-5ccb-4511-9c48-da6e422121bd deleted. May 14 21:46:59.517388 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] Create backup started, backup: 80f8bfbe-cfe9-410c-a773-475f03a46ab4 volume: ef327fb0-87a3-4d7a-8e78-931cec53ab4d. May 14 21:46:59.557491 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-14T21:46:59Z,data_timestamp=2022-05-14T21:46:59Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsAdminTest-Backup-1246479446',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=80f8bfbe-cfe9-410c-a773-475f03a46ab4,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='70f3e8ff1b404e28aa96706c849a21e4',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='7ca5af3c599842d39df5edf9876c892a',volume_id=ef327fb0-87a3-4d7a-8e78-931cec53ab4d) May 14 21:46:59.597720 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.utils [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.52', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:46:59.599227 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:46:59.608098 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.009s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:46:59.608474 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[4c401306-cf16-4989-9e16-1b480ac9bb06]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:828e4c9f8c8c\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:46:59.609746 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:46:59.615374 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 14 21:46:59.616044 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[f1b394eb-750d-4ba1-99ef-f799bb80d454]: [Errno 2] No such file or directory: 'systool' {{(pid=146745) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 14 21:46:59.616044 ncdv-master-1 cinder-backup[123784]: Traceback (most recent call last): May 14 21:46:59.616044 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 14 21:46:59.616044 ncdv-master-1 cinder-backup[123784]: ret = func(*f_args, **f_kwargs) May 14 21:46:59.616044 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 14 21:46:59.616044 ncdv-master-1 cinder-backup[123784]: return func(*args, **kwargs) May 14 21:46:59.616044 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 14 21:46:59.616044 ncdv-master-1 cinder-backup[123784]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 14 21:46:59.616044 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 14 21:46:59.616044 ncdv-master-1 cinder-backup[123784]: return putils.execute(on_execute=on_execute, May 14 21:46:59.616044 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 14 21:46:59.616044 ncdv-master-1 cinder-backup[123784]: obj = subprocess.Popen(cmd, May 14 21:46:59.616044 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 14 21:46:59.616044 ncdv-master-1 cinder-backup[123784]: self._execute_child(args, executable, preexec_fn, close_fds, May 14 21:46:59.616044 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 14 21:46:59.616044 ncdv-master-1 cinder-backup[123784]: raise child_exception_type(errno_num, err_msg, err_filename) May 14 21:46:59.616044 ncdv-master-1 cinder-backup[123784]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 14 21:46:59.617400 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[f1b394eb-750d-4ba1-99ef-f799bb80d454]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:46:59.617491 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:46:59.623626 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 14 21:46:59.624196 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[8ff790f8-020c-4043-9b61-81633db5533a]: [Errno 2] No such file or directory: 'systool' {{(pid=146745) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 14 21:46:59.624196 ncdv-master-1 cinder-backup[123784]: Traceback (most recent call last): May 14 21:46:59.624196 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 14 21:46:59.624196 ncdv-master-1 cinder-backup[123784]: ret = func(*f_args, **f_kwargs) May 14 21:46:59.624196 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 14 21:46:59.624196 ncdv-master-1 cinder-backup[123784]: return func(*args, **kwargs) May 14 21:46:59.624196 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 14 21:46:59.624196 ncdv-master-1 cinder-backup[123784]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 14 21:46:59.624196 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 14 21:46:59.624196 ncdv-master-1 cinder-backup[123784]: return putils.execute(on_execute=on_execute, May 14 21:46:59.624196 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 14 21:46:59.624196 ncdv-master-1 cinder-backup[123784]: obj = subprocess.Popen(cmd, May 14 21:46:59.624196 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 14 21:46:59.624196 ncdv-master-1 cinder-backup[123784]: self._execute_child(args, executable, preexec_fn, close_fds, May 14 21:46:59.624196 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 14 21:46:59.624196 ncdv-master-1 cinder-backup[123784]: raise child_exception_type(errno_num, err_msg, err_filename) May 14 21:46:59.624196 ncdv-master-1 cinder-backup[123784]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 14 21:46:59.625299 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[8ff790f8-020c-4043-9b61-81633db5533a]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:46:59.626076 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:46:59.634057 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.008s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:46:59.634374 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[42470e15-8e1a-4624-926a-d5b65ea6ff1f]: (4, ('/dev/sda1\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:46:59.635706 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:46:59.643903 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.008s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:46:59.644228 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[5c9857b5-c5eb-4a83-80c1-eea3483dce39]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:46:59.645329 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:46:59.653244 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.008s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:46:59.653661 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[c178a6fb-a5c0-449f-9a0e-235fcd25593b]: (4, ('04b32ac0-335b-a141-b9ee-83ea10e826d9\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:46:59.654401 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] Running cmd (subprocess): nvme version {{(pid=123784) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:46:59.673999 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] CMD "nvme version" returned: 0 in 0.019s {{(pid=123784) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:46:59.675689 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=123784) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 14 21:46:59.676072 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=123784) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 14 21:46:59.676441 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:5e2f7208-21be-4c5f-84d7-22d3a9f29e66 dsc: {{(pid=123784) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 14 21:46:59.677081 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.utils [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] <== get_connector_properties: return (78ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.52', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:828e4c9f8c8c', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '04b32ac0-335b-a141-b9ee-83ea10e826d9', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:5e2f7208-21be-4c5f-84d7-22d3a9f29e66', 'found_dsc': ''} {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:47:00.395448 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connector [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] Factory for rbd on None {{(pid=123784) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 14 21:47:00.397132 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-ef327fb0-87a3-4d7a-8e78-931cec53ab4d', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'ef327fb0-87a3-4d7a-8e78-931cec53ab4d', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:47:00.398235 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.linuxrbd [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] opening connection to ceph cluster (timeout=-1). {{(pid=123784) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 14 21:47:00.545115 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] <== connect_volume: return (146ms) {'path': } {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:47:00.553503 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_db.sqlalchemy.engines [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123784) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 14 21:47:00.625202 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] Starting backup of volume='ef327fb0-87a3-4d7a-8e78-931cec53ab4d'. {{(pid=123784) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} May 14 21:47:00.626020 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] Volume file is RBD: attempting optimized backup {{(pid=123784) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}} May 14 21:47:00.647137 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] Creating base image 'volume-ef327fb0-87a3-4d7a-8e78-931cec53ab4d.backup.80f8bfbe-cfe9-410c-a773-475f03a46ab4' {{(pid=123784) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}} May 14 21:47:00.647669 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=123784) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} May 14 21:47:01.063073 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] Using --from-snap 'None' for incremental backup of volume ef327fb0-87a3-4d7a-8e78-931cec53ab4d. {{(pid=123784) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}} May 14 21:47:01.063518 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] Creating backup snapshot='backup.80f8bfbe-cfe9-410c-a773-475f03a46ab4.snap.1652564821.0631037' {{(pid=123784) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}} May 14 21:47:02.027467 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] Performing differential transfer from 'volume-ef327fb0-87a3-4d7a-8e78-931cec53ab4d' to 'volume-ef327fb0-87a3-4d7a-8e78-931cec53ab4d.backup.80f8bfbe-cfe9-410c-a773-475f03a46ab4' {{(pid=123784) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} May 14 21:47:02.028345 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_m_qn6lk8 --pool volumes volumes/volume-ef327fb0-87a3-4d7a-8e78-931cec53ab4d@backup.80f8bfbe-cfe9-410c-a773-475f03a46ab4.snap.1652564821.0631037 -' into... {{(pid=123784) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} May 14 21:47:02.028345 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-ef327fb0-87a3-4d7a-8e78-931cec53ab4d.backup.80f8bfbe-cfe9-410c-a773-475f03a46ab4' {{(pid=123784) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} May 14 21:47:03.706553 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] Differential backup transfer completed in 1.6787s {{(pid=123784) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}} May 14 21:47:03.707199 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] Getting metadata type 'volume-base-metadata' {{(pid=123784) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 14 21:47:03.720519 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=123784) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 14 21:47:03.720935 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] Getting metadata type 'volume-metadata' {{(pid=123784) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 14 21:47:03.727595 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] No metadata type 'volume-metadata' available {{(pid=123784) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 14 21:47:03.727992 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] Getting metadata type 'volume-glance-metadata' {{(pid=123784) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 14 21:47:03.734691 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] No metadata type 'volume-glance-metadata' available {{(pid=123784) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} May 14 21:47:03.735241 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] Backing up metadata for volume ef327fb0-87a3-4d7a-8e78-931cec53ab4d. {{(pid=123784) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} May 14 21:47:03.896369 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] Backup '80f8bfbe-cfe9-410c-a773-475f03a46ab4' of volume ef327fb0-87a3-4d7a-8e78-931cec53ab4d finished. {{(pid=123784) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} May 14 21:47:03.898225 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-ef327fb0-87a3-4d7a-8e78-931cec53ab4d', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'ef327fb0-87a3-4d7a-8e78-931cec53ab4d', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:47:04.281582 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] <== disconnect_volume: return (382ms) None {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:47:04.430632 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-2626f84c-e633-41ee-bd9b-0a3058dd281c tempest-VolumesBackupsAdminTest-1826845199 None] Create backup finished. backup: 80f8bfbe-cfe9-410c-a773-475f03a46ab4. May 14 21:47:04.784819 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-d7207c9c-9fef-4a16-bc55-daf0fd6ec883 tempest-VolumesBackupsAdminTest-1611340375 None] Export record started, backup: 80f8bfbe-cfe9-410c-a773-475f03a46ab4. May 14 21:47:04.786710 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-d7207c9c-9fef-4a16-bc55-daf0fd6ec883 tempest-VolumesBackupsAdminTest-1611340375 None] Export record finished, backup 80f8bfbe-cfe9-410c-a773-475f03a46ab4 exported. May 14 21:47:05.118309 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-fd196ad7-745a-49f8-9d54-b0a808da3d37 tempest-VolumesBackupsAdminTest-1611340375 None] Import record started, backup_url: eyJkcml2ZXJfaW5mbyI6IHt9LCAiaWQiOiAiZmNkMDdkYjEtNWM2Ni00YTk4LTg0OGItMjI4NTVlYjVlZTVmIiwgInVzZXJfaWQiOiAiN2NhNWFmM2M1OTk4NDJkMzlkZjVlZGY5ODc2Yzg5MmEiLCAicHJvamVjdF9pZCI6ICI3MGYzZThmZjFiNDA0ZTI4YWE5NjcwNmM4NDlhMjFlNCIsICJ2b2x1bWVfaWQiOiAiZWYzMjdmYjAtODdhMy00ZDdhLThlNzgtOTMxY2VjNTNhYjRkIiwgImhvc3QiOiAibmNkdi1tYXN0ZXItMSIsICJhdmFpbGFiaWxpdHlfem9uZSI6IG51bGwsICJjb250YWluZXIiOiAiYmFja3VwcyIsICJwYXJlbnRfaWQiOiBudWxsLCAic3RhdHVzIjogImF2YWlsYWJsZSIsICJmYWlsX3JlYXNvbiI6IG51bGwsICJzaXplIjogMSwgImRpc3BsYXlfbmFtZSI6ICJ0ZW1wZXN0LVZvbHVtZXNCYWNrdXBzQWRtaW5UZXN0LUJhY2t1cC0xMjQ2NDc5NDQ2IiwgImRpc3BsYXlfZGVzY3JpcHRpb24iOiBudWxsLCAic2VydmljZV9tZXRhZGF0YSI6ICJ7XCJiYXNlXCI6IFwidm9sdW1lLWVmMzI3ZmIwLTg3YTMtNGQ3YS04ZTc4LTkzMWNlYzUzYWI0ZC5iYWNrdXAuODBmOGJmYmUtY2ZlOS00MTBjLWE3NzMtNDc1ZjAzYTQ2YWI0XCJ9IiwgInNlcnZpY2UiOiAiY2luZGVyLmJhY2t1cC5kcml2ZXJzLmNlcGguQ2VwaEJhY2t1cERyaXZlciIsICJvYmplY3RfY291bnQiOiAwLCAidGVtcF92b2x1bWVfaWQiOiBudWxsLCAidGVtcF9zbmFwc2hvdF9pZCI6IG51bGwsICJudW1fZGVwZW5kZW50X2JhY2t1cHMiOiAwLCAic25hcHNob3RfaWQiOiBudWxsLCAiZGF0YV90aW1lc3RhbXAiOiAiMjAyMi0wNS0xNFQyMTo0Njo1OVoiLCAicmVzdG9yZV92b2x1bWVfaWQiOiBudWxsLCAibWV0YWRhdGEiOiB7fSwgImVuY3J5cHRpb25fa2V5X2lkIjogbnVsbCwgImNyZWF0ZWRfYXQiOiAiMjAyMi0wNS0xNFQyMTo0Njo1OVoiLCAidXBkYXRlZF9hdCI6ICIyMDIyLTA1LTE0VDIxOjQ3OjA0WiIsICJkZWxldGVkX2F0IjogbnVsbCwgImRlbGV0ZWQiOiBmYWxzZX0=. May 14 21:47:05.124131 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_db.sqlalchemy.engines [None req-fd196ad7-745a-49f8-9d54-b0a808da3d37 tempest-VolumesBackupsAdminTest-1611340375 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123784) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 14 21:47:05.259288 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-fd196ad7-745a-49f8-9d54-b0a808da3d37 tempest-VolumesBackupsAdminTest-1611340375 None] Import record id fcd07db1-5c66-4a98-848b-22855eb5ee5f metadata from driver finished. May 14 21:47:07.813845 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] Restore backup started, backup: 80f8bfbe-cfe9-410c-a773-475f03a46ab4 volume: 71f49a8c-07af-4ed7-b50f-f0d553cd188e. May 14 21:47:07.828720 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.utils [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.52', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:47:07.830218 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:47:07.837967 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.008s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:47:07.839072 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[e827d964-7469-4c0b-b5d4-653167853299]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:828e4c9f8c8c\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:47:07.840567 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:47:07.846752 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 14 21:47:07.847416 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[63e9c0cd-135f-4e8d-9d07-3164d58644b0]: [Errno 2] No such file or directory: 'systool' {{(pid=146745) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 14 21:47:07.847416 ncdv-master-1 cinder-backup[123784]: Traceback (most recent call last): May 14 21:47:07.847416 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 14 21:47:07.847416 ncdv-master-1 cinder-backup[123784]: ret = func(*f_args, **f_kwargs) May 14 21:47:07.847416 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 14 21:47:07.847416 ncdv-master-1 cinder-backup[123784]: return func(*args, **kwargs) May 14 21:47:07.847416 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 14 21:47:07.847416 ncdv-master-1 cinder-backup[123784]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 14 21:47:07.847416 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 14 21:47:07.847416 ncdv-master-1 cinder-backup[123784]: return putils.execute(on_execute=on_execute, May 14 21:47:07.847416 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 14 21:47:07.847416 ncdv-master-1 cinder-backup[123784]: obj = subprocess.Popen(cmd, May 14 21:47:07.847416 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 14 21:47:07.847416 ncdv-master-1 cinder-backup[123784]: self._execute_child(args, executable, preexec_fn, close_fds, May 14 21:47:07.847416 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 14 21:47:07.847416 ncdv-master-1 cinder-backup[123784]: raise child_exception_type(errno_num, err_msg, err_filename) May 14 21:47:07.847416 ncdv-master-1 cinder-backup[123784]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 14 21:47:07.848769 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[63e9c0cd-135f-4e8d-9d07-3164d58644b0]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:47:07.848926 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:47:07.854823 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 14 21:47:07.855381 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[ff252968-9910-45c4-b192-d10c4beae8f1]: [Errno 2] No such file or directory: 'systool' {{(pid=146745) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 14 21:47:07.855381 ncdv-master-1 cinder-backup[123784]: Traceback (most recent call last): May 14 21:47:07.855381 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 14 21:47:07.855381 ncdv-master-1 cinder-backup[123784]: ret = func(*f_args, **f_kwargs) May 14 21:47:07.855381 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 14 21:47:07.855381 ncdv-master-1 cinder-backup[123784]: return func(*args, **kwargs) May 14 21:47:07.855381 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 14 21:47:07.855381 ncdv-master-1 cinder-backup[123784]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 14 21:47:07.855381 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 14 21:47:07.855381 ncdv-master-1 cinder-backup[123784]: return putils.execute(on_execute=on_execute, May 14 21:47:07.855381 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 14 21:47:07.855381 ncdv-master-1 cinder-backup[123784]: obj = subprocess.Popen(cmd, May 14 21:47:07.855381 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 14 21:47:07.855381 ncdv-master-1 cinder-backup[123784]: self._execute_child(args, executable, preexec_fn, close_fds, May 14 21:47:07.855381 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 14 21:47:07.855381 ncdv-master-1 cinder-backup[123784]: raise child_exception_type(errno_num, err_msg, err_filename) May 14 21:47:07.855381 ncdv-master-1 cinder-backup[123784]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 14 21:47:07.856672 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[ff252968-9910-45c4-b192-d10c4beae8f1]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:47:07.858980 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:47:07.866199 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.009s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:47:07.866554 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[05aace01-9470-4971-9673-ae15a069ae85]: (4, ('/dev/sda1\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:47:07.867788 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:47:07.876684 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.009s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:47:07.877026 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[8a16a942-3794-4c79-9106-080e9542ff3f]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:47:07.878100 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:47:07.884418 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.006s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:47:07.887077 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[67cc7a85-2b7d-4bdf-937d-745febd6d9b3]: (4, ('04b32ac0-335b-a141-b9ee-83ea10e826d9\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:47:07.887077 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] Running cmd (subprocess): nvme version {{(pid=123784) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:47:07.908633 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] CMD "nvme version" returned: 0 in 0.019s {{(pid=123784) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:47:07.908633 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=123784) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 14 21:47:07.908633 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=123784) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 14 21:47:07.908633 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:5e2f7208-21be-4c5f-84d7-22d3a9f29e66 dsc: {{(pid=123784) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 14 21:47:07.909031 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.utils [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] <== get_connector_properties: return (79ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.52', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:828e4c9f8c8c', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '04b32ac0-335b-a141-b9ee-83ea10e826d9', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:5e2f7208-21be-4c5f-84d7-22d3a9f29e66', 'found_dsc': ''} {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:47:08.569603 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connector [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] Factory for rbd on None {{(pid=123784) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 14 21:47:08.571165 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-71f49a8c-07af-4ed7-b50f-f0d553cd188e', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '71f49a8c-07af-4ed7-b50f-f0d553cd188e', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:47:08.572228 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.linuxrbd [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] opening connection to ceph cluster (timeout=-1). {{(pid=123784) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 14 21:47:08.733417 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] <== connect_volume: return (161ms) {'path': } {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:47:08.741175 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_db.sqlalchemy.engines [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123784) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 14 21:47:08.751919 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] Starting restore from Ceph backup=80f8bfbe-cfe9-410c-a773-475f03a46ab4 to volume=volume-71f49a8c-07af-4ed7-b50f-f0d553cd188e {{(pid=123784) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1343}} May 14 21:47:08.859899 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] Looking for snapshot of backup base 'volume-ef327fb0-87a3-4d7a-8e78-931cec53ab4d.backup.80f8bfbe-cfe9-410c-a773-475f03a46ab4' {{(pid=123784) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 14 21:47:08.860528 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] Found snapshot 'backup.80f8bfbe-cfe9-410c-a773-475f03a46ab4.snap.1652564821.0631037' {{(pid=123784) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} May 14 21:47:08.870863 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] Volume file is RBD. {{(pid=123784) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1245}} May 14 21:47:08.913245 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] Attempting differential restore. {{(pid=123784) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1298}} May 14 21:47:08.913727 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] Attempting incremental restore from base='volume-ef327fb0-87a3-4d7a-8e78-931cec53ab4d.backup.80f8bfbe-cfe9-410c-a773-475f03a46ab4' snap='backup.80f8bfbe-cfe9-410c-a773-475f03a46ab4.snap.1652564821.0631037' {{(pid=123784) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1145}} May 14 21:47:08.914142 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] Performing differential transfer from 'volume-ef327fb0-87a3-4d7a-8e78-931cec53ab4d.backup.80f8bfbe-cfe9-410c-a773-475f03a46ab4' to 'volume-71f49a8c-07af-4ed7-b50f-f0d553cd188e' {{(pid=123784) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} May 14 21:47:08.914972 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-ef327fb0-87a3-4d7a-8e78-931cec53ab4d.backup.80f8bfbe-cfe9-410c-a773-475f03a46ab4@backup.80f8bfbe-cfe9-410c-a773-475f03a46ab4.snap.1652564821.0631037 -' into... {{(pid=123784) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} May 14 21:47:08.915355 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_zrykcyql --pool volumes - volumes/volume-71f49a8c-07af-4ed7-b50f-f0d553cd188e' {{(pid=123784) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} May 14 21:47:10.813457 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] Restore transfer completed in 1.8988s {{(pid=123784) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1167}} May 14 21:47:10.814491 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=123784) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1354}} May 14 21:47:10.836573 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] Restoring volume base metadata {{(pid=123784) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} May 14 21:47:10.913305 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] No metadata of type 'volume-metadata' to restore {{(pid=123784) put /opt/stack/cinder/cinder/backup/driver.py:345}} May 14 21:47:10.913305 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] No metadata of type 'volume-glance-metadata' to restore {{(pid=123784) put /opt/stack/cinder/cinder/backup/driver.py:345}} May 14 21:47:10.915417 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] Restore to volume 71f49a8c-07af-4ed7-b50f-f0d553cd188e finished successfully. {{(pid=123784) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1361}} May 14 21:47:10.917930 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-71f49a8c-07af-4ed7-b50f-f0d553cd188e', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': '71f49a8c-07af-4ed7-b50f-f0d553cd188e', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:47:11.024055 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] <== disconnect_volume: return (105ms) None {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:47:11.070249 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.manager [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] Encryption key ID for volume 71f49a8c-07af-4ed7-b50f-f0d553cd188e already matches encryption key ID in backup 80f8bfbe-cfe9-410c-a773-475f03a46ab4. {{(pid=123784) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} May 14 21:47:11.304946 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-83ff4172-749b-4837-934e-6deed23cfb5c tempest-VolumesBackupsAdminTest-1826845199 None] Finished restoring backup 80f8bfbe-cfe9-410c-a773-475f03a46ab4 to volume 71f49a8c-07af-4ed7-b50f-f0d553cd188e. May 14 21:47:12.280267 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-61a5eaf1-0a1d-4249-9422-d587ec6ee7bd tempest-VolumesBackupsAdminTest-1611340375 None] Delete backup started, backup: fcd07db1-5c66-4a98-848b-22855eb5ee5f. May 14 21:47:12.282520 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-61a5eaf1-0a1d-4249-9422-d587ec6ee7bd tempest-VolumesBackupsAdminTest-1611340375 None] Delete started for backup=fcd07db1-5c66-4a98-848b-22855eb5ee5f {{(pid=123784) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} May 14 21:47:12.282949 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-61a5eaf1-0a1d-4249-9422-d587ec6ee7bd tempest-VolumesBackupsAdminTest-1611340375 None] Trying diff format basename='volume-ef327fb0-87a3-4d7a-8e78-931cec53ab4d.backup.80f8bfbe-cfe9-410c-a773-475f03a46ab4' for backup base image of volume ef327fb0-87a3-4d7a-8e78-931cec53ab4d. {{(pid=123784) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} May 14 21:47:12.603203 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-61a5eaf1-0a1d-4249-9422-d587ec6ee7bd tempest-VolumesBackupsAdminTest-1611340375 None] Looking for snapshot of backup base 'volume-ef327fb0-87a3-4d7a-8e78-931cec53ab4d.backup.80f8bfbe-cfe9-410c-a773-475f03a46ab4' {{(pid=123784) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 14 21:47:12.603639 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-61a5eaf1-0a1d-4249-9422-d587ec6ee7bd tempest-VolumesBackupsAdminTest-1611340375 None] Backup 'fcd07db1-5c66-4a98-848b-22855eb5ee5f' has no snapshot {{(pid=123784) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:962}} May 14 21:47:12.607121 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-61a5eaf1-0a1d-4249-9422-d587ec6ee7bd tempest-VolumesBackupsAdminTest-1611340375 None] No backup snapshot to delete {{(pid=123784) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:500}} May 14 21:47:12.797074 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.drivers.ceph [None req-61a5eaf1-0a1d-4249-9422-d587ec6ee7bd tempest-VolumesBackupsAdminTest-1611340375 None] Backup base image of volume ef327fb0-87a3-4d7a-8e78-931cec53ab4d still has 1 snapshots so skipping base image delete. May 14 21:47:13.008049 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-61a5eaf1-0a1d-4249-9422-d587ec6ee7bd tempest-VolumesBackupsAdminTest-1611340375 None] Metadata backup object 'backup.fcd07db1-5c66-4a98-848b-22855eb5ee5f.meta' not found - ignoring {{(pid=123784) remove_if_exists /opt/stack/cinder/cinder/backup/drivers/ceph.py:162}} May 14 21:47:13.012232 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-61a5eaf1-0a1d-4249-9422-d587ec6ee7bd tempest-VolumesBackupsAdminTest-1611340375 None] Delete of backup 'fcd07db1-5c66-4a98-848b-22855eb5ee5f' for volume 'ef327fb0-87a3-4d7a-8e78-931cec53ab4d' finished. {{(pid=123784) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} May 14 21:47:13.018534 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_db.sqlalchemy.engines [None req-61a5eaf1-0a1d-4249-9422-d587ec6ee7bd tempest-VolumesBackupsAdminTest-1611340375 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123784) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 14 21:47:13.110790 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.quota [None req-61a5eaf1-0a1d-4249-9422-d587ec6ee7bd tempest-VolumesBackupsAdminTest-1611340375 None] Created reservations ['5a6deebe-501b-45d1-8d48-5fefa604aa95', '3932ed7a-b8c9-4fe5-82f8-5a3177fb5769'] {{(pid=123784) reserve /opt/stack/cinder/cinder/quota.py:755}} May 14 21:47:13.328197 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-61a5eaf1-0a1d-4249-9422-d587ec6ee7bd tempest-VolumesBackupsAdminTest-1611340375 None] Delete backup finished, backup fcd07db1-5c66-4a98-848b-22855eb5ee5f deleted. May 14 21:47:13.435911 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-907c2297-0c76-4468-977d-4bbdc95a57b3 tempest-VolumesBackupsAdminTest-1826845199 None] Delete backup started, backup: 80f8bfbe-cfe9-410c-a773-475f03a46ab4. May 14 21:47:13.437353 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-907c2297-0c76-4468-977d-4bbdc95a57b3 tempest-VolumesBackupsAdminTest-1826845199 None] Delete started for backup=80f8bfbe-cfe9-410c-a773-475f03a46ab4 {{(pid=123784) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} May 14 21:47:13.437824 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-907c2297-0c76-4468-977d-4bbdc95a57b3 tempest-VolumesBackupsAdminTest-1826845199 None] Trying diff format basename='volume-ef327fb0-87a3-4d7a-8e78-931cec53ab4d.backup.80f8bfbe-cfe9-410c-a773-475f03a46ab4' for backup base image of volume ef327fb0-87a3-4d7a-8e78-931cec53ab4d. {{(pid=123784) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} May 14 21:47:13.557338 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-907c2297-0c76-4468-977d-4bbdc95a57b3 tempest-VolumesBackupsAdminTest-1826845199 None] Looking for snapshot of backup base 'volume-ef327fb0-87a3-4d7a-8e78-931cec53ab4d.backup.80f8bfbe-cfe9-410c-a773-475f03a46ab4' {{(pid=123784) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 14 21:47:13.557717 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-907c2297-0c76-4468-977d-4bbdc95a57b3 tempest-VolumesBackupsAdminTest-1826845199 None] Found snapshot 'backup.80f8bfbe-cfe9-410c-a773-475f03a46ab4.snap.1652564821.0631037' {{(pid=123784) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} May 14 21:47:13.558085 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-907c2297-0c76-4468-977d-4bbdc95a57b3 tempest-VolumesBackupsAdminTest-1826845199 None] Deleting backup snapshot='backup.80f8bfbe-cfe9-410c-a773-475f03a46ab4.snap.1652564821.0631037' {{(pid=123784) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}} May 14 21:47:15.277974 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_db.sqlalchemy.engines [None req-85b1c192-0a07-4fed-b3a8-7524167e72b8 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123784) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 14 21:47:15.757212 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.drivers.ceph [None req-907c2297-0c76-4468-977d-4bbdc95a57b3 tempest-VolumesBackupsAdminTest-1826845199 None] Deleting backup base image='volume-ef327fb0-87a3-4d7a-8e78-931cec53ab4d.backup.80f8bfbe-cfe9-410c-a773-475f03a46ab4' of volume ef327fb0-87a3-4d7a-8e78-931cec53ab4d. May 14 21:47:17.067684 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:47:17.068381 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:47:19.985647 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-907c2297-0c76-4468-977d-4bbdc95a57b3 tempest-VolumesBackupsAdminTest-1826845199 None] Base backup image='volume-ef327fb0-87a3-4d7a-8e78-931cec53ab4d.backup.80f8bfbe-cfe9-410c-a773-475f03a46ab4' of volume ef327fb0-87a3-4d7a-8e78-931cec53ab4d deleted. {{(pid=123784) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} May 14 21:47:20.125288 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-907c2297-0c76-4468-977d-4bbdc95a57b3 tempest-VolumesBackupsAdminTest-1826845199 None] Delete of backup '80f8bfbe-cfe9-410c-a773-475f03a46ab4' for volume 'ef327fb0-87a3-4d7a-8e78-931cec53ab4d' finished. {{(pid=123784) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} May 14 21:47:20.208007 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.quota [None req-907c2297-0c76-4468-977d-4bbdc95a57b3 tempest-VolumesBackupsAdminTest-1826845199 None] Created reservations ['74dd20df-7621-4bf7-b623-3e0458cd383c', 'edd55913-3c29-4c92-bce7-5d88c358404c'] {{(pid=123784) reserve /opt/stack/cinder/cinder/quota.py:755}} May 14 21:47:20.390658 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-907c2297-0c76-4468-977d-4bbdc95a57b3 tempest-VolumesBackupsAdminTest-1826845199 None] Delete backup finished, backup 80f8bfbe-cfe9-410c-a773-475f03a46ab4 deleted. May 14 21:47:22.710069 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] Create backup started, backup: 199436eb-6427-4066-a8d2-17a5f70f8025 volume: fd86482c-c455-4c8d-84d0-f0da98fab425. May 14 21:47:22.791174 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-14T21:47:22Z,data_timestamp=2022-05-14T21:47:22Z,deleted=False,deleted_at=None,display_description=None,display_name='tempest-VolumesBackupsAdminTest-Backup-1375944518',encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=199436eb-6427-4066-a8d2-17a5f70f8025,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='70f3e8ff1b404e28aa96706c849a21e4',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='7ca5af3c599842d39df5edf9876c892a',volume_id=fd86482c-c455-4c8d-84d0-f0da98fab425) May 14 21:47:22.829572 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.utils [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.52', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:47:22.830945 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:47:22.838739 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.008s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:47:22.839069 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[ab21992a-ab1a-4e68-ba7a-2f201223e8e4]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:828e4c9f8c8c\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:47:22.840302 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:47:22.847089 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 14 21:47:22.847697 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[9cd018b6-80af-48a1-989c-f6f908f4ceb7]: [Errno 2] No such file or directory: 'systool' {{(pid=146745) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 14 21:47:22.847697 ncdv-master-1 cinder-backup[123784]: Traceback (most recent call last): May 14 21:47:22.847697 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 14 21:47:22.847697 ncdv-master-1 cinder-backup[123784]: ret = func(*f_args, **f_kwargs) May 14 21:47:22.847697 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 14 21:47:22.847697 ncdv-master-1 cinder-backup[123784]: return func(*args, **kwargs) May 14 21:47:22.847697 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 14 21:47:22.847697 ncdv-master-1 cinder-backup[123784]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 14 21:47:22.847697 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 14 21:47:22.847697 ncdv-master-1 cinder-backup[123784]: return putils.execute(on_execute=on_execute, May 14 21:47:22.847697 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 14 21:47:22.847697 ncdv-master-1 cinder-backup[123784]: obj = subprocess.Popen(cmd, May 14 21:47:22.847697 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 14 21:47:22.847697 ncdv-master-1 cinder-backup[123784]: self._execute_child(args, executable, preexec_fn, close_fds, May 14 21:47:22.847697 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 14 21:47:22.847697 ncdv-master-1 cinder-backup[123784]: raise child_exception_type(errno_num, err_msg, err_filename) May 14 21:47:22.847697 ncdv-master-1 cinder-backup[123784]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 14 21:47:22.849055 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[9cd018b6-80af-48a1-989c-f6f908f4ceb7]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:47:22.849173 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:47:22.854880 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 14 21:47:22.855429 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[74e8f452-2b78-4a9e-811a-413f75535207]: [Errno 2] No such file or directory: 'systool' {{(pid=146745) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 14 21:47:22.855429 ncdv-master-1 cinder-backup[123784]: Traceback (most recent call last): May 14 21:47:22.855429 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 14 21:47:22.855429 ncdv-master-1 cinder-backup[123784]: ret = func(*f_args, **f_kwargs) May 14 21:47:22.855429 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 14 21:47:22.855429 ncdv-master-1 cinder-backup[123784]: return func(*args, **kwargs) May 14 21:47:22.855429 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 14 21:47:22.855429 ncdv-master-1 cinder-backup[123784]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 14 21:47:22.855429 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 14 21:47:22.855429 ncdv-master-1 cinder-backup[123784]: return putils.execute(on_execute=on_execute, May 14 21:47:22.855429 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 14 21:47:22.855429 ncdv-master-1 cinder-backup[123784]: obj = subprocess.Popen(cmd, May 14 21:47:22.855429 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 14 21:47:22.855429 ncdv-master-1 cinder-backup[123784]: self._execute_child(args, executable, preexec_fn, close_fds, May 14 21:47:22.855429 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 14 21:47:22.855429 ncdv-master-1 cinder-backup[123784]: raise child_exception_type(errno_num, err_msg, err_filename) May 14 21:47:22.855429 ncdv-master-1 cinder-backup[123784]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 14 21:47:22.856858 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[74e8f452-2b78-4a9e-811a-413f75535207]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:47:22.857072 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:47:22.866271 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.009s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:47:22.866596 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[be56f1f6-1296-4a39-91c1-4b090acedc22]: (4, ('/dev/sda1\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:47:22.867723 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:47:22.875917 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.008s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:47:22.876243 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[56700c93-deea-408c-8acb-67399d2e52f9]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:47:22.877311 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:47:22.883717 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.006s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:47:22.884024 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[dcc2d2f0-aca1-4ebb-8262-d2bad54a1c89]: (4, ('04b32ac0-335b-a141-b9ee-83ea10e826d9\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:47:22.884757 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] Running cmd (subprocess): nvme version {{(pid=123784) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:47:22.904678 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] CMD "nvme version" returned: 0 in 0.019s {{(pid=123784) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:47:22.906477 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=123784) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 14 21:47:22.906848 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=123784) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 14 21:47:22.907222 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:5e2f7208-21be-4c5f-84d7-22d3a9f29e66 dsc: {{(pid=123784) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 14 21:47:22.907891 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.utils [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] <== get_connector_properties: return (77ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.52', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:828e4c9f8c8c', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '04b32ac0-335b-a141-b9ee-83ea10e826d9', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:5e2f7208-21be-4c5f-84d7-22d3a9f29e66', 'found_dsc': ''} {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:47:23.556092 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connector [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] Factory for rbd on None {{(pid=123784) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 14 21:47:23.557716 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-fd86482c-c455-4c8d-84d0-f0da98fab425', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'fd86482c-c455-4c8d-84d0-f0da98fab425', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:47:23.558785 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.linuxrbd [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] opening connection to ceph cluster (timeout=-1). {{(pid=123784) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 14 21:47:23.734069 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] <== connect_volume: return (175ms) {'path': } {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:47:23.742448 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_db.sqlalchemy.engines [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123784) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 14 21:47:23.808080 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] Starting backup of volume='fd86482c-c455-4c8d-84d0-f0da98fab425'. {{(pid=123784) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} May 14 21:47:23.808908 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] Volume file is RBD: attempting optimized backup {{(pid=123784) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}} May 14 21:47:23.828978 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] Creating base image 'volume-fd86482c-c455-4c8d-84d0-f0da98fab425.backup.199436eb-6427-4066-a8d2-17a5f70f8025' {{(pid=123784) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}} May 14 21:47:23.829390 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=123784) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} May 14 21:47:24.298547 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] Using --from-snap 'None' for incremental backup of volume fd86482c-c455-4c8d-84d0-f0da98fab425. {{(pid=123784) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}} May 14 21:47:24.298964 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] Creating backup snapshot='backup.199436eb-6427-4066-a8d2-17a5f70f8025.snap.1652564844.2985759' {{(pid=123784) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}} May 14 21:47:25.564384 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] Performing differential transfer from 'volume-fd86482c-c455-4c8d-84d0-f0da98fab425' to 'volume-fd86482c-c455-4c8d-84d0-f0da98fab425.backup.199436eb-6427-4066-a8d2-17a5f70f8025' {{(pid=123784) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} May 14 21:47:25.565048 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_fjsd1bn1 --pool volumes volumes/volume-fd86482c-c455-4c8d-84d0-f0da98fab425@backup.199436eb-6427-4066-a8d2-17a5f70f8025.snap.1652564844.2985759 -' into... {{(pid=123784) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} May 14 21:47:25.565172 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-fd86482c-c455-4c8d-84d0-f0da98fab425.backup.199436eb-6427-4066-a8d2-17a5f70f8025' {{(pid=123784) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} May 14 21:47:27.283464 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] Differential backup transfer completed in 1.7185s {{(pid=123784) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}} May 14 21:47:27.284241 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] Getting metadata type 'volume-base-metadata' {{(pid=123784) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 14 21:47:27.297505 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=123784) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 14 21:47:27.297931 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] Getting metadata type 'volume-metadata' {{(pid=123784) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 14 21:47:27.304251 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] No metadata type 'volume-metadata' available {{(pid=123784) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 14 21:47:27.304643 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] Getting metadata type 'volume-glance-metadata' {{(pid=123784) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 14 21:47:27.310733 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] No metadata type 'volume-glance-metadata' available {{(pid=123784) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:147}} May 14 21:47:27.311853 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] Backing up metadata for volume fd86482c-c455-4c8d-84d0-f0da98fab425. {{(pid=123784) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} May 14 21:47:27.486342 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] Backup '199436eb-6427-4066-a8d2-17a5f70f8025' of volume fd86482c-c455-4c8d-84d0-f0da98fab425 finished. {{(pid=123784) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} May 14 21:47:27.489217 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-fd86482c-c455-4c8d-84d0-f0da98fab425', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'fd86482c-c455-4c8d-84d0-f0da98fab425', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:47:28.082067 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] <== disconnect_volume: return (591ms) None {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:47:28.253429 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-0ed8bdbf-e410-4aa6-abed-05911a707c22 tempest-VolumesBackupsAdminTest-1826845199 None] Create backup finished. backup: 199436eb-6427-4066-a8d2-17a5f70f8025. May 14 21:47:28.877449 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-3e729973-475b-404c-9913-828ce1ac6497 tempest-VolumesBackupsAdminTest-1611340375 None] Reset backup status started, backup_id: 199436eb-6427-4066-a8d2-17a5f70f8025, status: error. May 14 21:47:28.877449 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-3e729973-475b-404c-9913-828ce1ac6497 tempest-VolumesBackupsAdminTest-1611340375 None] Backup service: cinder.backup.drivers.ceph.CephBackupDriver. May 14 21:47:30.051362 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-83d42dd1-a633-46d5-b735-1008be6fc52b tempest-VolumesBackupsAdminTest-1826845199 None] Delete backup started, backup: 199436eb-6427-4066-a8d2-17a5f70f8025. May 14 21:47:30.053197 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-83d42dd1-a633-46d5-b735-1008be6fc52b tempest-VolumesBackupsAdminTest-1826845199 None] Delete started for backup=199436eb-6427-4066-a8d2-17a5f70f8025 {{(pid=123784) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} May 14 21:47:30.053776 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-83d42dd1-a633-46d5-b735-1008be6fc52b tempest-VolumesBackupsAdminTest-1826845199 None] Trying diff format basename='volume-fd86482c-c455-4c8d-84d0-f0da98fab425.backup.199436eb-6427-4066-a8d2-17a5f70f8025' for backup base image of volume fd86482c-c455-4c8d-84d0-f0da98fab425. {{(pid=123784) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} May 14 21:47:30.249740 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-83d42dd1-a633-46d5-b735-1008be6fc52b tempest-VolumesBackupsAdminTest-1826845199 None] Looking for snapshot of backup base 'volume-fd86482c-c455-4c8d-84d0-f0da98fab425.backup.199436eb-6427-4066-a8d2-17a5f70f8025' {{(pid=123784) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 14 21:47:30.250135 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-83d42dd1-a633-46d5-b735-1008be6fc52b tempest-VolumesBackupsAdminTest-1826845199 None] Found snapshot 'backup.199436eb-6427-4066-a8d2-17a5f70f8025.snap.1652564844.2985759' {{(pid=123784) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} May 14 21:47:30.250514 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-83d42dd1-a633-46d5-b735-1008be6fc52b tempest-VolumesBackupsAdminTest-1826845199 None] Deleting backup snapshot='backup.199436eb-6427-4066-a8d2-17a5f70f8025.snap.1652564844.2985759' {{(pid=123784) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}} May 14 21:47:32.460088 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.drivers.ceph [None req-83d42dd1-a633-46d5-b735-1008be6fc52b tempest-VolumesBackupsAdminTest-1826845199 None] Deleting backup base image='volume-fd86482c-c455-4c8d-84d0-f0da98fab425.backup.199436eb-6427-4066-a8d2-17a5f70f8025' of volume fd86482c-c455-4c8d-84d0-f0da98fab425. May 14 21:47:35.122900 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-83d42dd1-a633-46d5-b735-1008be6fc52b tempest-VolumesBackupsAdminTest-1826845199 None] Base backup image='volume-fd86482c-c455-4c8d-84d0-f0da98fab425.backup.199436eb-6427-4066-a8d2-17a5f70f8025' of volume fd86482c-c455-4c8d-84d0-f0da98fab425 deleted. {{(pid=123784) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} May 14 21:47:35.234140 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-83d42dd1-a633-46d5-b735-1008be6fc52b tempest-VolumesBackupsAdminTest-1826845199 None] Delete of backup '199436eb-6427-4066-a8d2-17a5f70f8025' for volume 'fd86482c-c455-4c8d-84d0-f0da98fab425' finished. {{(pid=123784) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} May 14 21:47:35.240220 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_db.sqlalchemy.engines [None req-83d42dd1-a633-46d5-b735-1008be6fc52b tempest-VolumesBackupsAdminTest-1826845199 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123784) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 14 21:47:35.333538 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.quota [None req-83d42dd1-a633-46d5-b735-1008be6fc52b tempest-VolumesBackupsAdminTest-1826845199 None] Created reservations ['a817d8de-062d-4260-b035-f5179a80e10e', '24500b90-0eb0-44d0-8368-48b53612db7f'] {{(pid=123784) reserve /opt/stack/cinder/cinder/quota.py:755}} May 14 21:47:35.482101 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-83d42dd1-a633-46d5-b735-1008be6fc52b tempest-VolumesBackupsAdminTest-1826845199 None] Delete backup finished, backup 199436eb-6427-4066-a8d2-17a5f70f8025 deleted. May 14 21:48:17.064574 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:48:17.065656 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:48:34.281479 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] Create backup started, backup: 0eb54d35-d84d-46e4-96c5-2f336ec4a342 volume: f3e8f8d6-a7b2-4a2b-9ca3-fda68b059872. May 14 21:48:34.384433 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2022-05-14T21:48:33Z,data_timestamp=2022-05-14T21:48:33Z,deleted=False,deleted_at=None,display_description=None,display_name=None,encryption_key_id=None,fail_reason=None,host='ncdv-master-1',id=0eb54d35-d84d-46e4-96c5-2f336ec4a342,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='fbd6e36a563a4245acae34e00437ca87',restore_volume_id=None,service='cinder.backup.drivers.ceph.CephBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='1b51720c7970425abf4a26b2406893b7',volume_id=f3e8f8d6-a7b2-4a2b-9ca3-fda68b059872) May 14 21:48:34.423540 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.utils [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.52', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:48:34.428233 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:48:34.435066 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.009s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:48:34.435563 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[71819d38-43d2-4243-9d05-9603243897fb]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:828e4c9f8c8c\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:48:34.437024 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:48:34.445697 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 14 21:48:34.446751 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[1ec73e66-5f18-40ea-8649-cb9b81c4e889]: [Errno 2] No such file or directory: 'systool' {{(pid=146745) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 14 21:48:34.446751 ncdv-master-1 cinder-backup[123784]: Traceback (most recent call last): May 14 21:48:34.446751 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 14 21:48:34.446751 ncdv-master-1 cinder-backup[123784]: ret = func(*f_args, **f_kwargs) May 14 21:48:34.446751 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 14 21:48:34.446751 ncdv-master-1 cinder-backup[123784]: return func(*args, **kwargs) May 14 21:48:34.446751 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 14 21:48:34.446751 ncdv-master-1 cinder-backup[123784]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 14 21:48:34.446751 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 14 21:48:34.446751 ncdv-master-1 cinder-backup[123784]: return putils.execute(on_execute=on_execute, May 14 21:48:34.446751 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 14 21:48:34.446751 ncdv-master-1 cinder-backup[123784]: obj = subprocess.Popen(cmd, May 14 21:48:34.446751 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 14 21:48:34.446751 ncdv-master-1 cinder-backup[123784]: self._execute_child(args, executable, preexec_fn, close_fds, May 14 21:48:34.446751 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 14 21:48:34.446751 ncdv-master-1 cinder-backup[123784]: raise child_exception_type(errno_num, err_msg, err_filename) May 14 21:48:34.446751 ncdv-master-1 cinder-backup[123784]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 14 21:48:34.448194 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[1ec73e66-5f18-40ea-8649-cb9b81c4e889]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:48:34.449115 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:48:34.456282 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 14 21:48:34.457400 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[2ecb8cd4-4f4a-4260-bdff-fa4bdc2c9ad2]: [Errno 2] No such file or directory: 'systool' {{(pid=146745) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 14 21:48:34.457400 ncdv-master-1 cinder-backup[123784]: Traceback (most recent call last): May 14 21:48:34.457400 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 14 21:48:34.457400 ncdv-master-1 cinder-backup[123784]: ret = func(*f_args, **f_kwargs) May 14 21:48:34.457400 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 14 21:48:34.457400 ncdv-master-1 cinder-backup[123784]: return func(*args, **kwargs) May 14 21:48:34.457400 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 14 21:48:34.457400 ncdv-master-1 cinder-backup[123784]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 14 21:48:34.457400 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 14 21:48:34.457400 ncdv-master-1 cinder-backup[123784]: return putils.execute(on_execute=on_execute, May 14 21:48:34.457400 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 14 21:48:34.457400 ncdv-master-1 cinder-backup[123784]: obj = subprocess.Popen(cmd, May 14 21:48:34.457400 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 14 21:48:34.457400 ncdv-master-1 cinder-backup[123784]: self._execute_child(args, executable, preexec_fn, close_fds, May 14 21:48:34.457400 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 14 21:48:34.457400 ncdv-master-1 cinder-backup[123784]: raise child_exception_type(errno_num, err_msg, err_filename) May 14 21:48:34.457400 ncdv-master-1 cinder-backup[123784]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 14 21:48:34.458711 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[2ecb8cd4-4f4a-4260-bdff-fa4bdc2c9ad2]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:48:34.462236 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:48:34.469245 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.009s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:48:34.469565 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[5783e2ac-3624-4fc7-8b74-70f44cf36977]: (4, ('/dev/sda1\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:48:34.470715 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:48:34.480665 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.010s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:48:34.480983 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[02fc37d5-c428-478d-b28e-cc1abd6df3d9]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:48:34.482127 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:48:34.490172 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.008s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:48:34.490502 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[06d5f165-721e-428a-be54-4eb99c1ba39f]: (4, ('04b32ac0-335b-a141-b9ee-83ea10e826d9\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:48:34.491265 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] Running cmd (subprocess): nvme version {{(pid=123784) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:48:34.510822 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] CMD "nvme version" returned: 0 in 0.019s {{(pid=123784) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:48:34.512593 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=123784) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 14 21:48:34.512992 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=123784) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 14 21:48:34.513352 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:5e2f7208-21be-4c5f-84d7-22d3a9f29e66 dsc: {{(pid=123784) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 14 21:48:34.514045 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.utils [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] <== get_connector_properties: return (89ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.52', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:828e4c9f8c8c', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '04b32ac0-335b-a141-b9ee-83ea10e826d9', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:5e2f7208-21be-4c5f-84d7-22d3a9f29e66', 'found_dsc': ''} {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:48:35.229911 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connector [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] Factory for rbd on None {{(pid=123784) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 14 21:48:35.232117 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-f3e8f8d6-a7b2-4a2b-9ca3-fda68b059872', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'f3e8f8d6-a7b2-4a2b-9ca3-fda68b059872', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:48:35.237811 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.linuxrbd [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] opening connection to ceph cluster (timeout=-1). {{(pid=123784) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 14 21:48:35.423222 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] <== connect_volume: return (190ms) {'path': } {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:48:35.435429 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_db.sqlalchemy.engines [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123784) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 14 21:48:35.550126 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] Starting backup of volume='f3e8f8d6-a7b2-4a2b-9ca3-fda68b059872'. {{(pid=123784) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1023}} May 14 21:48:35.551286 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] Volume file is RBD: attempting optimized backup {{(pid=123784) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1034}} May 14 21:48:35.578205 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] Creating base image 'volume-f3e8f8d6-a7b2-4a2b-9ca3-fda68b059872.backup.0eb54d35-d84d-46e4-96c5-2f336ec4a342' {{(pid=123784) _create_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:465}} May 14 21:48:35.578849 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] RBD also supports fast-diff, enabling it together with exclusive-lock and object-map {{(pid=123784) _get_rbd_support /opt/stack/cinder/cinder/backup/drivers/ceph.py:275}} May 14 21:48:36.154224 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] Using --from-snap 'None' for incremental backup of volume f3e8f8d6-a7b2-4a2b-9ca3-fda68b059872. {{(pid=123784) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:807}} May 14 21:48:36.154874 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] Creating backup snapshot='backup.0eb54d35-d84d-46e4-96c5-2f336ec4a342.snap.1652564916.1542535' {{(pid=123784) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:813}} May 14 21:48:38.103693 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] Performing differential transfer from 'volume-f3e8f8d6-a7b2-4a2b-9ca3-fda68b059872' to 'volume-f3e8f8d6-a7b2-4a2b-9ca3-fda68b059872.backup.0eb54d35-d84d-46e4-96c5-2f336ec4a342' {{(pid=123784) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} May 14 21:48:38.107421 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] Piping cmd1='rbd export-diff --id cinder --conf /tmp/brickrbd_y_lx77k0 --pool volumes volumes/volume-f3e8f8d6-a7b2-4a2b-9ca3-fda68b059872@backup.0eb54d35-d84d-46e4-96c5-2f336ec4a342.snap.1652564916.1542535 -' into... {{(pid=123784) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} May 14 21:48:38.107421 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] cmd2='rbd import-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups - backups/volume-f3e8f8d6-a7b2-4a2b-9ca3-fda68b059872.backup.0eb54d35-d84d-46e4-96c5-2f336ec4a342' {{(pid=123784) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} May 14 21:49:16.570421 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] Differential backup transfer completed in 38.4666s {{(pid=123784) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:833}} May 14 21:49:16.571172 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] Getting metadata type 'volume-base-metadata' {{(pid=123784) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:79}} May 14 21:49:16.584187 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] Completed fetching metadata type 'volume-base-metadata' {{(pid=123784) _save_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:98}} May 14 21:49:16.584618 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] Getting metadata type 'volume-metadata' {{(pid=123784) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:109}} May 14 21:49:16.590774 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] No metadata type 'volume-metadata' available {{(pid=123784) _save_vol_meta /opt/stack/cinder/cinder/backup/driver.py:123}} May 14 21:49:16.591149 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] Getting metadata type 'volume-glance-metadata' {{(pid=123784) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:132}} May 14 21:49:16.598049 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] Completed fetching metadata type 'volume-glance-metadata' {{(pid=123784) _save_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:145}} May 14 21:49:16.598611 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] Backing up metadata for volume f3e8f8d6-a7b2-4a2b-9ca3-fda68b059872. {{(pid=123784) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:998}} May 14 21:49:16.707324 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] Backup '0eb54d35-d84d-46e4-96c5-2f336ec4a342' of volume f3e8f8d6-a7b2-4a2b-9ca3-fda68b059872 finished. {{(pid=123784) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1064}} May 14 21:49:16.709100 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-f3e8f8d6-a7b2-4a2b-9ca3-fda68b059872', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'f3e8f8d6-a7b2-4a2b-9ca3-fda68b059872', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': True}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:49:17.102443 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] <== disconnect_volume: return (392ms) None {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:49:17.105696 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:49:17.106203 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:49:17.279799 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-18cf99ad-ac6f-40de-843e-b44a8e50d139 tempest-TestVolumeBackupRestore-1478043453 None] Create backup finished. backup: 0eb54d35-d84d-46e4-96c5-2f336ec4a342. May 14 21:49:19.108528 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] Restore backup started, backup: 0eb54d35-d84d-46e4-96c5-2f336ec4a342 volume: e466666d-dad2-4f32-a5b6-875fbf06b6c9. May 14 21:49:19.122856 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.utils [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] ==> get_connector_properties: call "{'root_helper': 'sudo cinder-rootwrap /etc/cinder/rootwrap.conf', 'my_ip': '10.222.0.52', 'multipath': False, 'enforce_multipath': False, 'host': None, 'execute': None}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:49:19.124328 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /etc/iscsi/initiatorname.iscsi {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:49:19.132688 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "cat /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.008s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:49:19.133172 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[8cb8d40c-dc98-48ef-9700-7d8aa13801e4]: (4, ('## DO NOT EDIT OR REMOVE THIS FILE!\n## If you remove this file, the iSCSI daemon will not start.\n## If you change the InitiatorName, existing access control lists\n## may reject this initiator. The InitiatorName must be unique\n## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\nInitiatorName=iqn.1993-08.org.debian:01:828e4c9f8c8c\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:49:19.135574 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:49:19.144362 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 14 21:49:19.145303 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[988899b0-c6f9-412b-a8d2-5cbebd1c3e36]: [Errno 2] No such file or directory: 'systool' {{(pid=146745) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 14 21:49:19.145303 ncdv-master-1 cinder-backup[123784]: Traceback (most recent call last): May 14 21:49:19.145303 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 14 21:49:19.145303 ncdv-master-1 cinder-backup[123784]: ret = func(*f_args, **f_kwargs) May 14 21:49:19.145303 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 14 21:49:19.145303 ncdv-master-1 cinder-backup[123784]: return func(*args, **kwargs) May 14 21:49:19.145303 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 14 21:49:19.145303 ncdv-master-1 cinder-backup[123784]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 14 21:49:19.145303 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 14 21:49:19.145303 ncdv-master-1 cinder-backup[123784]: return putils.execute(on_execute=on_execute, May 14 21:49:19.145303 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 14 21:49:19.145303 ncdv-master-1 cinder-backup[123784]: obj = subprocess.Popen(cmd, May 14 21:49:19.145303 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 14 21:49:19.145303 ncdv-master-1 cinder-backup[123784]: self._execute_child(args, executable, preexec_fn, close_fds, May 14 21:49:19.145303 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 14 21:49:19.145303 ncdv-master-1 cinder-backup[123784]: raise child_exception_type(errno_num, err_msg, err_filename) May 14 21:49:19.145303 ncdv-master-1 cinder-backup[123784]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 14 21:49:19.146788 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[988899b0-c6f9-412b-a8d2-5cbebd1c3e36]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:49:19.146900 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): systool -c fc_host -v {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:49:19.159617 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] 'systool -c fc_host -v' failed. Not Retrying. {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:473}} May 14 21:49:19.160211 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[6a1e0bbd-ee37-4e53-aabc-bd1ccb46f631]: [Errno 2] No such file or directory: 'systool' {{(pid=146745) _process_cmd /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:479}} May 14 21:49:19.160211 ncdv-master-1 cinder-backup[123784]: Traceback (most recent call last): May 14 21:49:19.160211 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line 476, in _process_cmd May 14 21:49:19.160211 ncdv-master-1 cinder-backup[123784]: ret = func(*f_args, **f_kwargs) May 14 21:49:19.160211 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py", line 274, in _wrap May 14 21:49:19.160211 ncdv-master-1 cinder-backup[123784]: return func(*args, **kwargs) May 14 21:49:19.160211 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root May 14 21:49:19.160211 ncdv-master-1 cinder-backup[123784]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs) May 14 21:49:19.160211 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute May 14 21:49:19.160211 ncdv-master-1 cinder-backup[123784]: return putils.execute(on_execute=on_execute, May 14 21:49:19.160211 ncdv-master-1 cinder-backup[123784]: File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 395, in execute May 14 21:49:19.160211 ncdv-master-1 cinder-backup[123784]: obj = subprocess.Popen(cmd, May 14 21:49:19.160211 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ May 14 21:49:19.160211 ncdv-master-1 cinder-backup[123784]: self._execute_child(args, executable, preexec_fn, close_fds, May 14 21:49:19.160211 ncdv-master-1 cinder-backup[123784]: File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child May 14 21:49:19.160211 ncdv-master-1 cinder-backup[123784]: raise child_exception_type(errno_num, err_msg, err_filename) May 14 21:49:19.160211 ncdv-master-1 cinder-backup[123784]: FileNotFoundError: [Errno 2] No such file or directory: 'systool' May 14 21:49:19.161596 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[6a1e0bbd-ee37-4e53-aabc-bd1ccb46f631]: (5, 'builtins.FileNotFoundError', (2, 'No such file or directory')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:49:19.162347 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:49:19.171202 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.009s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:49:19.171552 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[6fdafd80-6e7d-4caa-a8cf-de2c6ccac420]: (4, ('/dev/sda1\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:49:19.172732 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid /dev/sda1 -s UUID -o value {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:49:19.181989 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "blkid /dev/sda1 -s UUID -o value" returned: 0 in 0.009s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:49:19.182402 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[29703f65-1175-4406-8d08-08abd0b65748]: (4, ('02bb033c-a191-40c0-94fb-bb025c724b13\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:49:19.183600 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): cat /sys/class/dmi/id/product_uuid {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:49:19.190097 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [-] CMD "cat /sys/class/dmi/id/product_uuid" returned: 0 in 0.006s {{(pid=146745) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:49:19.190419 ncdv-master-1 cinder-backup[123784]: DEBUG oslo.privsep.daemon [-] privsep: reply[1588924d-c240-46ff-b282-e2efbc48eb82]: (4, ('04b32ac0-335b-a141-b9ee-83ea10e826d9\n', '')) {{(pid=146745) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}} May 14 21:49:19.191233 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] Running cmd (subprocess): nvme version {{(pid=123784) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}} May 14 21:49:19.210690 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_concurrency.processutils [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] CMD "nvme version" returned: 0 in 0.019s {{(pid=123784) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}} May 14 21:49:19.212636 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] LIGHTOS: [Errno 111] ECONNREFUSED {{(pid=123784) find_dsc /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:100}} May 14 21:49:19.213008 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] LIGHTOS: did not find dsc, continuing anyway. {{(pid=123784) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:78}} May 14 21:49:19.213385 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.lightos [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] LIGHTOS: finally hostnqn: nqn.2014-08.org.nvmexpress:uuid:5e2f7208-21be-4c5f-84d7-22d3a9f29e66 dsc: {{(pid=123784) get_connector_properties /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/lightos.py:81}} May 14 21:49:19.214035 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.utils [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] <== get_connector_properties: return (90ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '10.222.0.52', 'host': 'ncdv-master-1', 'multipath': False, 'initiator': 'iqn.1993-08.org.debian:01:828e4c9f8c8c', 'do_local_attach': False, 'uuid': '02bb033c-a191-40c0-94fb-bb025c724b13', 'system uuid': '04b32ac0-335b-a141-b9ee-83ea10e826d9', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:5e2f7208-21be-4c5f-84d7-22d3a9f29e66', 'found_dsc': ''} {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:49:19.883903 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connector [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] Factory for rbd on None {{(pid=123784) factory /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connector.py:284}} May 14 21:49:19.885495 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] ==> connect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-e466666d-dad2-4f32-a5b6-875fbf06b6c9', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'e466666d-dad2-4f32-a5b6-875fbf06b6c9', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, ': False}}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:49:19.886597 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.linuxrbd [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] opening connection to ceph cluster (timeout=-1). {{(pid=123784) connect /usr/local/lib/python3.8/dist-packages/os_brick/initiator/linuxrbd.py:70}} May 14 21:49:20.068941 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] <== connect_volume: return (182ms) {'path': } {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:49:20.076774 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_db.sqlalchemy.engines [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123784) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 14 21:49:20.087229 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] Starting restore from Ceph backup=0eb54d35-d84d-46e4-96c5-2f336ec4a342 to volume=volume-e466666d-dad2-4f32-a5b6-875fbf06b6c9 {{(pid=123784) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1343}} May 14 21:49:20.149047 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] Looking for snapshot of backup base 'volume-f3e8f8d6-a7b2-4a2b-9ca3-fda68b059872.backup.0eb54d35-d84d-46e4-96c5-2f336ec4a342' {{(pid=123784) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 14 21:49:20.149661 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] Found snapshot 'backup.0eb54d35-d84d-46e4-96c5-2f336ec4a342.snap.1652564916.1542535' {{(pid=123784) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} May 14 21:49:20.154816 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] Volume file is RBD. {{(pid=123784) _diff_restore_allowed /opt/stack/cinder/cinder/backup/drivers/ceph.py:1245}} May 14 21:49:20.192937 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] Attempting differential restore. {{(pid=123784) _restore_volume /opt/stack/cinder/cinder/backup/drivers/ceph.py:1298}} May 14 21:49:20.193450 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] Attempting incremental restore from base='volume-f3e8f8d6-a7b2-4a2b-9ca3-fda68b059872.backup.0eb54d35-d84d-46e4-96c5-2f336ec4a342' snap='backup.0eb54d35-d84d-46e4-96c5-2f336ec4a342.snap.1652564916.1542535' {{(pid=123784) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1145}} May 14 21:49:20.193841 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] Performing differential transfer from 'volume-f3e8f8d6-a7b2-4a2b-9ca3-fda68b059872.backup.0eb54d35-d84d-46e4-96c5-2f336ec4a342' to 'volume-e466666d-dad2-4f32-a5b6-875fbf06b6c9' {{(pid=123784) _rbd_diff_transfer /opt/stack/cinder/cinder/backup/drivers/ceph.py:654}} May 14 21:49:20.194414 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] Piping cmd1='rbd export-diff --id cinder-bak --conf /etc/ceph/ceph.conf --pool backups backups/volume-f3e8f8d6-a7b2-4a2b-9ca3-fda68b059872.backup.0eb54d35-d84d-46e4-96c5-2f336ec4a342@backup.0eb54d35-d84d-46e4-96c5-2f336ec4a342.snap.1652564916.1542535 -' into... {{(pid=123784) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:611}} May 14 21:49:20.194784 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] cmd2='rbd import-diff --id cinder --conf /tmp/brickrbd_81b9gk2y --pool volumes - volumes/volume-e466666d-dad2-4f32-a5b6-875fbf06b6c9' {{(pid=123784) _piped_execute /opt/stack/cinder/cinder/backup/drivers/ceph.py:612}} May 14 21:50:01.903105 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] Restore transfer completed in 41.7087s {{(pid=123784) _diff_restore_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.py:1167}} May 14 21:50:01.904372 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] Restore target I/O object does not support fileno() - skipping call to fsync(). {{(pid=123784) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1354}} May 14 21:50:01.928319 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] Restoring volume base metadata {{(pid=123784) _restore_vol_base_meta /opt/stack/cinder/cinder/backup/driver.py:177}} May 14 21:50:02.028691 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] No metadata of type 'volume-metadata' to restore {{(pid=123784) put /opt/stack/cinder/cinder/backup/driver.py:345}} May 14 21:50:02.029059 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.driver [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] Restoring volume glance metadata {{(pid=123784) _restore_vol_glance_meta /opt/stack/cinder/cinder/backup/driver.py:258}} May 14 21:50:02.953101 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] Restore to volume e466666d-dad2-4f32-a5b6-875fbf06b6c9 finished successfully. {{(pid=123784) restore /opt/stack/cinder/cinder/backup/drivers/ceph.py:1361}} May 14 21:50:02.954847 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] ==> disconnect_volume: call "{'self': , 'connection_properties': {'name': 'volumes/volume-e466666d-dad2-4f32-a5b6-875fbf06b6c9', 'hosts': ['10.222.0.52'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'e466666d-dad2-4f32-a5b6-875fbf06b6c9', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}, 'device_info': {'path': }, 'force': True, ': False}" {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:167}} May 14 21:50:03.091158 ncdv-master-1 cinder-backup[123784]: DEBUG os_brick.initiator.connectors.rbd [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] <== disconnect_volume: return (135ms) None {{(pid=123784) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:194}} May 14 21:50:03.138977 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.manager [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] Encryption key ID for volume e466666d-dad2-4f32-a5b6-875fbf06b6c9 already matches encryption key ID in backup 0eb54d35-d84d-46e4-96c5-2f336ec4a342. {{(pid=123784) _run_restore /opt/stack/cinder/cinder/backup/manager.py:824}} May 14 21:50:03.341658 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-7bdc71ed-122f-4528-94fb-53081a2ad142 tempest-TestVolumeBackupRestore-1478043453 None] Finished restoring backup 0eb54d35-d84d-46e4-96c5-2f336ec4a342 to volume e466666d-dad2-4f32-a5b6-875fbf06b6c9. May 14 21:50:17.111948 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:50:17.112581 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:51:17.117348 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:51:17.118296 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:52:17.122643 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:52:17.123290 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:53:17.119581 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:53:17.120309 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:54:17.115127 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:54:17.115982 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:55:17.120192 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:55:17.120880 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:55:46.830502 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-99e8003f-b5ec-406c-9bfd-4fc96f2eba2e tempest-TestVolumeBackupRestore-1478043453 None] Delete backup started, backup: 0eb54d35-d84d-46e4-96c5-2f336ec4a342. May 14 21:55:46.832371 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-99e8003f-b5ec-406c-9bfd-4fc96f2eba2e tempest-TestVolumeBackupRestore-1478043453 None] Delete started for backup=0eb54d35-d84d-46e4-96c5-2f336ec4a342 {{(pid=123784) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1370}} May 14 21:55:46.832797 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-99e8003f-b5ec-406c-9bfd-4fc96f2eba2e tempest-TestVolumeBackupRestore-1478043453 None] Trying diff format basename='volume-f3e8f8d6-a7b2-4a2b-9ca3-fda68b059872.backup.0eb54d35-d84d-46e4-96c5-2f336ec4a342' for backup base image of volume f3e8f8d6-a7b2-4a2b-9ca3-fda68b059872. {{(pid=123784) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:539}} May 14 21:55:47.174381 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-99e8003f-b5ec-406c-9bfd-4fc96f2eba2e tempest-TestVolumeBackupRestore-1478043453 None] Looking for snapshot of backup base 'volume-f3e8f8d6-a7b2-4a2b-9ca3-fda68b059872.backup.0eb54d35-d84d-46e4-96c5-2f336ec4a342' {{(pid=123784) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:952}} May 14 21:55:47.175051 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-99e8003f-b5ec-406c-9bfd-4fc96f2eba2e tempest-TestVolumeBackupRestore-1478043453 None] Found snapshot 'backup.0eb54d35-d84d-46e4-96c5-2f336ec4a342.snap.1652564916.1542535' {{(pid=123784) _get_backup_snap_name /opt/stack/cinder/cinder/backup/drivers/ceph.py:971}} May 14 21:55:47.175704 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-99e8003f-b5ec-406c-9bfd-4fc96f2eba2e tempest-TestVolumeBackupRestore-1478043453 None] Deleting backup snapshot='backup.0eb54d35-d84d-46e4-96c5-2f336ec4a342.snap.1652564916.1542535' {{(pid=123784) _delete_backup_snapshot /opt/stack/cinder/cinder/backup/drivers/ceph.py:497}} May 14 21:55:50.135811 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.drivers.ceph [None req-99e8003f-b5ec-406c-9bfd-4fc96f2eba2e tempest-TestVolumeBackupRestore-1478043453 None] Deleting backup base image='volume-f3e8f8d6-a7b2-4a2b-9ca3-fda68b059872.backup.0eb54d35-d84d-46e4-96c5-2f336ec4a342' of volume f3e8f8d6-a7b2-4a2b-9ca3-fda68b059872. May 14 21:55:55.479252 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_db.sqlalchemy.engines [None req-85b1c192-0a07-4fed-b3a8-7524167e72b8 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=123784) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} May 14 21:56:01.041963 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-99e8003f-b5ec-406c-9bfd-4fc96f2eba2e tempest-TestVolumeBackupRestore-1478043453 None] Base backup image='volume-f3e8f8d6-a7b2-4a2b-9ca3-fda68b059872.backup.0eb54d35-d84d-46e4-96c5-2f336ec4a342' of volume f3e8f8d6-a7b2-4a2b-9ca3-fda68b059872 deleted. {{(pid=123784) _try_delete_base_image /opt/stack/cinder/cinder/backup/drivers/ceph.py:587}} May 14 21:56:01.341380 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.backup.drivers.ceph [None req-99e8003f-b5ec-406c-9bfd-4fc96f2eba2e tempest-TestVolumeBackupRestore-1478043453 None] Delete of backup '0eb54d35-d84d-46e4-96c5-2f336ec4a342' for volume 'f3e8f8d6-a7b2-4a2b-9ca3-fda68b059872' finished. {{(pid=123784) delete_backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:1398}} May 14 21:56:01.491975 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.quota [None req-99e8003f-b5ec-406c-9bfd-4fc96f2eba2e tempest-TestVolumeBackupRestore-1478043453 None] Created reservations ['f09a5afb-b0be-44d5-973e-bf9b19c58a4b', '6f60d0f7-af65-44fc-bb64-dada72abc530'] {{(pid=123784) reserve /opt/stack/cinder/cinder/quota.py:755}} May 14 21:56:01.800825 ncdv-master-1 cinder-backup[123784]: INFO cinder.backup.manager [None req-99e8003f-b5ec-406c-9bfd-4fc96f2eba2e tempest-TestVolumeBackupRestore-1478043453 None] Delete backup finished, backup 0eb54d35-d84d-46e4-96c5-2f336ec4a342 deleted. May 14 21:56:17.125762 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:56:17.126462 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:57:17.121394 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:57:17.124406 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:58:17.120765 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:58:17.121387 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 21:59:17.116547 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 21:59:17.117647 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:00:17.112897 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:00:17.113628 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:01:17.118812 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:01:17.119562 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:02:17.114666 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:02:17.115392 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:03:17.113006 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:03:17.114821 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:04:17.110869 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:04:17.111628 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:05:17.117110 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:05:17.117765 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:06:17.121957 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:06:17.123322 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:07:17.118679 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:07:17.119323 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:08:17.123771 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:08:17.124477 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:09:17.123087 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:09:17.123766 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:10:17.128839 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:10:17.129491 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:12:17.125839 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:12:17.128745 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:13:17.126986 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:13:17.127698 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:14:17.133534 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:14:17.134198 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:15:17.138827 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:15:17.139436 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:16:17.144491 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:16:17.145512 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:17:17.149893 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:17:17.150574 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:18:17.147184 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:18:17.147891 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:19:17.152270 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:19:17.152907 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:20:17.157993 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:20:17.158593 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:21:17.155247 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:21:17.156337 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:22:17.152419 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:22:17.153109 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:23:17.157770 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:23:17.158402 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:24:17.162764 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:24:17.163408 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:25:17.190120 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:25:17.190120 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:26:17.168392 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:26:17.169017 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:27:17.173186 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:27:17.173731 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:28:17.178543 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:28:17.179193 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:29:17.184004 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:29:17.190379 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:30:17.189338 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:30:17.190051 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:31:17.194888 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:31:17.195701 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:32:17.200177 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:32:17.201184 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:33:17.205692 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:33:17.206355 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:34:17.211330 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:34:17.212072 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:35:17.216885 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:35:17.218071 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:36:17.213243 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:36:17.213965 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:37:17.218345 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:37:17.219029 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:38:17.223916 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:38:17.224574 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:39:17.219365 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:39:17.220100 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:40:17.214913 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:40:17.215597 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:41:17.210552 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:41:17.211213 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:42:17.215725 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:42:17.216390 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}} May 14 22:43:17.220251 ncdv-master-1 cinder-backup[123784]: DEBUG oslo_service.periodic_task [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Running periodic task BackupManager.publish_service_capabilities {{(pid=123784) run_periodic_tasks /usr/local/lib/python3.8/dist-packages/oslo_service/periodic_task.py:210}} May 14 22:43:17.220919 ncdv-master-1 cinder-backup[123784]: DEBUG cinder.manager [None req-afd614e7-2fd9-4b3f-adbf-14904419aca5 None None] Notifying Schedulers of capabilities ... {{(pid=123784) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:202}}